In the present paper, a novel fast peak detector for single- or three-phase unsymmetrical voltage sags is proposed. The proposed detector is modified from a single-phase digital phase-locked loop based on a d-q transformation using an all-pass filter (APF). APF generates a virtual phase with $90^{circ}$ phase delay. However, this virtual phase cannot reflect a sudden change of the grid voltage in the moment of voltage sag, which causes a peak value to be significantly distorted and to settle down slowly. Specifically, the settling time of the peak value is too long when voltage sag occurs around a zero crossing, such as phase $0^{circ}$ and $180^{circ}$. This paper describes the operating principle of the APF problem and proposes a modified all-pass filter (MAPF) to mitigate the inherent APF problem. In addition, a new fast peak detector using MAPF is proposed. The proposed detector is able to calculate a peak value within 0.5 ms, even when voltage sag occurs around zero crossing. The proposed fast peak detector is compared with the conventional detector using APF. Results show that the proposed detector has faster detection time in the whole phase range. Furthermore, the proposed fast peak detector can be effectively applied to unsymmetrical three-phase voltage sags. Simulation and experimental results verify the advantages of the proposed detector and MAPF.