The protection against transient instability and consequent out-of-step condition is a major concern for the utility industry. Unstable system may cause serious damage to system elements such as generators and transmission lines. Therefore, out-of-step detection is essential to operate a system safely. The detection of out-of-step is generally based upon the rate of movement of the apparent impedance. However such relay monitors only the apparent impedance which may not be sufficient to correctly detect all forms of out-of-step and cannot cope with out-of-step for a more complex type of instability such as very fast power swing. This paper presents the out-of-step detection algorithm using voltage frequency deviation. The digital filters based on discrete Fourier transforms (DFT) to calculate the frequency of a sinusoid voltage are used, and the generator angle is estimated using the deviation of the calculated frequency component of the voltage. The proposed out-of-step algorithm is based on the assessment of a transient stability using equal area criterion. The proposed out-of-step algorithm is verified and tested by using EMTP MODELS.