Protection against transient instability and a consequent out-of-step condition is a major concern for the utility industry. An unstable system may cause serious damage to system elements such as generators and transmission lines, therefore out-of-step detection is essential to operate a system safely. The traditional out-of-step relays detect out-of-step conditions by using distance relays and timers. However, such a relay monitors only apparent impedance which is an indirect function of generator angle, and the relay cannot cope with the out-of-step situation for the more severe instability situation of very fast power swings which can also cause damage to transmission lines if not detected fast enough, as is the case with conventional detection techniques. Digital filters based on discrete Fourier transforms are used to calculate the frequency of a sinusoidal voltage, and then the generator angle is estimated using the deviation of the calculated frequency component of the voltage. The proposed out-of-step detection algorithm is based on the assessment of transient stability using the equal-area criterion. It is verified and tested by using ATP/EMTP MODELS, and the simulation results show that the out-of-step conditions are detected accurately employing the proposed algorithm.