A large AC generator is an important component of the power system. It is a need for research of AC generator protection relay for the next-generation ECMS and an efficient operation of protection control system. However, most of protection and control systems used in power plants have been still imported as turn-key and operated in domestic. This may cause the lack of the correct understanding on the protection systems and methods, and thus have difficulties in optimal operation. In this paper, over-excitation protection relaying is one of the protective factors in generator protection IED was presented. The cause and protection principles were dealt and, DFT-based gain compensation algorithm was adopted for the frequency measurement. The Republic of Korea 345kV modeling data by EMTP-RV was used for performance evaluation.