An AC generator is an important component in producing an electric power and so it requires highly reliable protection relays to minimize the possibility of demage occurring under fault conditions. It is a need for research of digital generator protection system(DGPS) for the next-generation ECMS and an efficient operation of protection control system in power station. However, most of protection and control system used in power plants have been still imported as turn-key and operated in domestic. This may cause the lack of the correct understanding on the protection systems and methods, and thus have difficulties in optimal operation. In this paper, presented ratio differential relaying(RDR) is main protective element in generator protection IED. The fault detection technique, operation zone and setting value of the RDR were studied and, compared with two of the fault detection algorithm. For evaluation performance of the RDR, the data obtained from ATPDraw5.7p4 modeling was used. The proposed methods are shown to be able to rapidly identify internal fault and did not operate a miss-operation for all the external fault.