In this paper, a methodology for optimal PI supplementary controller using the modified genetic algorithm has been proposed to the oscillation damping in HDVC transmission system. These study processes are summarized as the formulation for load flow calculation in HVDC transmission system with SVC, the investigations on the basic control in HVDC system, the mathematical modeling for dynamic characteristics analyses, and the optimal design of MGA based PI controller generation the supplementary control signal of SVC. Its properties were verified through a series of computer simulations including dynamic stability. It means that the application of MGA-PI controller in HVDC transmission system can contribute the propriety to the improvement of the stability in HVDC transmission system and the design of MGA-OI controller has been proved indispensible when applied to HVDC transmission system.