The restructure of the electrical power industry is accompanied by the extension of the electrical power exchange. One of the key pieces of information used to determine how much power can be transferred through the network is known as available transfer capability (ATC). The traditional ATC deterministic approach is based on the severest case and it involves a complex procedure. Therefore, a novel approach for A TC calculation is proposed using cost optimization in this paper. The Jeju Island interconnected HVDC system has inland KEPCO (Korean Electric Power Corporation) systems, and its demand is increasing at the rate of about $10[%]$ annually. To supply this increasing demand, the capability of the HVDC system must be enlarged. This paper proposes the optimal transfer capability of the HVDC system between Haenam in the inland and Jeju in Cheju Island through cost optimization. The cost optimization is based on generating cost in Jeju Island, transfer cost through Jeju-Haenam HVDC system and outage cost with one depth (N-1 contingency).