This paper proposes new phase error compensation method of RVDT encoder in the FFT domain. Phase errors are measured with a small combination of compensation resistors and the changes of first order coefficients of FFT for each resistor are obtained. It is found that the coefficient change is inversely proportional to the inserted resistor. The proposed method takes less time and the size of the table is smaller than previous time domain approaches. In addition, the location of the compensation resistor can be found through axis transformation of the coefficients. Finally, the peak-to-peak phase error was improved to 0.57 which is two times better than previous one.