The nonlinearity of a laser interferometer usually ranges from sub-nanometer to several manometers. This nonlinearity, which has periodic characteristics, limits the accuracy of the interferometer at the sub-nanometer level. The nonlinearity error of the one-frequency homodyne interferometer with quadrature fringe detection results from a number of factors including polarization mixing by imperfect optical elements, unequal gain of photo detectors, lack of quadrature between two signals and misalignment. In this paper, we described a method for measuring and compensating the nonlinearity of homodyne interferometer using the elliptical fitting technique with least-square method. Experimental results demonstrate that $^pm$3.5 nm nonlinearity can be reduced to $^pm$0.2 nm level.