In this study we present a new improved nonlinear calibration method for vector magnetograms made by the Solar Flare Telescope of BOAO. To identify Fe I 6302.5 line, we have scanned monochromatic images of the line integrated over filter passband, changing the location of the central transmission wavelength of a Lyot filter. Then we obtained a filter-convolved line profile, which is in good agreement with spectral atlas data provided by the Sacramento Peak Solar Observatory. The line profile has been used to derive calibration coefficients of longitudinal and transverse fields, employing the conventional line slope method under the weak field approximation. Our improved nonlinear calibration method has also been used to calculate theoretical Stokes polarization signals with various angles of inclination of magnetic fields. For its numerical test, we have compared input magnetic fields with the calibrated ones, which have been derived from the new improved non-linear method and the conventional method respectively. The numerical test shows that the calibrated fields obtained from the improved method are consistent with the input fields, but not with those from the conventional method. Finally, we applied our new improved method to a dipole model which characterizes a typical field configuration of a single, round sunspot. It is noted that the conventional method remarkably underestimates the transverse field component near the inner penumbra.