Optical measurement methods make it possible to detect object displacements with high resolution and noncontact measurements. Also, they are very robust against EMI noises and have long operation range. An optical triangulation sensor is one of widely used displacement measurement sensors for its sub-micron resolution, fast response, simple structure, and low cost. However. there are several errors caused by inclinations of a surface. speckle effects, power fluctuations of light sources, and noises of detectors. In this paper, in order to minimize error effects, we performed error analysis and proposed a new structure. Then, we setup a new modeling method and verify it through simulations and experiments. Based on the new model. we propose a new sensor structure and establish design criteria. Finally, we design a signal processing system to overcome a resolution-limited problem of light detectors. The resolution of the proposed system is 0.2${mu}{ extrm}{m}$ in 5mm operating range.