The purpose of this paper is to suggest IDE (Incremental Displacement Estimation) algorithm for the previously proposed visually servoed paired structured light system. The system is composed of two sides facing with each other, each with one or two lasers with a 2-DOF manipulator, a camera, and a screen. The 6-DOF displacement between two sides can be estimated by calculating the positions of the projected laser beams and rotation angles of the manipulators. In the previous study, Newton-Raphson or EKF (Extended Kalman Filter) has been used as an estimation algorithm. Although the various experimental tests have validated the performance of the system and estimation algorithms, the computation time is relatively long since aforementioned algorithms are iterative methods. Therefore, in this paper, a non-iterative incremental displacement estimation algorithm which updates the previously estimated displacement with a difference of the previous and the current observed data is introduced. To verify the performance of the algorithm, experimental tests have been performed. The results show that the proposed non-iterative algorithm estimates the displacement with the same level of accuracy compared to the EKF with multiple iterations with significantly less computation time.