As civil structures are exposed to various external loads, it is essential to assess the structural condition, especially the structural displacement, in every moment. Therefore, a visually servoed paired structured light system was proposed in the previous study. The proposed system is composed of two screens facing with each other, each with a camera, a screen, and one or two lasers controlled by a 2-DOF manipulator. The 6-DOF displacement can be calculated from the positions of three projected laser beams and the rotation angles of the manipulators. In the estimation process, one of well-known iterative methods such as Newton-Raphson or extended Kalman filter (EKF) was used for each measurement. Although the proposed system with the aforementioned algorithms estimates the displacement with high accuracy, it takes relatively long computation time. Therefore, an incremental displacement estimation (IDE) algorithm which updates the previously estimated displacement based on the difference between the previous and the current observed data is newly proposed. To validate the performance of the proposed algorithm, simulations and experiments are performed. The results show that the proposed algorithm significantly reduces the computation time with the same level of accuracy compared to the EKF with multiple iterations.