A visually servoed paired structured light system (ViSP) was recently proposed as a novel estimation method of the 6-DOF (Degree-Of-Freedom) relative displacement in civil structures. In order to apply the ViSP to massive structures, multiple ViSP modules should be installed in a cascaded manner. In this configuration, the estimation errors are propagated through the ViSP modules. In order to resolve this problem, a displacement estimation error back-propagation (DEEP) method was proposed. However, the DEEP method has some disadvantages: the displacement range of each ViSP module must be constrained and displacement errors are corrected sequentially, and thus the entire estimation errors are not considered concurrently. To address this problem, a pose-graph optimized displacement estimation (PODE) method is proposed in this paper. The PODE method is based on a graph-based optimization technique that considers entire errors at the same time. Moreover, this method does not require any constraints on the movement of the ViSP modules. Simulations and experiments are conducted to validate the performance of the proposed method. The results show that the PODE method reduces the propagation errors in comparison with a previous work.