GPS/INS integrated systems do not guarantee robustness and accuracy of localization, because GPS has vulnerability to external disturbances. However, the overall performance and reliability of the system can be significantly improved by fusing multiple sensors with a different operating principle. In outdoor environments where GPS may be blocked, there are many features compared to the open space and these features can provide much information for UGV localization. Thus, this paper proposes an improved localization algorithm based on the hierarchical federation of three measurement layers, i.e., GPS, INS, and visual localization, to overcome the shortcomings of GPS/INS integrated systems. The proposed algorithm automatically switches the operation modes according to GPS status and a network of a ground-based reference station. A vocabulary tree with SURF is used in the visual localization method. In the data fusion of visual localization and INS, an asynchronous and time-delayed data fusion algorithm is presented because visual localization is always time-delayed compared with INS. By using DGPS to obtain the reference position under the dynamic conditions of the reference station, the restrictions of the conventional DGPS are overcome and all UGVs within WiBro communication range of the reference station can accurately estimate the position with a common GPS. The experiment results with a predefined path demonstrate enhancement of the robustness and accuracy of localization in outdoor environments.