This letter presents a multi-modal approach to tracking geographic objects such as buildings and road signs in a video sequence recorded from a moving vehicle. In the proposed approach, photogrammetric techniques are successfully combined with conventional tracking methods. More specifically, photogrammetry combined with positioning technologies is used to obtain 3-D coordinates of chosen geographic objects, providing a search area for conventional feature trackers. In addition, we present an adaptive window decision scheme based on the distance between chosen objects and a moving vehicle. Experimental results are provided to show the robustness of the proposed approach.