In this paper, we propose a moving object tracking algorithm by using motion estimation in UAV(Unmanned Aerial Vehicle) video. Proposed algorithm is based on generation of initial image from detected reference image, and tracking of moving object under the time-varying image. With a series of this procedure, tracking process is stable even when the UAV camera sways by correcting position of moving object, and tracking time is relatively reduced. A block matching algorithm is also utilized to determine the similarity between reference image and moving object. An experimental result shows that our proposed algorithm is better than the existing full search algorithm.