In this paper, we propose a robust moving object tracking(MOT) method based on motion energy and prediction. MOT consists of two steps: moving object extraction step(MOES) and moving object tracking step(MOTS). For MOES, we use improved motion energy method. For MOTS, we predict the next location of moving object based on distance and direction information among previous instances, so that we can reduce the search space for correspondence. We apply the method to both synthetic and real world sequences and find that the method works well even in the presence of occlusion and disocclusion.