With the development of mobile devices and intelligent surveillance system loaded with pan/tilt cameras, object tracking with non-stationary cameras has become a topic with increasing importancy. Since it is difficult to model a background image in a non-stationary camera environment, colors and texture are the most important features in the tracking algorithm. However, colors in the background similar to those in the target arise instability in the tracking. Recently, we proposed a robust color based tracking algorithm that uses an area weighted centroid shift. In this letter, we update the model such that it becomes more stable against background colors. The proposed algorithm also incorporates time filtering by adding an additional energy term to the energy functional.