In this paper, we present an effective covariance tracking algorithm based on adaptive size changing of tracking window. Recent researches have advocated the use of a covariance matrix of object image features for tracking objects instead of the conventional histogram object models used in popular algorithms. But, according to the general covariance tracking algorithm, it can not deal with the scale changes of the moving objects. The scale of the moving object often changes in various tracking environment and the tracking window(or object kernel) has to be adapted accordingly. In addition, the covariance matrix of moving objects should be adaptively updated considering of the tracking window size. We provide a solution to this problem by segmenting the moving object from the background pixels of the tracking window. Therefore, we can improve the tracking performance of the covariance tracking method. Our several simulations prove the effectiveness of the proposed method.