This paper propose a novel method for tracking moving object based on covariance matrix and Riemannian Manifolds. With image backgrounds continuously changed, we use the covariance matrices to extract features for tracking nonrigid object undergoing transformation and deformation. The covariance matrix can make fusion of different types of features and has its small dimension, therefore we enable to handle the spatial and statistical properties as well as the component correlation. The proposed method can estimate the position of the moving object by employing the covariance matrix of object region as a feature vector and comparing the candidate regions. Rimannian Geometry is efficiently adapted to object deformation and change of shape and improve the accuracy by using geodesic distance to predict the estimated position with the minimum distance. The experimental results have shown that the proposed method correctly tracked the moving object.