An effective algorithm for tracking rigid or non-rigid moving object(s) which segments local moving parts from image sequence in the presence of backgraound motion by camera movenment, predicts the direction of it, and tracks the object is proposed. It requires no camera calibration and no knowledge of the installed position of camera. In order to segment the moving object, feature points configuring the shape of moving object are firstly selected, feature flow field composed of motion vectors of the feature points is computed, and moving object(s) is (are) segmented by clustering the feature flow field in the multi-dimensional feature space. Also, we propose IRMAS, an efficient algorithm that finds the convex hull in order to cinstruct the shape of moving object(s) from clustered feature points. And, for the purpose of robjst tracking the objects whose movement characteristics bring about the abrupt change of moving trajectory, an improved order adaptive lattice structured linear predictor is used.