In this paper, we propose a new method of indexing and searching based on object-specific features at different semantic levels for video retrieval. A moving trajectory model is used as an indexing key for accessing the individual object in the semantic level. By tracking individual objects with segmented data, we can generate motion trajectories and set model parameters using polynomial curve fitting. The proposed searching scheme supports various types of queries including query by example, query by sketch, and query on weighting parameters for event-based video retrieval. When retrieving the interested video clip, the system returns the best matching event in the similarity order.