In this paper, we proposed a feature-based tracking system that traces moving objects with a pan-tilt camera after separating the global motion of an active camera and the local motion of moving objects. The tracking system traces only the local motion of the comer features in the foreground objects by finding the block motions between two consecutive frames using a block-based motion estimation and eliminating the global motion from the block motions. For the robust estimation of the camera motion using only the background motion, we suggest a dominant motion extraction to classify the background motions from the block motions. We also propose an efficient clustering algorithm based on the attributes of motion trajectories of corner features to remove the motions of noise objects from the separated local motion. The proposed tracking system has demonstrated good performance for several test video sequences.