An effective algorithm for implementation of unmanned surveillance system which detects moving object from image sequences, predicts the direction of it, and drives the camera in real time is proposed. Outputs of proposed algorithm are coordinates of location of moving object, and they are converted to the values according to camera model. As a pre- processing, extraction of moving object and shape discrimination are performed. Existence of the moving object or scene change is detected by computing the temporal derivatives of consecutive two or more images in a sequence, and this result of derivatives is combined with the edge map from one original gray level image to obtain the position of moving object. Shape discri-mination(Target identification) is performed by analysis of distribution of projection profiles in x and y directions. To reduce the prediction error due to the fact that the motion cha- racteristic of walking man may have an abrupt change of moving direction, an order adaptive lattice structured linear predictor is proposed.