In this paper, a new target extraction algorithm is proposed, in which the coordinates of target are obtained adaptively by using the difference image information and the optical BPEJTC(binary phase extraction joint transform correlator) with which the target object can be segmented from the input image and background noises are removed in the stereo vision system. First, the proposed algorithm extracts the target object by removing the background noises through the difference image information of the sequential left images and then controlls the pan/tilt and convergence angle of the stereo camera by using the coordinates of the target position obtained from the optical BPEJTC between the extracted target image and the input image. From some experimental results, it is found that the proposed algorithm can extract the target object from the input image with background noises and then, effectively track the target object in real time. Finally, a possibility of implementation of the adaptive stereo object tracking system by using the proposed algorithm is also suggested.