To reduce huge computation in the block matching, this paper proposes a fast block matching algorithm which limits search points in the search area. On the basis of two facts that most motion vectors are located in central part of search area and matching error is monotonic decreasing toward the best similar block, the proposed algorithm moves a matching pattern between steps by the one pixel, predicts the motion direction for the best similar block from similar blocks decided in previous steps, and limits movements of search points to ${pm}45^{circ}C$ on it. As a result, it could remove the needless search points and reduce the block matching computation. In comparison with the conventional similar algorithms, the proposed algorithm caused the trivial image degradation in images with fast motion but kept the equivalent image quality in images with normal motion, and it, meanwhile, reduced from about 20% to over 67% of the their block matching computation.