In this Paper, we propose fast block matching algorithm by dividing complex areas based on complexity order of reference block and square sub-block to reduce an amount of computation of full starch(FS) algorithm for fast motion estimation, while keeping the same prediction quality compared with the full search algorithm. By using the fact that matching error is proportional to the gradient of reference block, we reduced unnecessary computations with square sub-block adaptive matching scan based image complexity instead of conventional sequential matching scan and row/column based matching scan. Our algorithm reduces about $30\%$ of computations for block matching error compared with the conventional partial distortion elimination(PDE) algorithm without any prediction quality, and our algorithm will be useful in real-time video coding applications using MPEG-4 AVC or MPEG-2.