B frame bi-directional predictions and the DIRECT mode coding of the H.264 video compression standard necessitate a complex mode decision process, resulting in a long computation time. To make H.264 feasible, this paper proposes an image backtrack-based fast (IBFD) algorithm and evaluates the performances of two promising fast algorithms (i.e., AFDM and IBFD). Evaluation results show that an image backtrack-based fast (IBFD) algorithm can determine DIRECT mode macroblocks with 13% higher accuracy, as compared with the AFDM. Furthermore, IBFD is shown to reduce the motion estimation time of B frames by up to 23% with a negligible quality degradation.