The layered coding structure of scalable video coding (SVC) with adaptive inter-layer prediction causes noticeable computational complexity increments when compared to existing video coding standards. To lighten the computational complexity of SVC, we present a fast algorithm to speed up the inter-mode decision process. The proposed algorithm terminates inter-mode decision early in the enhancement layers by estimating the rate-distortion (RD) cost from the macroblocks of the base layer and the enhancement layer in temporal, spatial, and inter-layer directions. Moreover, a search range decision algorithm is also proposed in this paper to further increase the motion estimation speed by using the motion vector information from temporal, spatial, or inter-layer domains. Simulation results show that the proposed algorithm can determine the best mode and provide more efficient total coding time saving with very slight RD performance degradation for spatial and quality scalabilities.