This paper reports an efficient algorithm for accurate rolling shutter distortion removal. A hierarchical global motion estimation approach for a group of blocks reduces the level of computation by three orders of magnitude. In addition, the motion of each scanline is determined accurately by averaging two candidates obtained through cubic spline interpolation. The experimental results show that the proposed method produces accurate motion information with significant computation reduction and corrects the rolling shutter distortion effectively.