With the goal of developing an accurate and fast lane tracking system for the purpose of driver assistance, this paper proposes a vision-based fusion technique for lane tracking and forward vehicle detection to handle challenging conditions, i.e., lane occlusion by a forward vehicle, lane change, varying illumination, road traffic signs, and pitch motion, all of which often occur in real driving environments. First, our algorithm uses random sample consensus (RANSAC) and Kalman filtering to calculate the lane equation from the lane candidates found by template matching. Simple template matching and a combination of RANSAC and Kalman filtering makes calculating the lane equation as a hyperbola pair very quick and robust against varying illumination and discontinuities in the lane. Second, our algorithm uses a state transfer technique to maintain lane tracking continuously in spite of the lane changing situation. This reduces the computational time when dealing with the lane change because lane detection, which takes much more time than lane tracking, is not necessary with this algorithm. Third, false lane candidates from occlusions by frontal vehicles are eliminated using accurate regions of the forward vehicles from our improved forward vehicle detector. Fourth, our proposed method achieved robustness against road traffic signs and pitch motion using the adaptive region of interest and a constraint on the position of the vanishing point. Our algorithm was tested with image sequences from a real driving situation and demonstrated its robustness.