This paper presents 3D line segment extraction method, which can be used in generating 3D rooftop model. The core of our method is that 3D line segment is extracted by using line fitting of elevation data on 2D line coordinates of ortho-image. In order to use elevations in line fitting, the elevations should be reliable. To measure the reliability of elevation, in this paper, we employ the concept of self-consistency. We test the effectiveness of the proposed method with a quantitative accuracy analysis using synthetic images generated from Avenches data set of Ascona aerial images. Experimental results indicate that the proposed method shows average 30 line errors of .16 - .30 meters, which are about $10\%$ of the conventional area-based method.