We propose a new building detection and description algorithm for Lidar data and photogrammetric imagery using color segmentation, line segments matching, perceptual grouping. Our algorithm consists of two steps. In the first step, from the initial building regions extracted from Lidar data and the color segmentation results from the photogrammetric imagery, we extract coarse building boundaries based on the Lidar results with split and merge technique from aerial imagery. In the secondstep, we extract precise building boundaries based on coarse building boundaries and edges from aerial imagery using line segments matching and perceptual grouping. The contribution of this algorithm is that color information in photogrammetric imagery is used to complement collapsed building boundaries obtained by Lidar. Moreover, linearity of the edges and construction of closed roof form are used to reflect the characteristic of man-made object. Experimental results on multisensor data demonstrate that the proposed algorithm produces more accurate and reliable results than Lidar sensor.