In modern times, road pavement is mainly made of asphalt. Such roads enable stable high-speed travel of the vehicle due to excellent road surface packaging and provide a good ride for the driver. However, the road surface is repeatedly damaged by various causes. The typical problems are cracks, deformation, and road surface flaws, which bring about traffic accidents causing various personal injury. In order to prevent it, an efficient road surface management technique is required and various methods have been developed. Among them, a method using a black box type image acquisition skill has been proposed these days. Although various image processing techniques have existed using this method, image recognition technology based on deep neural network has been most actively studied these days. This paper also introduces the results of the image recognition algorithm using the deep neural network and the its real driving tests. The first is the asymmetric auto-encoded deep neural network for determining the road surface damage location. This deep neural network takes the image as input and determines whether the road surface is damaged or not, and detects the damaged area. The second is the road test that is performed in outdoor road environment using the above and the performance of the image recognition technology is\nevaluated.