This paper describes a road-following controller using the proposed neural network for autonomous vehicle. Road-following with visual sensor like camera requires intelligent control algorithm because analysis of relation from road image to steering control is complex. The proposed neural network, relative similarity modular network(RSMN), is composed of some learning networks and a partitioniing network. The partitioning network divides input space into multiple sections by similarity of input data. Because divided section has simlar input patterns, RSMN can learn nonlinear relation such as road-following with visual control easily. Visual control uses two criteria on road image from camera; one is position of vanishing point of road, the other is slope of vanishing line of road. The controller using neural network has input of two criteria and output of steering angle. To confirm performance of the proposed neural network controller, a software is developed to simulate vehicle dynamics, camera image generation, visual control, and road-following. Also, prototype autonomous electric vehicle is developed, and usefulness of the controller is verified by physical driving test.