This paper analyzes on the chaos characteristics of the chaotic neural networks and presents the convergence condition. Although the transient chaos of neural network sould be beneficial to overcome the local minimum problem and speed up the learning, the permanent chaotic response gives adverse effect on optimization problems and makes neural network unstable in general. This paper investigates the dynamic characteristics of the chaotic neural networks with the chaotic dynamic neuron, and presents the convergence condition for stabilizing the chaotic neural networks.