In this paper, a method of designing a controller which ensures the synchronization between the transmission and the reception ends of chaotic secure communication systems with interval time-varying delays is proposed. To increase communication security, the transmitted message is encrypted with the techniques of N-shift cipher and public key. And to reduce the conservatism of the stabilization criterion for error dynamic system obtained from the transmitter and receiver, a new Lyapunov-functional and bounding technique are proposed. Through a numerical example, the effectiveness of the proposed method is shown in the chaotic secure communication system.