Main sources of the nitration of a gear-pair system are backlash and transmission error, the difference between required and actual rotation during gear meshing. This paper presents the nonlinear dynamic characteristics of gear motions due to the existence of backlash and periodic variation of meshing stiffness, which is assumed as a one-term harmonic component. Gear motions are classified as three types with the consideration of backlash. Each response is calculated using the harmonic balance method and confirmed by numerical integration. The responses with the increase of the rotating speed show abrupt changes in its magnitude for the variation of the preload, exciting force, and damping coefficient. The result also shows that there is a chaotic motion with some specific design parameters and operating conditions In gear diving system. Consequently the design of gear driving system with low nitration and noise requires the study on the effects of nonlinear dynamic characteristics due to stiffness variation and backlash.