The purpose of this study is to predict the dynamic transmission error of the helical gear system. To do so, the equations of motion in the helical gear system which consists of motor, coupling, gear, torque sensor, and brake are derived. As the input parameters, the mass moment of inertia by a 3D CAD software and the equivalent stiffness of the bearings and shaft are calculated and the coupling stiffness is measured. The static transmission error as an excitation is calculated by in-house program. Dynamic transmission error is predicted by solving the equations of motion. Mode shape, the dynamic mesh force and the bearing force are also calculated. In this analysis, the relationship between the dynamic mesh force and the bearing force and mode shape behavior in gear mesh are checked. As a result, the magnitude of mesh force is highly related with the gear mesh behavior in mode shape. The finite element analysis is conducted to find out the natural frequency of gear system. The natural frequencies by finite element analysis have a good agreement with the results by equation of motion. Finally, dynamic transmission error is measured by the specially designed experiment and the results by equation of motion are validated.