The characteristics of a rock joint which influence the stability of rock mass structures such as cut slopes and tunnels are largely controlled by the conditions of the rock joint as well as its boundary conditions. The conditions of rock joints comprise asperity strength, roughness, and filling materials. Boundary conditions can be represented by assuming that the deformability(or stiffness) of the rock mass surrounding the joints is modelled by a spring with stiffness. A new direct shear apparatus was developed in this study, which adapts a servo control system using PID algorithm. This apparatus can be used to investigate the various aspects of shear characteristics of the rock joints at conditions of constant normal stress and constant normal stiffness and so on. The test results for saw-cut teeth joints show that shear strength should be evaluated by considering its specific boundary conditions far the design of tunnels and cut slopes.