This paper proposes a pitch error compensation technique with decomposition of mechanical signals based on giant magnetostrictive actuator (GMA) to eliminate tool tracking errors and to reach a high machining precision. In ultra-precision cutting, grinding and nontraditional machining, there is a difficult problem for achieving long travel and high precision. Considering the characteristics of giant magnetostrictive materials (GMM), a micro-displacement GMA with a flexure hinge micro-motion table is designed, and used for error compensation. The inherent hysteresis nonlinearity of the GMA is controlled effectively by a sliding mode robust adaptive controller (SMC). Aiming at reducing the positioning error of a precision working table driven by an AC servo motor, the measurement of the dynamical characteristics of the system is carried out by a laser interferometer. The leadscrew pitch error, which accounts for most of the error detected, is separated out by an improved signal filter algorithm. Experimental results show that the positioning error is reduced to an extent within ${pm}8{mu}m$ from ${pm}20{mu}m$ after compensation, which demonstrates the feasibility of the control and compensation method.