To improve productivity of a metal cutting process, it is required to monitor machining stability in real time. Since cutting environment is harsh against sensing conditions due to vibration, chip, and cutting fluid, etc., it is necessary to develop a robust and reliable sensing system for the practical application. In this work, a chatter monitoring system was developed and its effectiveness was proved. Spindle displacement caused by cutting was selected as a main monitoring parameter. A cylindrical capacitive displacement sensor was adopted. Chatter frequencies were identified through modal analysis. To quantify chatter vibrations, chatter correlation coefficient was introduced. The identification of the monitoring system showed a good agreement with the result of experiment.