This paper describes a real-time face tracking system using effective detector and Kalman filter. In the proposed system, an image is separated into a background and an object using a real-time updated face color for effective face detection. The face features are extracted using the five types of simple Haar-like features. The extracted features are reinterpreted using Principal Component Analysis (PCA), and interpreted principal components are used for Support Vector Machine (SVM) that classifies the faces and non-faces. The moving face is traced with Kalman filter, which uses the static information of the detected faces and the dynamic information of changes between previous and current frames. The proposed system sets up an initial skin color and updates a region of a skin color through a moving skin color in a real time. It is possible to remove a background which has a similar color with a skin through updating a skin color in a real time. Also, as reducing a potential-face region using a skin color, the performance is increased up to 50% when comparing to the case of extracting features from a whole region.