A realtime face tracking algorithm using motion analysis from image sequences and color information is proposed. Motion area from the realtime moving images is detected by calculating temporal derivatives first, candidate pixels which represent face region is extracted by the fusion filtering with multiple color models, and realtime face tracking is performed by discriminating face components which includes eyes and lips. We improve the stability of face tracking performance by using template matching with face region in an image sequence and the reference template of face components.