In this paper, we propose a video-based intelligent unmanned fire surveillance system using fuzzy color models. In general, to detect heat or smoke, a separate device is required for a fire surveillance system, this system, however, can be implemented by using widely used CCTV, which does not need separate devices and extra cost. The systems called video-based fire surveillance systems use mainly a method extracting smoke or flame from an input image only. The smoke is difficult to extract at night because of its gray-scale color, and the flame color depends on the temperature, the inflammable, the size of flame, etc, which makes it hard to extract the flame region from the input image. This paper deals with a intelligent fire surveillance system which is robust against the variation of the flame color, especially at night. The proposed system extracts the moving object from the input image, makes a decision whether the object is the flame or not by means of the color obtained by fuzzy color model and the shape obtained by histogram, and issues a fire alarm when the flame is spread. Finally, we verify the efficiency of the proposed system through the experiment of the controlled real fire.