In this paper, we propose an intelligent surveillance system using fuzzy contrast and HOG method. This surveillance system is mainly for the intruder detection. In order to enhance the brightness difference, we apply fuzzy contrast and also apply subtraction method to before/after the surveillance. Then the system identifies the intrusion when the difference of histogram between before/after surveillance is sufficiently large. If the incident happens, the camera stops automatically and the analysis of the screen is performed with fuzzy binarization and Blob method. The intruder is detected and tracked in real time by HOG method and linear SVM. The proposed system is implemented and tested in real world environment and showed acceptable performance in both detection rate and tracking success rate.