In this paper, we propose the robust object detection algorithm with spatial gradient information. To do this, first, we eliminate error values that appear due to complex environment and various illumination change by using prior methods based on hue and intensity from the input video and background. Visible shadows are eliminated from the foreground by using an RGB color model and a qualified RGB color model. And unnecessary values are eliminated by using the HSI color model. The background is removed completely from the foreground leaving a silhouette to be restored using spatial gradient and HSI color model. Finally, we validate the applicability of the proposed method using various indoor and outdoor conditions in a complex environments.