A video segmentation algorithm, which can extract objects even with non-stationary backgrounds, is proposed in this work. The proposed algorithm is composed of three steps. First, we perform an initial segmentation interactively to build the probability density functions of colors per each macro block via kernel density estimation. Then, for each subsequent frame, we construct a coherence strip, which is likely to contain the object contour, by exploiting spatio-temporal correlations. Finally, we perform the segmentation by minimizing an energy function composed of color, coherence, and smoothness terms. Experimental results on various test sequences show that the proposed algorithm provides accurate segmentation results.