A new algorithm for removing salt and pepper impulse noise in image using impulse noise detector and total variation optimization is presented. The proposed two types of noise detectors which are based on the adaptive median filter, can detect impulse noise with high accuracy while reducing the probability of detecting image details as impulses. And the detectors maintain its performance independent of noise density. For removing impulses, total variation optimization is applied only to those detected noise candidate to reduces unnecessary computation. The proposed approach successfully remove impulse noise while preserving image details.