A defect in an image is a set of pixels forming an irregular shape. Since a defect, in most cases, is not easy to be modeled mathematically, the defect detection problem still resides in a research area. If a given image, however, composed by certain patterns, a defect can be detected by the fact that a non-defect area should be explained by another patch in terms of a rotation, translation, and noise. In this paper, therefore, the defect detection method for a repeated multi-patterned image is proposed. The proposed defect detection method is composed of three steps. First step is the interest point detection step, second step is the selection step of a appropriate patch size, and the last step is the decision step. The proposed method is illustrated using SEM images of semiconductor wafer samples.