In this paper, we proposed an image retrieval system with a simple and effective relevance feedback, called RAP(Reward and Punishment) algorithm. First, color and texture features were extracted from the images. Next, the extracted feature values were used for image retrieval in various forms. We applied the relevance feedback to the initial retrieved images from the image retrieval system, and compared its result with that of the conventional system. In the experiment using the test image database of 16 class 512 images, the proposed system showed the better retrieval performance of about 10∼l7 % than that of the conventional INRIA system in each relevance feedback step.