In this paper, we propose a new color histogram model for object tracking. The proposed model incorporates the color arrangement of the target that encodes the relative spatial distribution of the colors inside the object. Using the color arrangement, we can determine which color bin is more reliable for tracking. Based on the proposed color histogram model, we derive a mean shift framework using a modified Bhattacharyya distance. In addition, we present a method of updating an object scale and a target model to cope with changes in the target appearance. Unlike conventional mean shift based methods, our algorithm produces satisfactory results even when the object being tracked shares similar colors with the background.