In this paper, we presented an attention measurement method in 2D and 3D image/video to be applied for image and video retargeting and compressive processing. 2D attention is derived from the three main components, intensity, color, and orientation, while depth information is added for 3D attention. A rarity-based attention method is presented to obtain more interested region or objects. Displaced depth information is matched to attention probability in distorted stereo images and finally a stereo distortion predictor is designed by integrating low-level HVS responses. As results, more efficient attention scheme is developed from the conventional methods and performance is proved by applying for video retargeting.