With the growing popularity of digital video applications, those areas of the efficient transmit, storage management, and search technology for video data are emerging as an important core technology. To be an effective video indexing system, users need to be able to find the video segments that they want. Unfortunately, video data is difficult to manage because of its unstructured data type and large volume with linear forms. This paper proposes a shot verification using visual rhythm and video retrieval system. The shot verification is designed to detect a segment from video easily and quickly, known as shot boundaries, just by changing the visual rhythm without playing the image. Therefore, this can decrease the false detected shots and generate the unidentified shots and key frames. The retrieval system is constructed in terms of visual descriptor from the list of MPEG-7. The effectiveness of the proposed shot verification process and video retrieval system is demonstrated.