In this paper, we propose a loitering detection using trajectory probability distribution and local direction descriptor for intelligent surveillance system. We use a background modeling method for detecting moving object and extract the motion features from each moving object for making feature vectors. After that, we detect the loitering behavior person using K-Nearest Neighbor classifier. We test the proposed method in real world environment and it can achieve real time and robust detection results.