Human detection is a challenging task in many fields because it is difficult to detect humans due to their variable appearance and posture. Furthermore, it is also hard to track the detected human because of their dynamic and unpredictable behavior. The evaluation speed of method is also important as well as its accuracy. In this paper, we propose unified detection and tracking method for humans using Gaussian-PSO (Gaussian Particle Swarm Optimization) with the HOG (Histograms of Oriented Gradients) features to achieve a fast and accurate performance. Keeping the robustness of HOG features on human detection, we raise the process speed in detection and tracking so that it can be used for real-time applications. These advantages are given by a simple process which needs just one linear-SVM classifier with HOG features and Gaussian-PSO procedure for the both of detection and tracking.