In this paper, we propose an eye detection method that is robust to facial pose changes using gradient directional features of brightness and a particle filter. The rejection function of incorrect eye detection in the proposed method allows for the eye detection again even if the eye is incorrectly detected. In this method, to estimate the boundary between the iris and sclera or eyelid, the gradient intensities are calculated by four directional Prewitt filters in four regions. The likelihood used in the particle filter is obtained by averaging the gradient intensities for the specific direction in the four regions and the upper eyelid area. From experimental results, the average detection rates of both eyes for roll, yaw, and pitch angles of the face are more than 90% by using rejection function for incorrect eye detection. The rejection function produces the 4.4%, 4.5%, and 4.9% increases in average detection rates of both eyes for roll, yaw, and pitch facial angles, respectively. The proposed eye detection method can track both eye in real-time (about 20 ms) and is robust to the facial pose changes.