In this paper, we propose a drowsy driver detection system with a novel eye state detection method that is adaptive to various vehicle environment such as glasses, light and so forth using SURF(Speed Up Robust Feature) which can extract quickly local features from images. Also the performance of eye state detection is improved as individual three eye-state templates of each driver can be made using Bayesian inference. The experimental results under various environment with average 98.1% and 96.1% detection rate in the daytime and at night respectively and those in the opened ZJU database with average 97.8% detection rate show that the proposed method outperforms the current state-of-the-art.