Due to the widespread use of smart devices, threats of direct observation attacks such as shoulder surfing and recording attacks, by which user secrets can be stolen at user interfaces, are increasing greatly. Although formal security models are necessary to evaluate the possibility of and security against those attacks, such a model does not exist. In this paper, based on the previous work in which a HCI cognitive model was firstly utilized for analyzing security, we propose STM-GOMS model as an improvement of GOMS-based model with regard to memory limitations. We then apply STM-GOMS model for analyzing usability and security of a password entry scheme commonly used in smart devices and show the scheme is vulnerable to the shoulder-surfing attack. We finally conduct user experiments to show the results that support the validity of STM-GOMS modeling and analysis.