Condition-based maintenance has become more popular in recent years because of its advantages in terms of minimizing downtime, extending lifetime, and reducing cost. This kind of maintenance strategy is based on condition monitoring of machinery in operation. Condition monitoring is a key step in maintenance decision analysis. Numerous non-stationary signal processing methods have been developed to reveal fault characteristics in rotating machinery. In this study, an adaptive signal analysis method called empirical mode decomposition is employed for gearbox vibration signal preprocessing. Considering a modulation phenomenon that appeared in a faulty gear, the Hilbert Transform is applied to obtain an envelope signature, which usually contains abundant fault-related signatures. Being different from other failure classification problems, this paper is concerned with determining the probability of normal condition based on current observations describing the condition of a gearbox. Moreover, according to Bayes rule, this problem can be translated to estimate the conditional probability of current observations given normal gearbox condition using a Hidden Markov Model. From this point, a novel probabilistic health description index called Average Probability Index is proposed for gearbox health evaluation. For automatic detection, a semi-dynamic threshold is presented to detect an early fault in a gear. At last, validation and comparative studies are performed using two sets of gearbox lifetime accelerated testing vibration data. The results show the advantages of the proposed method for gearbox condition monitoring.