Varying speed machinery condition detection and fault diagnosis are more difficult due to non-stationary machine dynamics and vibration. Therefore, most conventional signal processing methods based on time invariant carried out in constant time interval are frequently unable to provide meaningful results. In this paper, a study is presented to apply order cepstrum and radial basis function (RBF) artificial neural network (ANN) for gear fault detection during speedup process. This method combines computed order tracking, cepstrum analysis with ANN. First, the vibration signal during speed-up process of the gearbox is sampled at constant time increments and then is re-sampled at constant angle increments. Second, the re-sampled signals are processed by cepstrum analysis. The order cepstrum with normal, wear and crack fault are processed for feature extracting. In the end, the extracted features are used as inputs to RBF for recognition. The RBF is trained with a subset of the experimental data for known machine conditions. The ANN is tested by using the remaining set of data. The procedure is illustrated with the experimental vibration data of a gearbox. The results show the effectiveness of order cepstrum and RBF in detection and diagnosis of the gear condition.