The unstable signature that is defined as frequency change with respect to the time or frequency modulation, is caused by the external loading variation in specific machinery component and Doppler shift etc. In this study, we analyze the generation mechanism of the unstable signature and apply the Extended Kalman filter (EKF) algorithm for its detection. The performance of Extended Kalman Filter is examined for numerical and measured signals and the results show its validity for unstable signature detection.