In this paper, a closed-loop system constructed with a linear plant and nonlinearity in the feedback connection is considered to argue against its planar orbital stability. Through a state space approach, a main result that presents a sufficient stability criterion of the limit cycle predicted by solving the harmonic balance equation is given. Preliminarily, the harmonic balance of the nonlinear feedback loop is assumed to have a solution that determines the characteristics of the limit cycle. Using a state-space approach, the nonlinear loop equation is reformulated into a linear perturbed model through the introduction of a residual operator. By considering a series of transformations, such as a modified eigenstructure decomposition, periodic averaging, change of variables, and coordinate transformation, the stability of the limit cycle can be simply tested via a scalar function and matrix. Finally, the stability criterion is addressed by constructing a composite Lyapunov function of the transformed system.