This paper deals with the consensus problem of the second-order multi-agent systems based on linear system theory. By a system transformation, the consensus problem is converted to the stability problem of a linear system. The second-order systems considered include the systems with both the fixed topology and the Markov switching topology. Necessary and sufficient conditions of consensus or mean square consensus are obtained. The results are extended to the cases of the uncertain transition probability rates and time-delay, respectively. Finally, a simulation example is given to show the effectiveness of the presented results.