In this study, a wavelet-based covariance-driven system identification technique is proposed for damage assessment of structures under ambient excitation. Assuming the ambient excitation to be a white-noise process, the covariance computation is shown to be able to separate the effect of random excitation from the response measurement. Wavelet transform (WT) is then used to convert the covariance response in the time domain to the WT magnitude plot in the time-scale plane. The wavelet coefficients along the curves where energy concentrated are extracted and used to estimate the modal properties of the structure. These modal property estimations lead to the calculation of the stiffness matrix when either the spectral density of the random loading or the mass matrix is given. The predicted stiffness matrix hence provides a direct assessment on the possible location and severity of damage which results in stiffness alteration. To demonstrate the proposed wavelet-based damage assessment technique, a numerical example on a 3 degree-of-freedom (DOF) system and an experimental study on a three-story building model, which are all under a broad-band excitation, are presented. Both numerical and experimental results illustrate that the proposed technique can provide an accurate assessment on the damage location. It is however noted that the assessment of damage severity is not as accurate, which might be due to the errors associated with the mode shape estimations as well as the assumption of proportional damping adopted in the formulation.