A sophisticated story-wise stiffness identification method for a shear building structure is applied to the case where the shear building is subjected to an actual micro-tremor. While the building responses to earthquake ground motions are necessary in the previous method, it is shown that micro-tremors can be used for identification within the same framework. This enhances the extended usability and practicality of the previously proposed identification method. The difficulty arising in the limit manipulation at zero frequency in the previous method is overcome by introducing an ARX model. The weakness of small SN ratios in the low frequency range is avoided by using the ARX model together with filtering and introducing new constraints on the ARX parameters.