Damage detection based on a reference set of measured data usually has the problem of different environmental temperature in the two sets of measurements, and the effect of temperature difference is usually ignored in the subsequent model updating. This paper attempts to identify the structural damage including the temperature difference with artificial measurement noise. Both local damages and the temperature difference are identified in a gradient-based model updating method based on dynamic response sensitivity. The sensitivities of dynamic response with respect to the system parameters and temperature difference are calculated by direct integration method. The measured dynamic responses of the structure from two different states are used directly to identify the structural local damages and the temperature difference. A single degree-of-freedom mass-spring system and a planar truss structure are studied to illustrate the effectiveness of the proposed method.