This paper presents the damage assessment of a building structure by using a novel optical fiber accelerometer system. Especially, a sub-scaled building model is designed and manufactured to check up the feasibility of the optical fiber accelerometer for structural health monitoring. The novel accelerometer exploits the moir$acute{e}$ fringe optical phenomenon and two pairs of optical fibers to measure the displacement with a high accuracy, and furthermore a pendulum to convert the displacement into acceleration. A prototype of optical fiber accelerometer system has been successfully developed that consists of a sensor head, a control unit and a signal processing unit. The building model is also designed as a 4-story building with a rectangular shape of $200{ imes}300$ mm of edges. Each floor is connected to the next ones by 6 steel columns which are threaded rods. Basically, a random vibration test of the building model is done with a shaker and all of acceleration data is successfully measured at the assigned points by the optical fiber accelerometer. The experiments are repeated in the undamaged state and the damaged state. The comparison of dynamic parameters including the natural frequencies and the eigenvectors is successfully carried out. Finally, the optical fiber accelerometer is proven to be prospective to evaluate dynamic characteristics of a building structure for the damage assessment.