This study was performed to grasp how the rustling sounds of silk fabrics affected physiological responses of human beings to predict auditory comfort of fabric sound. Sound parameters and physical properties of test specimens were measured and their relationships with the physiological signals were analyzed, respectively. Physiological signals were acquired when each sound was presented to participants. The physiological signals employed in this study were electroencephalogram(EEG), pulse volume(PV), skin conductance level(SCL), and the ratio of low frequency over high frequency(LF/HF) from the power spectrum of heart rate variability(HRV). Sound parameters including LPT(level pressures of total sound), ΔL, Δf, loudness(Z), and sharpness(Z) were calculated and physical properties were measured by the KES-FB system. WT(tensile energy) and T(thickness) were the main factors used to determine loudness(Z) and LPT. LPT and loudness(Z) were found to be the most influential acoustic parameter on physiological responses. By the increase of these acoustic parameters, slow alpha wave and PV decreased, on the other hand, LF/HF and SCL increased. The increase of T of silk fabric was followed by the decrease of slow alpha wave and the increase of SCL, while the increase of WT induces the increase of PV. The threshold values of WT and T which begin to activate physiological responses were 7.8 gf.cm/$ extrm{cm}^2$ for PV and 0.26 mm for SCL.