This study deals with the perceptual sensitivity to Head-Related Impulse Responses (HRIRs) in the median plane based on a series of subjective listening tests using a pair of headphones. First, the non-individualized HRIRs were modeled from 12 principal components (PCs) extracted from Principal Components Analysis (PCA) of the CIPIC HRTF database. The Just Noticeable Difference (JND) in weight of PCs (PCWs) at each elevation was estimated. It was not observed the common elevation-dependent tendency or PCW-dependent tendency of JND in PCW s across the five subjects who participated in the tests, and the inter-subject variation of JND in PCWs was large. The JND in HRIRs can be estimated indirectly from the JND in PCWs because the HRIRs can be represented by a linear summation of the PCs weighted by PCWs. The common elevation-dependent tendency of JND in Directional Impulse Responses (DIRs), which are the mean-subtracted HRIRs, across the five subjects can be found. The change in PCWs does not seem to contribute to our perception of sound source characteristics; however, the resulting change in HRIRs due to the change in PCWs seems to contribute. The subjects showed larger JND in DIRs in the frontal region than in the rear region. This means that our perception of sound source characteristics is more sensitive for frontal sources than rear sources.