The conformational dynamics of heme pocket, a small vacant site near the binding site of heme proteins -myoglobin (Mb) and hemoglobin (Hb), was investigated after photolysis of carbon monoxide from MbCO and HbCO in D$_2$O solution at 283 K by probing time-resolved vibrational spectra of photolyzed CO. Two absorption bands, arising from CO in the heme pocket, evolve nonexponentially in time. The band at higher energy side blue shifts and broadens with time and the one at lower energy side narrows significantly with a negligible shift. These spectral evolutions are induced by protein conformational changes following photolysis that modify structure and electric field of heme pocket, and ligand dynamics in it. The conformational changes affecting the spectrum of photolyzed CO in heme pocket likely modulates ligand-binding activity.