In order to measure fine rotation angles by magneto-optic effects of magneto-optical recording thin films, a polarization modulation method is used. In the experiment, the polarization of laser (He-Ne laser) beam is modulated by a Faraday rotator and the amplified modulated signals are selectively detected by phase sensitive detector. The magnetic Kerr rotation and Faraday rotation hysteresis loops are investigated by this method for thermally evaporated amorphous TbFeCo thin films and RF sputtered garnet thin films. Rotation angles about $0.25^{circ}$ are measured easily from TaFeCo thin films. In the case of longitudinal Kerr rotation, very small rotation angle of $2.5 imes10^{-3^circ}$ is measured with good accuracy of the measurement (about $1 imes10^{-3^circ}$). And it is found that each thin films have the hysteresis curves of high coercivity and good squareness.