The growth orientation and the surface roughness of AIN/Si(111) films grown by radio frequency (RF) reactive magnetron sputtering were investigated using in-situ x-ray scattering technique and atomic force microscopy (AFM). AIN films were initially grown with the <001> preferred growth orientation under most growth conditions. As the film gets thicker, however, the growth orientation changes significantly, especially at high substrate temperature and high RF powers. We attribute the observed behavior to the competition between the surface energy that prefers the <001> growth orientation and the strain energy that randomizes the growth orientation. In addition, we investigated the evolution of the surface morphology during the growth using the x-ray reflectivity measurement.