Pass-by noises of various trains were recorded by binaural recorder. The measured noises were evaluated using a sound pressure level and psychoacoustic metrics objectively. To analyze a binaural effect on annoyance that can be caused by train noise, auditory experiments were performed. Through the experiments, the annoyance differences between monaural and binaural train noises, the annoyance differences due to measurement angle and the annoyance differences due to kinds of train and measurement distance were determined. A correlation analysis was conducted to determine which index can affect the annoyance more strongly. As the result, a maximum rate of change of sharpness difference between left and right channels was proposed to predict the annoyance.