To enhance the machinability of the austenite stainless alloys, Mn and S were added to form MnS in the austenite matrix. Recently, Ca is also added to increase machinability. The alloying elements, such as C, Mn, S, Ca, and Al, are known to affect machinability, but those roles are not well understood. In this study, the ingots, controlled of alloying elements, C, Mn, S, Ca, Al, were prepared in the 304 stainless steel. The relationship between microstructure and machinability was compared to understand the role of alloying elements. It was proved that Mn and S enhanced machinability but C reduced it by analyzing cutting force on machining in the lathe. The alloying elements, Ca and Al, made a complex oxide compound of Mn-S-Ca-Al-Si-O, which results in increasing tool life. The ferrite volume fraction was changed with alloying elements and the effect of the ferrite fraction on machining was also discussed.