In this paper band-stop filters that reject the undesired 3th and/or 5th harmonics generated from high frequency generator are analyzed using equivalent circuit concept, and optimized using iterative optimization algorithm. The length of the band-stop filter is setup initially a quarter of a wavelength of the harmonic, but initial filter structure is exactly unable to cutoff the harmonics because of the fringing effect at the discontinuous boundary. To reduce the mismatch between the cutoff frequency and the harmonics, iterative LMA(Levenberg-Marquardt Algorithm) is introduced and band-stop filters in coaxial line are optimized.