Image analysis algorithm for the quality evaluation of ginseng seedling was investigated. The images of ginseng seedling were acquired with a color CCD camera and processed with the image analysis methods, such as binary conversion, labeling, and thinning. The processed images were used to calculate the length and weight of ginseng seedlings. The length and weight of the samples could be predicted with standard errors of 0.343 mm, and 0.0214 g respectively, $R^2$ values of 0.8738 and 0.9835 respectively. For the evaluation of the three quality grades of Gab, Eul, and abnormal ginseng seedlings, features from the processed images were extracted. The features combined with the ratio of the lengths and areas of the ginseng seedlings efficiently differentiate the abnormal shapes from the normal ones of the samples. The grade levels were evaluated with an efficient pattern recognition method of support vector machine analysis. The quality grade of ginseng seedling could be evaluated with an accuracy of 95% and 97% for training and validation, respectively. The result indicates that color image analysis with support vector machine algorithm has good potential to be used for the development of an automatic sorting system for ginseng seedling.