In this paper, we studied the use of the foreground and background features and SVM classifier to improve the accuracy of offline handwritten numeral recognition. The foreground features are two directional features: directional gradient feature by Kirsch operators and directional stroke feature by projection runlength, and the background feature is concavity feature which is extracted from the convex hull of the numeral, where concavity feature functions as complement to the directional features. During classification of the numeral, these three features are combined to obtain good discrimination power. The efficiency of our feature sets was tested by recognition experiments on the handwritten numeral database CENPARMI, where we used SVM with RBF kernel as a classifier. The experimental results showed that each combination of two or three features gave a better performance than a single feature. This means that each single feature works with a different discriminating power and cooperates with other features to enhance the recognition accuracy. By using the composite feature of the three features, we achieved a recognition rate of 98.90%.