This paper presents a useful and simple method to design the passive skimming system for homogeneous permanent magnets based on numerical optimization. To simulate the effects of manufacturing and assembling tolerances, the actual geometrical parameter of the magnet with a derivation is suggested. Then, the optimal design model oi a passive shim system is set up to correct the derivative of field homogeneity. The numbers, sizes and locations of the passive shims are optimized by the steepest descent algorithm combined with design sensitivity analysis. Two implementations show that the proposed method can achieve the required homogeneity of the field with the minimum quantity of ferromagnetics.