This paper presents an optimum design of a rotor-bearing spindle system for a ultra centrifuge (80,000 RPM) supported by ball bearings with nonlinear stiffness characteristics. To obtain the nonlinear bearing stiffnesses, a ball bearing is modeled in five degrees of freedom and is analyzed quasi-statically. The dynamic behaviors of the nonlinear rotor-bearing system are analyzed by using a transfer-matrix method iteratively. For optimization. we use the cost function that simultaneously minimizes the weight of a rotor and maximizes the separation margins to yield the critical speeds as far from the operating speed as possible. Augmented Lagrange Multiplier (ALM) method is employed for the nonlinear optimization problem. The result shows that the rotor-bearing spindle system is optimized to obtain 9.5％ weight reduction and 21％ separation margin.