Oil, hysteretic and inertial mass dampers are representatives of passive dampers used for smart enhancement of seismic performance of building structures. Since oil dampers have a nonlinear relief mechanism and hysteretic dampers possess nonlinear restoring-force characteristics, several difficulties arise in the evaluation of buildings including such dampers. The purpose of this paper is to propose a practical method for simultaneous optimal use of such dampers. The optimum design problem is formulated so as to minimize the maximum interstory drift under design earthquakes in terms of a set of damper quantities subject to an equality constraint on the total cost of dampers. The proposed method to solve the optimum design problem is a successive procedure which consists of two steps. The first step is a sensitivity analysis by using nonlinear time-history response analyses, and the second step is a modification of the set of damper quantities based upon the sensitivity analysis. Numerical examples are conducted to demonstrate the effectiveness and validity of the proposed design method.