A complex cam shape optimization problem is studied to optimize a unique cam mechanism for a new cam drive engine. First, the optimization problem is defined through analyzing the unique cam mechanism. Multiple design specifications are included in the optimization problem by defining the output torque of the engine as the objective function and the contact stress, radius of curvature, and pressure angle as the constraints. Second, an analytical scenario is designed to find the best ever cam profiles through manipulating the different combinations of cam profile representations and optimization methods. Two types of curve representations, including general polynomial spline and B-spline, are employed in cam profile synthesis. In addition, both a classical optimization technique and a genetic algorithm (GA) based method are applied to solve the complex optimization problem. Finally, comparative studies are performed among the initial profile and the optimal profiles to demonstrate the effectiveness of these proposed design approaches on solving the cam profile optimization problem. Results show that the best profiles are obtained from a combination of the B-spline representation and the GA-based method. In addition, compared to the initial design, the engine performance is improved greatly by the proposed optimization approaches.