Analyses of flow and heat transfer characteristics and shape optimization of internally finned circular tubes have been performed for three-dimensional periodically fully developed turbulent flow and heat transfer. CFD and mathematical optimization are coupled in order to optimize the shape of heat exchanger. The design variables such as fin widths $(d_{1},;d_{2})$ and fin height (h) are numerically optimized by minimizing the pressure loss and maximizing the heat transfer rate for limiting conditions of $d_{1}=0.2~1.5;mm,;d_{2}=0.2~1.5;mm,$ and $h=0.2~1.5mm$. Due to the periodic boundary conditions along main flow direction, the three layers of meshes are considered. The flow and thermal fields are predicted using the finite volume method and the optimization is carried out by means of the sequential quadratic programming (SQP) method which is widely used in the constrained nonlinear optimization problem.