Integrated Starter Generator (ISG) system improves the fuel economy of hybrid electric vehicles by using idle stop and go function, and regenerative braking system. To obtain the high performance and durability of ISG motor under continuously high load condition, the motor needs to properly design the cooling system (cooling fan and cooling structure). In this study, we suggested the enhanced design by modifying the thermal design of the ISG motor and then analyzed the improvement of the cooling performance under high-speed condition and generating mode by CFD simulation. The temperatures at the coil and the magnet of the enhanced model were decreased by about $4^{circ}C$ and $6^{circ}C$, respectively, compared to those of the conventional model. Therefore, we verified the cooling performance enhancement of the novel thermal design in the case of core loss increment due to the higher speed condition.