A convolution with HRTF DB and the original sound is generally used to make the method of sound image localization for virtual speaker realization. But it can decline localization by the confusion between up and down or front and back directions due to the non-individual HRTF depending on each listener. In this paper, we study a virtual speaker using a new HRTF, which is grouping the HRTF around the virtual speaker to improve localization between up and down or front and back directions. To effective HRTF grouping, we decide the location and number of HRTF using informal listening test. A performance test result of virtual speaker using the grouped HRTF shows that the proposed method improves the front-back and up-down sound localization characteristics much better than the conventional methods.