The ant colony optimization (ACO) algorithm is a new heuristic algorithm that offers good robustness and searching ability. With in-depth exploration, the ACO algorithm exhibits slow convergence speed, and yields local optimization solutions. Based on analysis of the ACO algorithm and the genetic algorithm, we propose a novel hybrid genetic ant colony optimization (NHGAO) algorithm that integrates multi-population strategy, collaborative strategy, genetic strategy, and ant colony strategy, to avoid the premature phenomenon, dynamically balance the global search ability and local search ability, and accelerate the convergence speed. We select the traveling salesman problem to demonstrate the validity and feasibility of the NHGAO algorithm for solving complex optimization problems. The simulation experiment results show that the proposed NHGAO algorithm can obtain the global optimal solution, achieve self-adaptive control parameters, and avoid the phenomena of stagnation and prematurity.