The tree topology in multicast systems has high transmission efficiency, low latency, but poor resilience to node failures. In our work, some nodes are selected as backbone nodes to construct a tree-like core overlay. Backbone nodes are reliable enough and have strong upload capacity as well, which is helpful to overcome the shortcomings of tree topology. The core overlay is organized into a spanning tree while the whole overlay is of mesh-like topology. This paper focuses on improving the performance of the application-layer multicast overlay by optimizing the core overlay which is periodically adjusted with the proposed optimization algorithm. Our approach is to construct the overlay tree based on the out-degree weighted reliability where the reliability of a node is weighted by its upload bandwidth (out-degree). There is no illegal solution during the evolution which ensures the evolution efficiency. Simulation results show that the proposed approach greatly enhances the reliability of the tree-like core overlay systems and achieves shorter delay simultaneously. Its reliability performance is better than the reliability-first algorithm and its delay is very close to that of the degree-first algorithm. The complexity of the proposed algorithm is acceptable for application. Therefore the proposed approach is efficient for the topology optimization of a real multicast overlay.