The main role of an acoustic diffuser is to diffuse reflected sound field spatially. Since the pioneering work of Schroeder, there have been investigations to improve its performance by using shape/sizing optimization methods. In this paper, a gradient-based topology optimization algorithm is newly presented to find the optimal distribution of reflecting materials for maximizing diffuser performance. Time-harmonic acoustic analysis in a two-dimensional acoustic domain is carried out where the domain is discretized by finite elements. Perfectly matched layers are placed to surround the domain to simulate non-reflecting boundary conditions. Design variables are assigned to each element of which material properties are interpolated between those of air and those of a rigid body. An approach to extract the reflected field from the total acoustic field is employed. To validate the effectiveness of the proposed method, design problems are solved at different frequencies. The performance of the optimized diffusers obtained by the proposed method is compared against that of the conventional Schroeder diffusers.