The objective of this investigation is to formulate and carry out the topology optimization of a magnetic system consisting of permanent magnets and yokes. Earlier investigations on magnetic field topology optimization have been limited on the design optimization of yokes or permanent magnets alone. After giving the motivation for the simultaneous design of permanent magnets and yokes, we develop the topology optimization formulation of the coupled system by extending the technique used in structural problems. In the present development, we will also examine the effects of the functional form for permeability penalization on the optimized topology.