Shape From Focus (SFF) is a passive optical technique to recover 3D structure of an object that utilizes focus information from 2D images of the object taken at different focus levels. Mostly, SFF methods use a single focus measure to compute image focus quality of each pixel in the image sequence. However, it is difficult to recover accurate 3D shape using a single focus measure, as different focus measures perform differently in diverse conditions. In this paper, a nonlinear Total Variation (TV) based approach is proposed for 3D shape recovery. To improve the result of surface reconstruction, several initial depth maps are obtained using different focus measures and the resultant 3D shape is obtained by diffusing them through TV. The proposed method is tested and evaluated by using image sequences of synthetic and real objects. The results and comparative analysis demonstrate the effectiveness of our method.