This paper presents adaptive neuro-fuzzy inference based defect detection method for various defect types, such as micro-crack, fingerprint and contamination, in heterogeneously textured surface of polycrystalline solar wafers. Polycrystalline solar wafer consists of various crystals so the surface of solar wafer shows heterogeneously textures. Because of this property the visual inspection of defects is very difficult. In the proposed method, we use local binary feature and fuzzy reasoning for defect detection. Experimental results show that our proposed method achieves a detection rate of 80%~100%, a missing rate of 0%~20% and an over detection (overkill) rate of 9%~21%.