Reverse engineering of reliefs aims to turn an existing relief superimposed on an underlying surface into a geometric model which may be applied to a different base surface. Steps in this process include segmenting the relief from the background, and describing it as an offset height field relative to the underlying surface. We have previously considered relief segmentation using a geometric snake. Here, we show how to use this initial segmentation to estimate the background surface lying under the relief, which can be used (i) to refine the segmentation and (ii) to express the relief as an offset field. Our approach fits a B-spline surface patch to the measured background data surrounding the relief, while tension terms ensure this background surface smoothly continues underneath the relief where there are no measured background data points to fit. After making an initial estimate of relief offset height everywhere within the patch, we use a support vector machine to refine the segmentation. Tests demonstrate that this approach can accurately model the background surface where it underlies the relief, providing more accurate segmentation, as well as relief height field estimation. In particular, this approach provides significant improvements for relief concavities with narrow mouths and can segment reliefs with small internal holes.