Hello crow82,
In Height Field mode PhotoScan will generated 2.5D surface that can be described as Z = f(X, Y), where XY plane is parallel to the red-colored side of the bounding box. The interpolation in this mode will create interpolated geometry in the field of view of at least one camera.
I think you should use Arbitrary option in the Build Mesh dialog.