the difference on Z axis in relation with the values on X,Y is normal in photogrammetry only if it is in a range, this differences belong to many factor as:
1. focal length: short focal length gives lower error on the vertical axis but higher on the plane (X,Y), longer focal length will give better results on the plane but worst on vertical axis
2. terrain or pixel quality/ detail: the results are better on naked terrain with rocks and high contrast nice illumination than in the forest with clouds shades, the sun inclined giving non uniform light
3. quality of the optics, better lenses with the right f or aperture gives sharper images, so better cameras with better lenses and right adjustments produce better results
4. Scale: the scale is a relation of cost / information required, exactitude.
5. processing of the information, I mean the setup of agisoft
Photogrammetry as a measurement method that use a lot of tools, techniques, terrain, etc, gives a precision and exactitude that have to be take into account, from the early stages with the client to the final stages producing the final deliverables. a factor of 1 to 3 is nice in Z axis in relation with X,Y.
As an example, if you made a comparison with LIDAR, the results of photogrammetry are better on x,y, LIDAR is the best on Z but worst on x,y.
so each project has limits on budget, precision, exactitude, time and tolerances with the final deliverables