It sounds like you're making an error in the proccessing, there should be no issues like you're describing. Ignore Alex's post above, he's trying to sell you something. PPK is not needed and Agisoft does a great job.
I know the documentation isn't very good a few questions though:
Have you disabled the photos coordinates but unchecking them in the reference panel? Do a low quality alignment, create mesh, pick control points, uncheck cameras, optimize and then do higher quality alignment and build. The issue may be the assigned default accuracy is still to optimistic and Agisoft of holding that weighting in the final adjustment
Also, you will see an error due to you working on orthometric height and the planes capturing either AGL or ellipsoid heights. Pay close attention to the coordinates in the reference panel.