Hi,
I'm not sure about your case but when I have similar problems I find I need to make adjustments to my weightings. I spend quite a bit of time here making sure my ground control is satisfactory before forming the dense cloud.
Go to the Ground Control Settings dialog box in the Ground Control Pane to adjust your weighting.
I uncheck camera positions and rely on ground control. All ground control points should be checked.
Then optimise. Note that when you optimise you will loose your dense cloud.
My typical settings are 10m, 0.005m, 0.005m 1pix, 2pix. Note that on the last option Agisoft recommends leaving this at 4pix.
This should give you a + and - range for your errors.
If errors remain large also check for errors in each ground control points.