I've often found this to be the case too. I believe this is to do with the scene texture and the length scales of that texture. Using lower accuracy settings forces the software to only use much "larger" features in the image, which may be more stable if the small scale features are very uniform.
One way I get round this is to run the model at low accuracy, and georeference it and save the estimated camera locations and orientations. Then I re-run at high resolution having imported the prior orientations as the initial parameters (with control based pre-selection).
However, if "low" accuracy alignment is sufficient for your project, I'd probably just stick with that.