Actually, if you are confident in the coordinate information used for the camera centers, please specify, whether the orientation angels are loaded as well (yaw, pitch and roll)? Maybe they are not accurately measured, so I can suggest to use large values for Camera Accuracy (deg) parameter in the Reference pane settings dialog, thus lowering the weight of the orientation angles in the georeferencing process.