Hello Tim,
I suggest to check the errors shows after initial alignment without marker utilization: align cameras, place markers (or import from the XML), uncheck cameras in the Reference pane and press Update button on the pane's toolbar.
Then look at the markers that have highest errors in meters and pixels.