Hello,
did you also use ground control points? Which drone do you use?
I have had similar issues. In some cases I was aware about the reason, in some other cases not really.
One reason I found out was related to the geo-referencing of the images. Due to missing geo-tags (already one is enough) the tagging of the photos using the software of my drone company interpolated the geo value for all other photos, so there was a slight shift, which however led to shifts or slight rotations of the model. But this was more than 1 year ago and the geo-tagging process was improved.
Nevertheless, in my last work, I see also a shift and slight rotation of the final model, although the error should be smaller according to the results of PS. I'm not able to understand this and I will write to my drone company to ask about possible connections to the GPS values of the drone.
I'll let you know if I find something out.