GCP data negative impact on accuracy?

[LTP Drones]

Hi guys,

please forgive me being a noob, I have flown a couple of M3E RTK missions using a rover and base GNSS with NTRIP.  When I look at my tolerances, it would seem to my noob eyes that the RTKimage allignment results are quite accurate.  But when I try to incorporate the 2 GCP's on this 1 hectare job.  Accuracy is greatly effected.
When I see the control points on the 3d model they are precisely in the right position when disabled, but if they're enabled everything goes wrong.
I have matched both rtk images  and gcps to show GDA2020/AMG xone 50 (EPSG:20350)
This is the one fly in my ointment at the moment, and it's doing my head in, can anyone point me in the right direction, I can only guess the GCP CSV point surveys are wrong?

[dpitman]

When I see the control points on the 3d model they are precisely in the right position when disabled, but if they're enabled everything goes wrong.
....
I have matched both rtk images  and gcps to show GDA2020/AMG xone 50 (EPSG:20350)

By that description, you are seeing an offset due to an incorrect CRS for one set or the other.