Adrian,
The only time I've ever seen anything like this was when I tried "Camera Based" to align 2 chunks (see attached)
If that's not what you're doing, you might try to decrease the "Accuracy" value for your images prior to alignment to get Photoscan to have a little more faith in the EXIF GPS values.
I normally run my Phantom Pro 3 images at "1", but I include the heading (yaw) at the same time.
It looks from the un-aligned cameras in the image you sent that you're running a straight orthogonal grid pattern. If so, it's pretty simple to create an import file by hand that assigns the accuracy value and heading at the same time.
As with most Photoscan problems, if you go back and fly that same area with a closer grid pattern and a slower speed to increase the overlap, you'll probably get better results.
Hank
Texas