As already mentioned, oblique and rotated images add a lot to a more solid bundle adjustment, so certainly give that a try.
But it may also really help to calibrate your Gopros in a more controlled "lab-setup" (this doesn't necessaraly mean photos of a calibration pattern).
I've recently used a few GoPros and their lens distortions are very "challenging", but the cams have their use, just like in your project.
The advantage for pre-calibrating Gopros could be that they have a fixed focus, so this parameter will not change.
So you could just try importing the calibration (individually per camera, not just generic per cam model). Of course their will still be some variations in the field (or below water

due to temperature differences, general calibration tolerences etc...
Attached a downsampled pic of the UV-distortion map for a Gopro Hero 10: pretty crazy, especially the corners.