While creating models of small surface area patches of ground, I would occasionally run into vertical spikes or misaligned planes off angle from the actual ground surface.
To fix this issue I looked at camera position estimates and disabled cameras which where progressively closer to the surface/ object being scanned and re ran alignment.
Hopefully this will help someone, as I could not find a fix during my surface level googling.