And have you checked on "Constrain features by mask" option in the Align Photos dialog?
Perhaps the 4 sides of the tower look too similar.