Hello Erik,
Constrain features by mask option should be checked on, otherwise the masked areas would be also taken into consideration during the feature point detection.
If the alignment fails even when the mask option is turned on, please send the new PSZ project with the alignment results to support@agisoft.com.