Hello ignarec,
How you were performing the chunk alignment?
In case you have used point based alignment option - then you should have applied masks to all the images from both sub-sets covering the static background.
However, if you do so, then it may be reasonable to align all the images in the same chunk (using Constrain Features by Mask option) to avoid the possible issues related to the inaccurate chunk alignment.