Hello Hans,
If the camera orientation is saved in EXIF tag only (i.e. the rotation is not actually applied to the images - all have same width-height, where width>height), then you should keep all the images in the same calibration group.
But if the camera is rotating the images instead of EXIF tag usage, then it is better to split the groups for each rotation (0, 90, 180, 270 degrees). Moreover, you would not be able to merge to the same group images with different resolution/proportions (like 2000 x 1000 and 1000 x 2000).