When you first align your images, tie points are created between pairs of images. These points are what defines and constrains the alignment in the absence of any other data. When you merge chunks, the chunks are aligned to each other based on whichever method of alignment you use, but no more tie points are created so there is nothing holding the chunks together if you subsequently add more information to the system, such as ground control points. Adding ground control points to create a simple coordinate system and scale (when you just click the update button in ground control pane) affects the chunk as a whole, but optimise rejiggles the cameras individually using the new ground control data, but still taking into amount the original tie points. The tie points constrain the cameras in about the same place as they were, but since there are no tie points between images in different chunks it is possible for chunks to move significantly relative to each other, but this should only happen if the chunkalignment is bad to start with, or the ground control is bad. So as wishgranter says, it would help to see images of what the subject is, how the chunk alignment looks and how your ground control is distributed etc.