Technical: Optimisation of Merged Chunks

[ThomasVD]

Dear Pros and Developers!

I have a technical question which I have not seen discussed here but which I feel can be very important to more advanced projects. 

In the optimisation of a 'normal' (not merged) chunk, PhotoScan uses whatever constraints it has (tie points, known distances, known coordinates, pre-calibrated camera parameters, ...) to optimize the sparse point cloud, camera position / orientation and camera calibration. The 'weight' of each of these constraints can be influenced by adjusting the Reference Settings pane. By removing certain constraints (such as bad tie points) the alignment result is optimised (a new "best-fit" model is calculated by bundle adjustment) and thereby the overall accuracy is hopefully increased.
In the case of a single chunk, all of these constraints are "linked", since all images have been compared and the tie points (generally the most important constraint in modern photogrammetry) form a single interlinked whole.

So far so good.

My question is: How does optimisation in a merged chunk work, and is this different depending on which chunk alignment method was used?

• With point based chunk alignment the tie points of both chunks are compared to one another again (time consuming). If two chunks that have been aligned based on point based chunk alignment are merged, are matches between images from different chunks considered as valid matches between those images during the optimisation of the merged chunk?
• With marker based chunk alignment the tie points from both chunks have not been compared to each other, so presumably after merging both chunks, there are no tie points between images from the different chunks, so these can't be taken into account as valid matches between images during optimisation? Are the two merged chunks then essentially optimised independently from each other (only taking non-matching feature-constraints into account for the optimisation as a whole)? Are the markers that were used to align the models considered "valid matches" between the images from both datasets (or is this perhaps only the case if you choose "merge markers" option during chunk merging?)
• With camera based chunk alignment are the tie points of matching cameras merged? As an example: chunk 1 has valid matches between photo A and photo B, whereas chunk 2 has valid matches between photo B and photo C (photo A is not included in chunk 2). If I then perform camera based chunk alignment based on overlapping photo B, will the matches of photo B be "merged" in the merged chunk? Do I now have a single photo B which has matches in both photo A (from chunk 1) and photo C (from chunk 2). In this case the camera based chunk alignment would allow to again create a single inter-linked network of tie points, but I think this is not the case in PhotoScan since after merging chunks from camera-based chunk alignment the duplicate photos are also duplicated in the merged chunk. Is there a possibility to "merge" these camera positions into a single picture with tie points stretching across photos A, B and C?

If anyone has answers to these questions, or simply wants to contribute their thoughts on this matter, I think that'd be really useful!

It has important implications for questions such as: Which chunk alignment method allows for best optimisation? When precisely would be best to perform optimisation (before merging, after merging, ..)? Is good optimisation only possible if all images are aligned together (not working with chunks)?

Cheers!

[Yoann Courtois]

Hello Thomas !

Your topic looks important for me too, and while we are waiting for more detailed answers from Alexey, I can already share what I know:

As you said, in point based alignment, tie points are generated between the chunks you want to align. However, all those points are totally deleted after calculation so obviously you cannot find them again after merging.
The direct consequence (that nothing is kept after chunk alignment) is if you "update" the sub-chunk aligned on the main one, you lose everything and you need to align the chunks again. Very annoying thing by the way...
Moreover, this point based align chunk tool is a deep black box. And you nearly cannot have an overview of the accuracy of this alignment. I got some projects where around 100 000 matches were found, but finally chunks were aligned based on 1 000 points. Mean point error given was around 3 pixels (4mm/pixel give an expected centimeter accuracy) and I finally got more than 8cm when I handled some manual measurements between chunks...  that makes me disappointed...

For both other alignments, no tie points are used in any way. So no more points are created.
In marker based alignment, I guess only a 3D affine transformation (3 translations, 3 rotations, 1 scale factor) is used to put a chunk onto the referenced one.

To summarize, I guess only point based alignment should be used to get the best accuracy. But for the moment there is a lack is the process. Otherwise it could be gruesomely useful !

Hope to get more here anyway !

[Alexey Pasumansky]

Hello Thomas,

The method of the chunk alignment operation doesn't make any difference, it's only a tool that orients chunks relatively to each other, no new tie points are introduced and no additional connection between the chunks is created.
Almost the only way to "connect"  the merged chunks before the optimization is to use markers that have projections on the images from both subsets (for example, if you use Merge Markers) option. So unless you are merging the markers, the subsets would be optimized independently in the merged chunk.

[ThomasVD]

Thank you Yoann and Alexey,

So just trying to wrap my head around the implications - please correct me if I'm wrong:

1. If working with multiple chunks, it's best to simply optimise each chunk individually (this gives you more control over the optimisation parameters for that specific part of the dataset), afterwards merge the chunks (and perform no more optimisation of the merged chunk). There is no advantage to optimising a merged chunk vs optimising the individual chunks, since the image sets from different chunks aren't "interlinked" after merging.

Exception: if using "marker-based chunk alignment" the chunks in question are interlinked, since the markers are considered as valid matches between images from the different chunks, but only when the "merge markers" option is chosen in the merge chunks dialogue.
Question: in this case is there a difference between marker types (green flags, blue flags and grey flags) in the photos during optimisation procedure (ie do the different flags for instance get a different weight)?

2. The only way to get a truly interlinked alignment result for optimisation, is to align all images in a single chunk.

Question: say I have a large dataset, and my cameras have no known coordinates - would it in this case be a good idea to align the images in different chunks, then align those chunks, export the approximate camera coordinates as reference, create a new chunk with all images in a single chunk, import the camera coordinates and re-align all images in a single chunk using "pair-preselection: reference"? In that case you would save on alignment time since we now have reference camera coordinates + all images would be interlinked in a single alignment result.

3. Is there a good reason why the chunk alignment results are not merged in "point-based chunk alignment" or "camera-based chunk alignment"?

In case of marker-based chunk alignment you already have the option to choose "merge markers" in merge chunks dialogue, thereby interlinking the chunks. Couldn't you just as easily: 
=> In the case of point-based chunk alignment have an option to choose "merge tie points" in merge chunks dialogue, since tie points have been compared between chunks?
=> In the case of camera-based chunk alignment (my personal preference) have an option to choose "merge cameras" in merge chunk dialogue, thereby avoiding duplicate cameras in the merged chunk AND connecting the tie points from photo A to photo B to photo C (see my example in original post)?

Wouldn't this allow working in different chunks (thereby significantly reducing alignment time) AND good optimisation of a merged chunk as a whole by essentially linking tie points from different chunks? Or am I missing something?

Cheers,

Tom

[Yoann Courtois]

Thank you for going deeply on this way.

1. Indeed you don't need to optimize the merged chunk afterward, to my mind it could only create problems ^^
About flags, I would like to have an answer from Alexey. Actually for me it's clear for green and grey flags: Green flags are "considered" points whereas grey flags are not, i.e. there appear just to suggest you "Hey it may be a point there !" but they are not "attached" to the picture.
But I have no idea about those blue flags which appeared recently. It's seems they are "fixed" (in contrary with grey ones) on pictures based on points you have clic on others. For me those blue ones has just created fear to me because I don't see their utility. Maybe Alexey explain us what is their aim.

2. Humm looks complicate ! I guess you should minimize as well as possible the use of chunks alignement because no matter what kind of alignment you use, it's a deep black box and you cannot adjust anything afterward. For large dataset, use a single chunk with "generic preselection" which optimize your calculation.

3. Every kind of possibility to keep the tie point cloud resulting from chunk alignment is a f***ing good idea ! Please Alexey, can you take this into consideration as a "request feature" for the future ? It would be nice in many cases.

Regards

[JMR]

very interesting questions here, indeed!

from my experience the use of chunks helps to face three problems.
1. The project is simply too big for my computer
2. the survey was conducted also in chunks, with time lapses or space gaps in between, changing lens setting, and that were not necessarily interlinked.
3. the partition serves for better management at the output stage.

In every case I rather prefer to work in one chunk at least during alignment and optimization. I think Photoscan is pretty capable of handling huge alignment tasks and you can create chunks afterwards before the more resource demanding processes (dense cloud and meshing). You can split manually using duplication and partial deletion or use a smarter way by means of python. I must say I always get frustrated when our Alexey resumes something saying, "this can be done with Python...". (I suspect he smiles like an evil swot when he adds " so easily"... Yes it is great to have the py api, but not everyone knows how to use it, and there are dozens of simple but terribly useful things that should already be among menu items, and "Split in chunks" is one of them.

But even in the case that this is not convenient or feasible one chunk from the begining and one ends having several chunks, I usually merge them at the end after chunk alignment (no matter what method) and re-align the merged chunk. I'm almost sure that Photoscan takes the camera orientations in the merged chunk as initials so there is no need to export and create a new chunk from scratch, but on the contrary one can build the sparse cloud and optimize directly.

One goal of this procedure is to keep cameras grouped (calibration wise) as they were originally and that might be the right path in some cases. i.e. when chunks were photographed in diferent sesions with the same camera but eventually with diferent focus settings. (to create a chunk from blank might lead Pscan to put all photos in the same basket)

I think that there must be a reason to not keep (merge) tie points created during chunk to chunk alignment. My guess is that the quality of the feature point matches ch2ch cannot be evaluated in the same way as is done inside a chunk where they link photos-to-photos and their residuals can be miminized by moving photos relatively. On the contrary, during ch2ch alignment, the relative orientations between photos inside a chunk must be kept while the entire block moves during iterations to mimimize the global residual for all at once. After ch2ch alignment one tie point could have a very big reprojection error due to the rigidnes of the two structures it belongs to... well this is just my theory.

Cheers, GEOBIT

[Yoann Courtois]

Thanks for your interesting answer JMR.

I just want to react to the last part of your message, about keeping ch2ch alignment tie point cloud.
Indeed the reprojection error of ch2ch alignment should be quite big as it's an alignment of 2 rigid blocks.
But I have made many tests and it's actually not so bigger than reprojection errors inside both chunks.
For example, the last 2ch2 alignment I've made give me a reprojection error of 2.74 (I guess it's in pixels unit). And when I look after generated report for the 2 chunks, it's given 1.5 pxl (main chunk) and 2.94 pxl (sub-chunk). So the ch2ch alignment error looks not unbelievable so far !

Also, from my point of view, the tie point cloud is not essential. Compulsory results I would need are detailed errors as we can get from a single chunk alignment. Because I handled some ch2ch alignment where the reprojection error looked good (3 pxl => 1.2 cm) but manual check points measurement gave me outstanding errors (10cm)
I would ask more to know precisely:
- how many points were used to align ch2ch
- where are located those points (a density map could be enough over a first phase)
- what does the reprojection error -really- refer to
- basically how accurate the ch2ch alignment is, how much faith I can give to this tool...

Regards

[ThomasVD]

Thank you for your comments Geobit and Yoann,

Just a few thoughts, perhaps it's easiest if I use quotes: 

I must say I always get frustrated when our Alexey resumes something saying, "this can be done with Python...". (I suspect he smiles like an evil swot when he adds " so easily"... Yes it is great to have the py api, but not everyone knows how to use it, and there are dozens of simple but terribly useful things that should already be among menu items, and "Split in chunks" is one of them.

- I agree some things could just be integrated into the software, rather than having to rely on Python. For people who don't know Python (like me), remember there is also this library with scripts, including a "split in chunks" script: http://wiki.agisoft.com/wiki/Python I know the scripts didn't work anymore the day after PS 1.3 was released, but perhaps they've now been updated.

I usually merge them at the end after chunk alignment (no matter what method) and re-align the merged chunk. I'm almost sure that Photoscan takes the camera orientations in the merged chunk as initials so there is no need to export and create a new chunk from scratch, but on the contrary one can build the sparse cloud and optimize directly.

- I don't think this is in fact the case. I think if you re-run "align images" in a merged chunk, PhotoScan simply starts from scratch (perhaps unless you use pre-selection: reference) and re-calculates everything, ignoring the previously calculated camera orientations.

One goal of this procedure is to keep cameras grouped (calibration wise) as they were originally and that might be the right path in some cases. i.e. when chunks were photographed in diferent sesions with the same camera but eventually with diferent focus settings. (to create a chunk from blank might lead Pscan to put all photos in the same basket)

- Remember that in such cases you can also simply work in a single chunk, and then group photographs taken in different sessions in different camera calibration groups, using the camera calibration groups on the left of the Tools -> Camera Calibration window.

I think that there must be a reason to not keep (merge) tie points created during chunk to chunk alignment. My guess is that the quality of the feature point matches ch2ch cannot be evaluated in the same way as is done inside a chunk where they link photos-to-photos and their residuals can be miminized by moving photos relatively. On the contrary, during ch2ch alignment, the relative orientations between photos inside a chunk must be kept while the entire block moves during iterations to mimimize the global residual for all at once. After ch2ch alignment one tie point could have a very big reprojection error due to the rigidnes of the two structures it belongs to... well this is just my theory.

- This might be true. Even then though, in case they can't be compared in the same way, surely there is some way to interlink everything, and perhaps simply give less weight to tie points matched during chunk alignment. In case of camera-based chunk alignment though, if both chunks have been aligned using the same image alignment settings, I would think you could almost directly tranfser tie points from photo A to B to C with the same weight. But of course I don't know what the magic behind the scenes in the PS algorithms is, so this is something Alexey would have to comment on

Cheers,

Thomas