Investigating Alignment Parameters

[IceDreamer]

Greetings

My workplace have started capturing drone footage for 3D reconstruction of cliffs etc (We're a geology company) and I've been investigating the various settings and trying to optimise workflow and hardware. I seek clarification on a simple question: Higher is better, or lower is better?

RMS Reprojection Error - I am assuming that lower is better here

Max Reprojection Error - Again, lower is better?

Mean Key Point Size - I think I understand what this is, but am completely unsure which way is better in general. I've been running the same image set over and over with various settings and recording the outputs. Logic dictates that smaller would be better here, because it means your feature is more in 'focus'?

Effective Overlap - Really not sure here. Again, logically I think this should be higher is better, but it's hard to say because the various settings are giving answers that don't quite tally.



Can anyone help out?

[IceDreamer]

So then, going by the response, 0% of Photoscan users know what the parameters they're using actually do. That's reassuring...

[SAV]

Hi IceDreamer,

Must have missed your post. Sorry about that.

You should always aim for a SMALL REPROJECTION ERROR, because it will tell you of how well the 3D coordinates of computed points are matching the points on the associated images. This value should be around or below 1 (pixel) for high quality photogrammetric models.

Regarding Mean Key Point Size. For a point described by a series of key points, its projection accuracy is computed as the mean key point size. Consequently, large mean key point sizes are associated with lower accuracies.

Regarding Effective Overlap, a higher value is better. It’s called effective because only high quality key points  are considered, and not image areas.

Regards,
SAV

[IceDreamer]

Ah, thank you. That all makes sense and tallies logically. The results I have been getting from running benchmarks, however, have thrown very interesting results. EDIT: The right hand of each image is cut off... Grey is "Medium", blue is "High" and orange is "Highest".







These tests were done by running the same 495 image set, with masks, through image alignment while changing the settings. The Key Point number is 10 times the Tie Point Limit in all cases.

If we take it that both low RMS reprojection error AND low key point size are desired and indicative of greater accuracy, then this data makes very little sense, as the lowest accuracy setting gives the best result for RMS Error, and the highest the best result for Key Point Size. Which makes it very difficult to decide which setting is actually giving the best output! Just to make things messy, the best overall setting for Overlap is the third option!

Anyone got any ideas about how this is happening?

[SAV]

Hi IceDreamer,

Thanks for sharing your benchmark results. Very interesting.

Older posts in this forum have mentioned that a key point limit larger than 40,000 and a tie point limit of 4,000 does not add much to the overall accuracy. Your results seem to match these observations (curve flattens at ~4,000 tie points). Note that these optimum key point and tie point values might change if you use photographs with very high resolutions (larger than 36 Megapixel). What was the resolution of your images?

Regarding the accuracy settings. If you choose HIGH, Photoscan is using all images at FULL RESOLUTION, when you choose MEDIUM it is using photos at 50% and when choose HIGHEST it is actually 'upscaling' your images to 200%. That's why I generally don't use HIGHEST for Photo Alignment because I am not really sure what this 'upscaling' process does to the project (as somehow confirmed by your results). HIGHEST might be helpful for very low resolution (less than 2 Megapixel) imagery, where otherwise SIFT wouldn't be able to identify enough key points (some multispectral cameras have quite low resolutions, for example).
Therefore, it might be a quite difficult to properly 'interpret' your orange line.

Ignoring the orange line, let's have a look again at your graphs.

Mean key point size for HIGH is lower than for MEDIUM. Makes sense. 

RMS projection error seems to be better (lower value) for MEDIUM than for HIGH, but if you consider that we are already talking about a sub-pixel error (~0.20 vs ~0.35 for 4,000 tie points) for images where the smallest unit is 1 pixel, then I think both of them are very good. 

Another check if your overall accuracy between HIGH and MEDIUM is changing significantly would be to use (properly surveyed) markers (check points & control points). Assess their errors.

In case you haven't already done so, run OPTIMISATION after Photo Alignment and after removing points that show a a reprojection error higher than 1 (EDIT > GRADUAL SELECTION). I think this will further improve your accuracy.

Regards,
SAV

[IceDreamer]

Yes it's interesting huh.

Regarding Highest, it almost certainly uses a bicubic interpolation to double the image size. While this will create some 'false' data, the majority of generated data will lend additional accuracy and precision in that area, and make detecting points in particular easier.

It's a 12MP, 4000*3000 camera from the Mavic Pro drone. So not particularly high resolution in my opinion. Optimisation has been performed in all test cases.

The biggest curiosity is that common wisdom suggests that a small key point size is associated with a low reprojection error, yet the data does not hold this to be true.

[SAV]

Hi IceDreamer,

DJI Mavic Pro has a pretty small sensor (1/2.3 inch, 28 mm²), hence the signal-to-noise ratio is not great. I wonder how much this attributes to the general outcome of your test. Would be interesting to do the same test using a Phantom 4 Pro which has a much larger sensor (1 inch, 116 mm² = 4 times the area).

Regards,
SAV

[Arie]

Additonally, wouldn't the electronic shutter also influence the results regarding the reprojection error (i.e. rolling shutter effect)?

[SAV]

Yep. Electronic shutter can be an issue, especially if the UAV is moving fast and/or close to the object. I agree Arie, some of IceDreamer's 'strange' results could be related to it.

Phantom 4 Pro uses a mechanical shutter for still photography (and has a much bigger sensor, as already mentioned in my previous post), hence it is a much better choice for UAV photogrammetry.

Regards,
SAV

Additonally, wouldn't the electronic shutter also influence the results regarding the reprojection error (i.e. rolling shutter effect)?