Clarification of Reconstruction uncertainty

[ozbigben]

I'm working through improving the quality of the dense point cloud and found this helpful explanation.

"Reconstruction uncertainty". In case you have only two cameras and a point is being triangulated by intersection of two rays there is a direction in which the variation for the point position is maximal and another direction with the minimal variation. Dividing one on another (max to min) we've got reconstruction uncertainty value. Mostly this criterion is designed for visualization and estimating the errors. In general this value characterizes the accuracy of positioning points in cloud.

"Reprojection error" demonstrates the accuracy of point positioning and is specified in pixels. We recommend to remove points with huge reprojection errors before optimizing photo alignment.

This has been really useful and is probably worth putting in the wiki.  I'm just trying to properly understand how the numerical value for Reprojection error Reconstruction uncertainty relates to quality (and thus what a practical minimum value would be). Is it relative to the input image size or does it relate to the size of image pattern that PS uses to "recognise" features.

I've managed to remove too many points a couple of times which is handy as I now know what effect that has on the dense point cloud.

[ozbigben]

So far I've found significant improvements in the range of 150-50, although at lower values it might take out too many points causing other problems. If there are significant gaps in the sparse point cloud using 50, this seem to be an indication of not enough images from different positions for those areas, which might make a useful quality check while shooting objects/buildings(?)

Definitely a big help for buildings and I'm starting to get some promising results. (http://studio.verold.com/projects/53939f62ac832c020000039f)

[Marcel]

Reprojection error is in pixels, so if you are using values between 50 and 150 you are probably talking about the 'reconstruction uncertainty' value.

I find it hard to give set values for culling points using the reconstruction uncertainty, because it tends to vary a lot (and probably also depends on subject type).
Instead I tend to remove a certain percentage of the points from the cloud (for example I remove 25%). After that I run another gradual selection and use reprojection error, usually with a value of 1. 0 (so one pixel).

If you have a scan of a flat surface, you can sometimes see that it selects points in the outer parts of the photos (where the lens is less sharp than in the center). This underlines again the extreme sharpness that Photoscan needs.

We align our projects with a high max point count, so our Sparse Clouds have a lot of points. Even if we remove 50% of the points using the gradual selection, we still have more than enough left. The idea is that by starting with much more points and doing heavy culling, the quality of the remaining points is much higher.

What kind of effect did you see when you removed too many points from the Sparse Cloud? I would like to know more about the relationship between Sparse and Dense Cloud (are the Sparse Cloud points used as some kind of starting point for the points in the Dense Cloud, or is it only important for the alignment of the images?)

[ozbigben]

I'll blame Alexey for explaining them in the wrong order 

Yes, I meant reconstruction uncertainty. I've been increasing the size of the sparse point cloud already, although for no real reason so at least now I have one.  If the point cloud was too sparse I saw things like sections of flat walls becoming waved, particularly in areas where there were fewer overlapping images.  Given enough points I think I'd aim for something in the order of 50-100.

The percentage of points does seem to increase with the complexity of the object/scene rather than potential distortion errors. Using image sets from a GoPro on "wide" most of the bad points were taken out by the reprojection error, with relatively few points being taken out using an uncertainty of 50 after that (but those that were removed were mostly around the edges of image pairs. With images from my DSLR the spread of points was more even and I chickened out at 70 as they were much more evenly scattered but it may have still been a comparable increase in quality.

It may be worth adjusting the size of the initial sparse point cloud to suit the scene complexity with the goal of using a single value (or at least more consistent values) for reconstruction uncertainty. Something to keep an eye on while I get some more experience.

[ozbigben]

I've been experimenting some more and have a few ideas for some more tests to try and find optimum values for producing the best quality sparse point cloud.  Is there a way of accessing the error values of points in the sparse point cloud?  If it's not easy I can always use gradual selection to get the numbers manually.

I still have to run some definitive tests but on appearance it looks like the best option would be to align photos with a point limit of 0 to get everything, and then use gradual selection to reduce the sparse point cloud. I've been using reprojection error = 0.5 and reconstruction uncertainty = 50 as mentioned earlier.  Using high values for the point limit in the align photos sometimes resulted in a much smaller sparse point cloud after gradual selection and I'd chicken out and use values of 0.7 and 70 respectively to increase the number of points left. Starting off with all available points let me go back to my preferred values for gradual selection.

I'd still like to understand how the numerical value of reconstruction uncertainty relate to the quality of the point. It certainly appears to be an important component of cleaning up the point cloud. Points removed (following a reduction using reprojection error) seem to be mainly distant points, errors near the edge of images and/or points appearing in fewer cameras.

[Marcel]

You can use 'show info' on the chunk (right mouse). You'll get a dialog window with information like reconstruction uncertainty.

I made some tests a while back  by running the same alignment many times with an increasing number of max points. The accuracy kept increasing, although the returns get smaller and smaller. The big question is what kind of accuracy is enough.

I'm travelling at the moment, when I am back I will show the graph I made (max points vs reconstruction uncertainty and reprojection error).

[ozbigben]

Thanks marcel, there is some useful info in that window, although it only shows reprojection error (mean and max). I'm not really concerned with the averages of the errors in the point cloud after camera alignment, I'm looking at how many bad points I can get rid of so I'm looking for the distribution of the error values.  e.g. for one of my GoPro image sets I end up removing 90% of the sparse point cloud.

Reprojection errors seem to be more evenly distributed throughout the point cloud whereas reconstruction uncertainty seems to be more localised.

[ozbigben]

I ran some comparison tests using gradual selection for one variable only and looking at the stats for the resulting point cloud. I did this for an image set from my GoPro and iPhone, the iPhone image set is the one in the attached images.  Clearly reconsruction uncertainty and reprojection error are completely unrelated so removing bad data points from each is beneficial.

The screengrabs show the final sparse point cloud. Notice that all of the distant points had relatively high reconstruction uncertainty but not necessarily a high reprojection error. The questions that remain for me:
* How do I judge when there are too few points in the sparse point cloud?
* What values for reconstruction uncertainty can be considered "good"?
* Is reconstruction uncertainty related to source image size?

Getting the reprojection error down in the sparse point cloud also seems to speed up the dense point cloud creation.  Using a drastic reduction of just reconstruction uncertainty usually increases the mean reconstruction error for the remaining points and generating the dense point cloud was taking ~10-15x longer than other gradual selections.

[Marcel]

Why do you use an iPhone or GoPro? Unless your goal is to test working with lower quality images, I think you are making it hard on yourself?

Getting the reprojection error down in the sparse point cloud also seems to speed up the dense point cloud creation.

This could be because less camera pairs are selected for processing? When Photoscan starts the Dense Cloud reconstruction, it selects which camera pairs it uses based on the points in the Sparse Cloud. There is probably some threshold (like "two cameras need to share at least 1000 points to be considered a valid camera pair").

You can see the number of camera pairs in the console, it says something like "92 out of 120 camera pair selected.

If you have the console log output to a text file enabled you might be able to look it up without having to re-run the scans.

[ozbigben]

I'm using lower quality images specifically to have a wider range of errors and making any improvements to the mesh more obvious... although the iPhone is giving pretty good results (using PureShot to get dRAW TIFF). With DSLR images the differences are more subtle.

I'm running the dense cloud and mesh creation on 4 variations of the same sparse point cloud to see where/if the improvements are more obvious.  I've started saving the console output with these tests to have a look at later but this lot won't be finished until tomorrow night.

[Marcel]

Ah, that make sense then. You are suffering for science

[James]

In the most basic terms (i.e. how i understand it) reconstruction uncertainty is inversely related to 'image baseline distance' i.e. if all the images that contribute to determining the 3d position of a point are very close together, relative to the distance to that point, then the uncertainty is high (although the reprojection error may well be low). Reconstruction uncertainty is therefore unrelated to how well the cameras are aligned/modelled. It is something like the ratio between XY uncertainty and Z uncertainty I have read on here somehwere, but don't have time to search it out.

Reprojection error is based on how well the camera alignments 'agree' with each other. Each point in the sparse cloud has a single XYZ position calculated from potentially several cameras, but only in an ideal world would that single XYZ position appear in exactly the right place in each image if the 'virtual cameras' were real, so the reprojection error describes how far these points lie from their true image coordinates when you project them back from their 3d XYZ position into the XY coordinates of the source images.

If I wasn't so busy I'd try and make that make more sense/be more helpful.

[ozbigben]

Thanks James, that makes sense.

[Marcel]

James, thanks for explaining it in simple terms. All explanations about reconstruction uncertainty I've read before gave my brain a blue screen

So if it depends on the relative position of the camera pair to the point, does this mean all points coming from a single camera pair share (almost) same reconstruction uncertainty?

I think one of the big reasons I have trouble with understanding reconstruction uncertainty is because it is a unit-less value. (For the reprojection error it's easier because it is in pixels, if you know that a point is 2 pixels off between a camera pair, it gives you a rough idea what it means).

[James]

The way i understand it, if you imagine 2 cameras looking straight ahead with a given distance between them, then this setup can give very good XY accuracy but not so good in Z, whereas if the cameras are convergent then you get better Z accuracy at the expense of XY accuracy. The ratio of the XY accuracy to Z accuracy is the reconstruction uncertainty so if the XY accuracy is equal to the Z accuracy then you get an reconstruction uncertainty of 1, when they cameras are somewhat convergent. As the cameras get closer together then if they maintain their angle to each other it is effectively the same as becoming 'more parallel' since the area of overlap gets smaller and what's left is the portion of the field of view that still looks straight ahead. So the smaller baseline is equivalent in a sense to being more parallel so the XY accuracy increases, the Z accuracy decreases, and the ratio between them goes up so therefore so does the reconstruction uncertainty, so it is a combination of position and angle.

To not really answer your question very well, because I'd need to work it out/test it, i think you see a kind of radial gradient of reconstruction uncertainty across the points from an image pair, with points near the middle being more 'uncertain' than points at the edges, but i may have that the wrong way round, but im sure i've seen that before as i slide the slider.

The most uncertain thing is how confident i am about any of this, but i dont think im too far off

i will try and test it all out and think about it more thoroughly at some point before too long!