Agisoft Metashape
Agisoft Metashape => General => Topic started by: ozbigben on June 07, 2014, 07:25:15 AM
-
I'm working through improving the quality of the dense point cloud and found this helpful explanation.
"Reconstruction uncertainty". In case you have only two cameras and a point is being triangulated by intersection of two rays there is a direction in which the variation for the point position is maximal and another direction with the minimal variation. Dividing one on another (max to min) we've got reconstruction uncertainty value. Mostly this criterion is designed for visualization and estimating the errors. In general this value characterizes the accuracy of positioning points in cloud.
"Reprojection error" demonstrates the accuracy of point positioning and is specified in pixels. We recommend to remove points with huge reprojection errors before optimizing photo alignment.
This has been really useful and is probably worth putting in the wiki. I'm just trying to properly understand how the numerical value for Reprojection error Reconstruction uncertainty relates to quality (and thus what a practical minimum value would be). Is it relative to the input image size or does it relate to the size of image pattern that PS uses to "recognise" features.
I've managed to remove too many points a couple of times which is handy as I now know what effect that has on the dense point cloud.
-
So far I've found significant improvements in the range of 150-50, although at lower values it might take out too many points causing other problems. If there are significant gaps in the sparse point cloud using 50, this seem to be an indication of not enough images from different positions for those areas, which might make a useful quality check while shooting objects/buildings(?)
Definitely a big help for buildings and I'm starting to get some promising results. (http://studio.verold.com/projects/53939f62ac832c020000039f (http://studio.verold.com/projects/53939f62ac832c020000039f))
-
Reprojection error is in pixels, so if you are using values between 50 and 150 you are probably talking about the 'reconstruction uncertainty' value.
I find it hard to give set values for culling points using the reconstruction uncertainty, because it tends to vary a lot (and probably also depends on subject type).
Instead I tend to remove a certain percentage of the points from the cloud (for example I remove 25%). After that I run another gradual selection and use reprojection error, usually with a value of 1. 0 (so one pixel).
If you have a scan of a flat surface, you can sometimes see that it selects points in the outer parts of the photos (where the lens is less sharp than in the center). This underlines again the extreme sharpness that Photoscan needs.
We align our projects with a high max point count, so our Sparse Clouds have a lot of points. Even if we remove 50% of the points using the gradual selection, we still have more than enough left. The idea is that by starting with much more points and doing heavy culling, the quality of the remaining points is much higher.
What kind of effect did you see when you removed too many points from the Sparse Cloud? I would like to know more about the relationship between Sparse and Dense Cloud (are the Sparse Cloud points used as some kind of starting point for the points in the Dense Cloud, or is it only important for the alignment of the images?)
-
I'll blame Alexey for explaining them in the wrong order ;)
Yes, I meant reconstruction uncertainty. I've been increasing the size of the sparse point cloud already, although for no real reason so at least now I have one. If the point cloud was too sparse I saw things like sections of flat walls becoming waved, particularly in areas where there were fewer overlapping images. Given enough points I think I'd aim for something in the order of 50-100.
The percentage of points does seem to increase with the complexity of the object/scene rather than potential distortion errors. Using image sets from a GoPro on "wide" most of the bad points were taken out by the reprojection error, with relatively few points being taken out using an uncertainty of 50 after that (but those that were removed were mostly around the edges of image pairs. With images from my DSLR the spread of points was more even and I chickened out at 70 as they were much more evenly scattered but it may have still been a comparable increase in quality.
It may be worth adjusting the size of the initial sparse point cloud to suit the scene complexity with the goal of using a single value (or at least more consistent values) for reconstruction uncertainty. Something to keep an eye on while I get some more experience.
-
I've been experimenting some more and have a few ideas for some more tests to try and find optimum values for producing the best quality sparse point cloud. Is there a way of accessing the error values of points in the sparse point cloud? If it's not easy I can always use gradual selection to get the numbers manually.
I still have to run some definitive tests but on appearance it looks like the best option would be to align photos with a point limit of 0 to get everything, and then use gradual selection to reduce the sparse point cloud. I've been using reprojection error = 0.5 and reconstruction uncertainty = 50 as mentioned earlier. Using high values for the point limit in the align photos sometimes resulted in a much smaller sparse point cloud after gradual selection and I'd chicken out and use values of 0.7 and 70 respectively to increase the number of points left. Starting off with all available points let me go back to my preferred values for gradual selection.
I'd still like to understand how the numerical value of reconstruction uncertainty relate to the quality of the point. It certainly appears to be an important component of cleaning up the point cloud. Points removed (following a reduction using reprojection error) seem to be mainly distant points, errors near the edge of images and/or points appearing in fewer cameras.
-
You can use 'show info' on the chunk (right mouse). You'll get a dialog window with information like reconstruction uncertainty.
I made some tests a while back by running the same alignment many times with an increasing number of max points. The accuracy kept increasing, although the returns get smaller and smaller. The big question is what kind of accuracy is enough.
I'm travelling at the moment, when I am back I will show the graph I made (max points vs reconstruction uncertainty and reprojection error).
-
Thanks marcel, there is some useful info in that window, although it only shows reprojection error (mean and max). I'm not really concerned with the averages of the errors in the point cloud after camera alignment, I'm looking at how many bad points I can get rid of so I'm looking for the distribution of the error values. e.g. for one of my GoPro image sets I end up removing 90% of the sparse point cloud.
Reprojection errors seem to be more evenly distributed throughout the point cloud whereas reconstruction uncertainty seems to be more localised.
-
I ran some comparison tests using gradual selection for one variable only and looking at the stats for the resulting point cloud. I did this for an image set from my GoPro and iPhone, the iPhone image set is the one in the attached images. Clearly reconsruction uncertainty and reprojection error are completely unrelated so removing bad data points from each is beneficial.
The screengrabs show the final sparse point cloud. Notice that all of the distant points had relatively high reconstruction uncertainty but not necessarily a high reprojection error. The questions that remain for me:
* How do I judge when there are too few points in the sparse point cloud?
* What values for reconstruction uncertainty can be considered "good"?
* Is reconstruction uncertainty related to source image size?
Getting the reprojection error down in the sparse point cloud also seems to speed up the dense point cloud creation. Using a drastic reduction of just reconstruction uncertainty usually increases the mean reconstruction error for the remaining points and generating the dense point cloud was taking ~10-15x longer than other gradual selections.
-
Why do you use an iPhone or GoPro? Unless your goal is to test working with lower quality images, I think you are making it hard on yourself?
Getting the reprojection error down in the sparse point cloud also seems to speed up the dense point cloud creation.
This could be because less camera pairs are selected for processing? When Photoscan starts the Dense Cloud reconstruction, it selects which camera pairs it uses based on the points in the Sparse Cloud. There is probably some threshold (like "two cameras need to share at least 1000 points to be considered a valid camera pair").
You can see the number of camera pairs in the console, it says something like "92 out of 120 camera pair selected.
If you have the console log output to a text file enabled you might be able to look it up without having to re-run the scans.
-
I'm using lower quality images specifically to have a wider range of errors and making any improvements to the mesh more obvious... although the iPhone is giving pretty good results (using PureShot to get dRAW TIFF). With DSLR images the differences are more subtle.
I'm running the dense cloud and mesh creation on 4 variations of the same sparse point cloud to see where/if the improvements are more obvious. I've started saving the console output with these tests to have a look at later but this lot won't be finished until tomorrow night.
-
Ah, that make sense then. You are suffering for science ;)
-
In the most basic terms (i.e. how i understand it) reconstruction uncertainty is inversely related to 'image baseline distance' i.e. if all the images that contribute to determining the 3d position of a point are very close together, relative to the distance to that point, then the uncertainty is high (although the reprojection error may well be low). Reconstruction uncertainty is therefore unrelated to how well the cameras are aligned/modelled. It is something like the ratio between XY uncertainty and Z uncertainty I have read on here somehwere, but don't have time to search it out.
Reprojection error is based on how well the camera alignments 'agree' with each other. Each point in the sparse cloud has a single XYZ position calculated from potentially several cameras, but only in an ideal world would that single XYZ position appear in exactly the right place in each image if the 'virtual cameras' were real, so the reprojection error describes how far these points lie from their true image coordinates when you project them back from their 3d XYZ position into the XY coordinates of the source images.
If I wasn't so busy I'd try and make that make more sense/be more helpful.
-
Thanks James, that makes sense.
-
James, thanks for explaining it in simple terms. All explanations about reconstruction uncertainty I've read before gave my brain a blue screen :)
So if it depends on the relative position of the camera pair to the point, does this mean all points coming from a single camera pair share (almost) same reconstruction uncertainty?
I think one of the big reasons I have trouble with understanding reconstruction uncertainty is because it is a unit-less value. (For the reprojection error it's easier because it is in pixels, if you know that a point is 2 pixels off between a camera pair, it gives you a rough idea what it means).
-
The way i understand it, if you imagine 2 cameras looking straight ahead with a given distance between them, then this setup can give very good XY accuracy but not so good in Z, whereas if the cameras are convergent then you get better Z accuracy at the expense of XY accuracy. The ratio of the XY accuracy to Z accuracy is the reconstruction uncertainty so if the XY accuracy is equal to the Z accuracy then you get an reconstruction uncertainty of 1, when they cameras are somewhat convergent. As the cameras get closer together then if they maintain their angle to each other it is effectively the same as becoming 'more parallel' since the area of overlap gets smaller and what's left is the portion of the field of view that still looks straight ahead. So the smaller baseline is equivalent in a sense to being more parallel so the XY accuracy increases, the Z accuracy decreases, and the ratio between them goes up so therefore so does the reconstruction uncertainty, so it is a combination of position and angle.
To not really answer your question very well, because I'd need to work it out/test it, i think you see a kind of radial gradient of reconstruction uncertainty across the points from an image pair, with points near the middle being more 'uncertain' than points at the edges, but i may have that the wrong way round, but im sure i've seen that before as i slide the slider.
The most uncertain thing is how confident i am about any of this, but i dont think im too far off :)
i will try and test it all out and think about it more thoroughly at some point before too long!
-
I don't understand how the angle of the cameras matters for the accuracy of the point? I would think only the position of the cameras matters? Parallel or convergent changes the position of the point in the photo, but the relative position in 3D space stays the same. I need to sketch this out on paper :)
If the cameras are convergent then you have more overlap, but that just means you have more matching points between the two cameras?
I've seen the radial gradient as well, but it was definitely the points at the edges of the photo that had worse accuracy. I'm also not certain if it was when I was using Reprojection Error or Reconstruction Uncertainty selection. I think it was Reprojection error, and my guess was that it was an effect of lens softness at the edges of the frame. Unfortunately I can't test it at the moment.
-
You have more matching points but if you project back from a point to it's theoretical position in an image, a small error in the angle will result in a larger error for more distant points. The wider the fov of the lens, the greater the distance of points at the edges tends to be, so you would expect to see higher uncertainty errors at the edges. This is certainly apparent with the GoPro image sets I've been using. As you get to very low values for uncertainty it acts like a depth filter as you can see in my screen grabs. This might be useful for refining the camera positions for an object if you haven't masked the object. It also picks out points that are relatively distant above/below surfaces.
I'm only using the standard version at the moment so it just removes some of the cameras that have a high proportion of bad points, but it's a noticeable improvement. Now for the notes I was writing while people were also replying...
Didn't have time to do some screengrabs before heading off to work but the models had finished.
The differences were subtle at times, but significant enough.
Raw dense point cloud used 90% of cameras. Other combinations of settings I used only lowered this by another 10%. Dense point clouds only varied by about 10% in number of points, but this would have also been due to a reduction in noise.
The addition a low reconstruction uncertainty threshold along with a low reprojection error: reduced noise in underexposed areas, reduced the number of small floating artefacts, removed areas of high noise in flat surfaces, better definition of small protrusions of the object (e.g. tips of the owl's ears)
I also had a brief look at gradual selection based on the image count, but that ended up removing too many cameras when used in conjunction the other two settings but there are probably other circumstances where this would be useful.
Now it's going to be a case of fine tuning the balance of settings, although I think the values for uncertainty used will vary a bit for objects depending on the relative camera positions and fov of the lens but this could possibly be determined by starting with just a few images.
-
The wider the fov of the lens, the greater the distance of points at the edges tends to be, so you would expect to see higher uncertainty errors at the edges. This is certainly apparent with the GoPro image sets I've been using. As you get to very low values for uncertainty it acts like a depth filter as you can see in my screen grabs.
I would blame the image quality of the photo at the edges for this effect, rather than the points being further away. Especially with a GoPro which doesn't have a very good lens (no offence intended).
-
I was basing this on the points that were selected after removing reprojection error > 0.5 It's probably a combination of both in the case of the GoPro, and possibly more appropriate to say that the uncertainty seems to be higher when the matching points are near the edges of all matching images. You'll see this with pretty much any lens once you select a low enough uncertainty threshold. especially around the edges of a scene (or the outer edges of an aerial grid).
-
I need to sketch this out on paper :)
Definitely, me too! I struggle to imagine more than one camera in my head at a time :)
-
I hope the attachment below helps a bit. It covers just a small number of the very many permutations!
There are 4 basic setups, and assume an extremely low res camera model, so you can see the fov of an individual pixel.
Sketching this really helped because it shows that the reconstruction uncertainty value (as i thought i understood it - maybe i was wrong!) is not completely reliable.
#1. Cameras close together. almost parallel cameras looking at a point on a ground plane more or less dead ahead. XY uncertainty is 1.38 and Z uncertainty is 12.706, so the overall reconstruction uncertainty is 9.207.
#2. Cameras far apart, angled to look at the same point as in #1 in the center of the camera fov. XY uncertainty is slightly higher than #1, but Z uncertainty is much better, and so is overall reconstruction uncertainty.
#3. Cameras far apart, same distance as #2, looking parallel to each other. The same point as in #1 and #2 is now in a small area of overlap at the edge of the cameras fov, which has barely any effect on uncertainties and the reconstruction uncertainty is just very slightly higher.
#4 Cameras back close together again, same distance as in #1, but with the point of interest near the edge of the fov of both cameras. The uncertainty here is actually lower than in #1 for identical camera positions and orientations. This may be misleading though, because the XY uncertainty is greater than #1, and so is the Z uncertainty, so should be more uncertain overall, but it is the ratio between them that I thought was reported in photoscan.
#5 Identical to #4 except taking the 'Z' uncertainty as the maximum distance rather than perpendicular to ground, I'm not sure which is actually the correct one in this case, but still shows more or less the same result.
Now I am much more confused than I was to start with :)
-
Thanks for the effort James :) To some extent I like to understand the technical bits so that I can understand how to use them, and in the absence of exact answers I also go by observation. Using similar image counts and settings for generating sparse point clouds I'm finding that I'm using similar thresholds for uncertainty regardless of the camera used. Given the wide range of values for this error I was initially reluctant to drag it so far across to the low end. With image sets in the range of 120-200 for objects I'm finding 30-50 to be practical.
Looking at the points that get removed, I'm thinking it would be nice to be able to filter the dense cloud in the same way.
-
Here's my first test on a DSLR image set (Canon 5D mkIII, 50mm macro, 160 images) for an object, an armillary sphere. There are a few gaps I have to go back and fill in (must stop rushing these in my lunch break) but it's looking awesome so far. Generating a mesh was problematic even with a medium dense cloud, had to go up to high giving me 49M points.
-
Nice work James, I like how you added numbers to your sketches, that really shows whats happening.
Are you sure about the XY/Z relationship for the Reconstruction Uncertainty? Because from the explanation by Alexey in past forum posts, I have a slightly different interpretation. I'll try to sketch it out when I am back home (July). The conclusion is the same though, enough distance between the cameras is important and the camera angle is not relevant (as long as there's overlap).
This discussion has been very helpful, I think I have a better understanding on how to place the camera in some situations. Thanks!
Nice scan Ozbigben! It never ceases to amaze me when Photoscan accurately builds objects with thin or hollow parts like this.
-
No I'm not sure of anything much any more :)
I am certain I've seen diagrams like that in books talking about similar things in photogrammetry, but I may have just put 2 and 2 together and got 5 or so!
It is bound to be more complex than I suggest in any case, since this is such a simplistic 2D scenario with just a single stereo pair. Ideally I should test these wild theories out with some actual experimentation but just too busy at the moment.
Yeah that is a beautiful scan ozbigben! What kind of scale is that?
-
To some extent I like to understand the technical bits so that I can understand how to use them, and in the absence of exact answers I also go by observation.
I think I'm the same, I'm no good at remembering rules for things, so make more effort to try and understand what's going on so I can work out what I'm supposed to be doing!
-
Yeah that is a beautiful scan ozbigben! What kind of scale is that?
Thanks. It's about 1.5m high. I've made a number of attempts including 123D Catch but this is the first time I've managed to get the sphere to work. The metal rings are just under 1cm thick and without sufficient points on opposite surfaces you get holes, which has usually meant losing the top 2/3 of the sphere.