Your opinion on USGS Agisoft Processing Workflow

[MaciekK]

Hello,
I think gradual selection is a very good tool for cleaning aerotriangulation. I went a little further than the USGS shows and I received accuracy below 0.2pix. This required the gradual removal of less accurate projections. I used 3 options: Reconstruction uncertainty, Projection accuracy, Reprojection error. After each correction, I optimized the camera and watched SEUW = ​​~ 1. I am a bit worried that a lot of points about 50% occurs only in 2 pictures - I have to work on it. Still, the construction seems very rigid. After enabling the coordinates of projection centers and camera optimization, I received estimated GCPs xyz coordinates differing max 0.02m from those measured in the field with the GNNS receiver. In addition, there is no dense cloud noise and it generates very quickly.
All this on P4RTK with a relatively weak camera. Metashape calculates the quality of my photos always above 0.7pix
I look forward to your experiences
Regards MK

[Paulo]

Hi Maciek,

your final numbers are very impressive but I think that your Tpts reduction is extreme from 729 000 to 69 000 for a 280 image data set. This represents about average of 626 projections per image... Some images may have much less projections and that may affect the depth map generation (needing at least 100 common points between image pairs so that it can be used).
I have processed a similar P4RTK data set and I reduce Tpts using gradual selection but keep a limit of 50% of original points to not go over... See comparative screenshots of your adjustment and mine. My repro error is also under 0.2 pixel....

[RoryG]

Our workflow pretty much follows the USGS processing workflow, however we do try and not remove over 50% of the sparse cloud points. We always aim to achieve a reprojection error under 0.3px which gives us very accurate outputs - a recent test project we achieved 1.4cm RMSE XYZ on our surveyed checkpoints. You can read the article here: https://www.linkedin.com/pulse/post-processed-kinematic-gnss-mapping-rory-gillies/

Here's a couple of screen grabs from the report:

[MaciekK]

Thanks for the feedback, guys, I'm also wondering if I'm doing the right thing, deleting so many points. On the other hand, I worked a few years with classic aerial photogrammetry and there was no question of 100 stereo pairing points. Good relative orientation came out at about 20 points. Hence, I have no resistance when I have only 100 projections in the picture (usually they are pictures on the edge of the block). I read somewhere - I do not remember the source that the quality of Dense Cloud depends primarily on the correct parameters of relative orientation and I hope that these about 100 good points guarantee - this is confirmed by my tests. The topic of camera calibration also remains in my head - I do not know to what extent the reletive orientation is responsible for the correct calibration and to what extent it is external. Please, Alexey, join the discussion ..... In the coming days I will work a block of about 1500 photos. This will be a more representative example ...
Thanks again for your data, I've been working on Metashape for about a month and I don't understand all the procedures.
Regards

[gsmarshall]

Hello all,

This may not fall directly under the topic of this thread, but it seems there are lots of experienced metashape users with helpful insights on here, so here we are.

I am working with two fairly small sets of historic aerial photos (30-50 images each, from 1943 and 1978), and have some questions about what the error rates mean and how I might improve my workflow. I am not certain of exactly how the images were digitized, but I believe they came from negatives scanned in with a high quality (but not photogrammetric) scanner, and are stored as .tifs. After initially aligning, I have been incorporating the basic elements of the USGS workflow mentioned above: editing the sparse cloud by reconstruction uncertainty and projection accuracy, importing control points, editing by reprojection error, and optimizing cameras after each step. The dense point cloud and DEM generated from the images has too much noise and too many holes to produce a good orthomosaic, so I have been using a lidar DEM and have gotten better results with that.

The fiducial marks on the images are not auto-detectable so I have so far not used them (I picked up this project fairly recently), but I would be very interested to hear about anyone's experience using them/how much difference they make.

So far, camera errors for one set are pretty rough (1-2px), with better control point error. The orthomosaic has few visual errors in my area of interest (near the center of the scene, with good dispersal of GCPs, and at higher elevations than the surrounding area), but significant visual errors in the valleys. The images for this set are not cropped perfectly - there is still some border on one side of the image, and some of the fiducials are cut off - so that may be hampering the quality of my results, but I am unsure of how much difference that makes.
 
The other set has much better camera error (~0.4px) and control point error, and the orthomosaic is good but still has some relatively small visual errors (disjointed roads, duplicated areas over seams, etc).

I have two main questions: For the worse image set, placing more GCPs in the scene increases the camera error and disperses the visual errors across the scene, so that the valleys aren't as bad but the uplands (where I'm interested) are worse. Why might this happen? Is it the lack of geometric correction from fiducials, poor overlap, poor quality GCPs (all of which are possible)?

Second, regarding the better image set, how are the camera errors, control point errors, and visual errors related? My goal is to measure vegetation change with a supervised classification, so it seems to me like the visual errors in both orthomosaics would harm the accuracy of this measurement; but maybe a lower camera error produces better results across the whole image, while the visual errors are isolated to the seamlines?

I am quite new to metashape and photogrammetry, and any help would be much appreciated.

[Alexey Pasumansky]

It seems that just a couple of weeks ago a new version of a detailed USGS workflow (for Metashape 1.6) has been published:

https://pubs.er.usgs.gov/publication/ofr20211039
https://pubs.usgs.gov/of/2021/1039/ofr20211039.pdf

[c-r-o-n-o-s]

The procedure in the instructions is very interesting, and I had already tested it some time ago.
Unfortunately, I have to say that the effort is in no relation to the result.

I use levelled fixed points to check the accuracy. (+/- 1mm)
 
1.) If you calculate without a tie point limit, the calculation time increases immensely.
2.) Up to a certain point of the gradual selection I get nearly the same results as if I use the "normal" 40000/4000 values.
3.) But the worst thing is:
The more I do the "optimisation" after a certain point, the worse the control points fit!
It comes to a kind of overcorrection, which delivers worse results for me.

In the best case , I am 1mm better; but have a significantly longer calculation time.

[PROBERT1968]

Hey Alexey,

There is another one from US BLM  that had written about Close range photogrammetry on their website that might be helpful for others that needed help to understand.  When I took the training from my US Forest Service I learned how to use the software back then I think it was in 2017 . It was PhotoScan before you rebrand to Metashape .

So I had that copy in PDF but it is big file to upload here so I found the link for everyone in here that wants to read can download there.. It is not the workflow but it explains how it works .

https://www.blm.gov/documents/national-office/blm-library/technical-note/aerial-and-close-range-photogrammetric

The BLM and US Forest Service are more focus on using UAS to fight the wildfires here in the USA. 

I think I have posted a few on this site for anyone who wants to learn about UAS and Drones.


May I suggest you that it might be a good idea to add a new technical or workshop under your forum for anyone to share the links. Just saying ...

[dpitman]

For anyone using MS with aerial drone image sets, this is very interesting.  It all seems straight forward until you get to the the removal of points using gradual selection.  I did a quick test with the project I'm working on and if I follow the recipe, the values for photo errors go down but the error for the check points goes up. I'm talking very small movement, but movement in the wrong direction. Since my confidence is highest in the check point coordinates, I abandoned the idea of reducing sparse cloud points if the reported check point errors go up.

Perhaps I'm misunderstanding this "subjective step" (used in the latter USGS workflow) and that it's not really suitable for projects where in the end, I uncheck all of the photos (cameras) and use only the coordinates of the ground control points?

It would be great if you experienced guys would weigh in.

Thanks,
Dave

[c-r-o-n-o-s]

For anyone using MS with aerial drone image sets, this is very interesting.  It all seems straight forward until you get to the the removal of points using gradual selection.  I did a quick test with the project I'm working on and if I follow the recipe, the values for photo errors go down but the error for the check points goes up. I'm talking very small movement, but movement in the wrong direction. Since my confidence is highest in the check point coordinates, I abandoned the idea of reducing sparse cloud points if the reported check point errors go up.

Perhaps I'm misunderstanding this "subjective step" (used in the latter USGS workflow) and that it's not really suitable for projects where in the end, I uncheck all of the photos (cameras) and use only the coordinates of the ground control points?

It would be great if you experienced guys would weigh in.

Thanks,
Dave

That is exactly what I have found.
I think it depends on the project you are working on. What camera and what altitude etc.
Metashape does a good job here "out-of-the-box".
However, turning off RTK coordinates altogether should not be the goal.

[Paulo]

Hello all,

I have adapted my gradual selection procesure according to latest USGS workflow and I have found on 2 drone projects flown at 4 cm GSD that :
- in 1st case  it does increase check point accuracy during gradual selection last phase (Reprojection error);
- in 2nd case it does not, it actually decreases during 3rd phase  reprojection error optimization so stopping after Reconstruction uncertainty and Projection Accuracy phases gives better results.....
So results are mixed refarding check point accuracy. Of course it does decrease average reprojection error as well as tie point size. The key is to iterate the reprojection gradual selection so that each iteration only selects 10% of remaining Tie points up to user defined limit (in my case I stop when remaining points are 20% of original number of Tie pts) and check if Control/Check points error increase or decrease.

But for alignment, I do not use Tie Point limit of 0 as it increases processing time excessively (v1.7.3) instead I use 40 000 Key pt limit and 7 000 Tie pt limit and it gives good resuts for 16 Mpix imagery.

Also as dpitman in last phase (reprojection error) optimization I do uncheck all my cameras as my drone is equipped with a consumer grade GPS (5 to 10 m accuracy) and check all my GCPS (accuracy 2 to 3 cm) except my Check points....

[dpitman]

[snip] However, turning off RTK coordinates altogether should not be the goal.

[snip] Also as dpitman in last phase (reprojection error) optimization I do uncheck all my cameras as my drone is equipped with a consumer grade GPS (5 to 10 m accuracy) and check all my GCPS (accuracy 2 to 3 cm) except my Check points....

Also in my case, a P4P (not RTK) so the camera coordinates are not highly accurate.

[andyroo]

Enjoying the discussion on this topic a lot.

We developed the most recent USGS workflow using fixed lens DSLR cameras with survey-grade RTK positions and precise (sub-millisecond) event marks, so I wouldn't be at all surprised if the "best" workflow for built-in drone cameras (often with rolling shutter) and using only GCPs and/or consumer grade drone GPS is much different.

I am especially interested in the discussion limiting keypoints and tiepoints - since as Paulo noted, that increases processing time substantially. I hope to get around to doing some experiments on that soon, but again I expect the results will be somewhat specific to different camera types.

A couple notes re optimization from my current "best" workflow, which is very similar to the current USGS published workflow:

1) I've found that there doesn't appear to be any significant difference in how many/which points are selected whether  you optimize or not between performing gradual selection on Reconstruction Uncertainty (RU) and on Projection Accuracy (PA), and because I'm trying to minimize the number of times I optimize (both for speed and error propagation) I perform both RU and PA gradual selection before my first optimization.

2) At the moment I'm only performing 1 RU, 1 PA, and 2 Reprojection Error (RE) optimizations.

3) for all but the last RE optimization, I only optimize f, cx, cy, k1, k2, k3, p1, p2, and for the last RE optimization, I add Fit Additional Corrections (FAC), but I DO NOT add b1, b2, k4, p3, p4. I haven't re-evaluated recently, but last time I took a deep dive into the lens model parameters, I found that enabling b1, b2, k4, p3, p4 resulted in residual errors for those variables that were a significant fraction of the variable value (much more so than the other variables) - without significantly improving camera position errors or GCP errors. This implied strongly to me that I was overfitting - Again - this is specific to fixed-lens DSLR cameras with precise camera and ground control positions.

4) I've noticed that if I do multiple optimizations with FAC enabled, error (camera and GCP) appears to increase, so I only enable Fit Additional Corrections on my final optimization.

5) I've had mixed results, sometimes worse, and at best insignificant improvement using the above methods and trying to tighten tie point accuracy values (tighten to the previously optimized RE step) on the last iteration, so at the moment I'm leaving tie point accuracy at the default. I expect that if I do multiple RE optimizations, I might be able to improve tie point accuracy, but in general I'm able to meet accuracy targets using the above methods with our camera systems.

[c-r-o-n-o-s]

I am especially interested in the discussion limiting keypoints and tiepoints - since as Paulo noted, that increases processing time substantially. I hope to get around to doing some experiments on that soon, but again I expect the results will be somewhat specific to different camera types.

Unfortunately, Agisoft's automatic limitation of tie points is a black box.
We don't know which "formulas" are used to reduce the points.
I only know that it works very well.
As already mentioned, it is certainly possible to get another millimetre out of it, but the effort for this increases a lot.
It is possible that metashape uses a different but similar algorithm to limit the tiepoint.

By the way, the hole workflow is similar to the one described here:
https://www.hcu-hamburg.de/fileadmin/documents/Geomatik/Labor_Photo/publik/cncg2018_mayer_et_al.pdf