Forum

Author Topic: 1.6.5 align/optimize more accurate than 1.7.2 (aerial survey with PPK/RTK & GCP)  (Read 1967 times)

andyroo

  • Sr. Member
  • ****
  • Posts: 397
    • View Profile
I was a little surprised to find that our accuracy decreased, and processing time and memory requirements increased, when processing imagery with the most recent versions of Metashape.

I compared alignment and optimization results for 1.6.5/1.6.6 and 1.7.2 using eight overlapping flights (F1 through F8 in table below, ~140,000 images total) processed with precise camera positions and a single GCP.  These eight flights were aligned and optimized in two batches of four temporally adjacent flights using a 4D technique (sensu Warrick et al 2017), with each batch being roughly 70,000 images total. Results were compared by differencing 1m integer-cell-bounded DSMs produced by metashape for each flight with RTK elevations from 34 GCPs distributed in a region comprising about 20% of the total flight area.

The same source project and processing steps were used for both 1.6.5 and 1.7.2 (alignment, optimization, dense cloud processing, and DEM production and export were entirely batch processing or python API). Settings such as alignment quality and keypoint/tiepoint limit were identical (70,000/0). After alignment and optimization, a sub-region with ground control points was reconstructed and DEM was output, then the DEM elevations at 34 different GCPs were extracted (one of which was used in the optimization).

The table below shows mean and median signed and unsigned error when differencing 34 GCPs with DEMs created from each flight, as well as the average  of each error for all flights (last data column). In all cases, the 1.7.2 values were elevated vs 1.6.5.

F1F2F3F4F5F6F7F8Avg
-0.01-0.03-0.06-0.07-0.11-0.09-0.10-0.11-0.07mean signed differenceV 1.6.5 RuPaReX2+FitAddl on last iteration
-0.03-0.07-0.08-0.09-0.14-0.13-0.14-0.13-0.10mean signed differenceV 1.7.2 RuPaReX2+ FitAddl on last iteration
     
0.00-0.02-0.04-0.05-0.09-0.09-0.09-0.11-0.06median signed differenceV 1.6.5 RuPaReX2+FitAddl on last iteration
-0.04-0.06-0.08-0.07-0.13-0.12-0.11-0.13-0.09median signed differenceV 1.7.2 RuPaReX2+ FitAddl on last iteration
     
0.060.070.090.090.120.110.150.120.10mean absolute differenceV 1.6.5 RuPaReX2+FitAddl on last iteration
0.120.160.170.180.210.240.260.210.19mean absolute differenceV 1.7.2 RuPaReX2+ FitAddl on last iteration
     
0.050.060.050.060.090.100.110.110.08median absolute differenceV 1.6.5 RuPaReX2+FitAddl on last iteration
0.060.080.090.070.130.120.110.130.10median absolute differenceV 1.7.2 RuPaReX2+ FitAddl on last iteration

Interestingly, 1.7.2 found more points, and kept more points after optimization. A few other observations (drawn from one of the two batches - sorry for the code formatting, was having trouble with column alignment):

Code: [Select]
1.6.5 1.7.2 1.7.2/1.6.5
Matching time (h) 14 28 200%
Alignment time (h) 44 60 136%
Optimize time 15.1 6.2 41%
Matching memory(GB) 48 217 452%
Alignment memory 97 132 136%


Total Align/Opt (h) 73.1 94.2 129%

Reference:

Jonathan A. Warrick, Andrew C. Ritchie, Gabrielle Adelman, Kenneth Adelman, Patrick W. Limber "New Techniques to Measure Cliff Change from Historical Oblique Aerial Photographs and Structure-from-Motion Photogrammetry," Journal of Coastal Research, 33(1), 39-55, (1 January 2017), https://doi.org/10.2112/JCOASTRES-D-16-00095.1


macsurveyr

  • Jr. Member
  • **
  • Posts: 64
    • View Profile
Andy,

Interesting.

Do you have differences between just GCP checkpoints - results just from the alignment and optimizations? Since the surfaces from depth maps is a different process, as I understand it, from 1.6.5 and 1.7.x I am wondering if the difference is because of the DEMs and how they are created.

Tom

dpitman

  • Full Member
  • ***
  • Posts: 158
    • View Profile
I was a little surprised to find that our accuracy decreased, and processing time and memory requirements increased, when processing imagery with the most recent versions of Metashape.

Thank you for your testing.  Although the results are a bit disheartening, right.

c-r-o-n-o-s

  • Jr. Member
  • **
  • Posts: 84
    • View Profile
To be honest, I don't really understand the procedure yet...
Are there deviations in the pure camera calibration / control points?
(Without a point cloud or an elevation map being calculated?).
« Last Edit: July 11, 2021, 01:28:11 PM by c-r-o-n-o-s »

andyroo

  • Sr. Member
  • ****
  • Posts: 397
    • View Profile
Do you have differences between just GCP checkpoints - results just from the alignment and optimizations? Since the surfaces from depth maps is a different process, as I understand it, from 1.6.5 and 1.7.x I am wondering if the difference is because of the DEMs and how they are created.

Hi Tom,

I have only placed and enabled a single GCP (as a control point) because I am trying to (a) minimize all of my manual and subjective pointing and clicking, and (b) evaluate how well we can reconstruct  surfaces with minimal ground control. All of the other GCPs/checkpoints are in the project, but they are disabled and unplaced on individual images. I should be able to make a GCP master file with each point placed on every photo and import that and enable it (with the other GCPs unchecked) to compare accuracy - I hope to do that soon, but summer field season is going to make it a fall/winter project I think.

To clarify re DEM production - I am still generating DEMs from the dense cloud - again, using the same process, exactly (scripted/batched) for both versions of metashape, but from what Alexey has said (and I think maybe what you were referring to), I understand that they made significant changes in both the alignment and the depth maps algorithms. I did notice with the dense clouds that there is much more complete reconstruction in areas that previously had larger gaps (particularly in areas of low texture) - maybe those areas are being more thoroughly, but more noisily reconstructed? Not sure.

So, in other words, yes. I wouldn't be surprised if some aspect of the error is attributable to changes in depth maps/dense cloud reconstruction, and is propagating to the DEM from the dense cloud. One thing I noticed that makes me think there's error attributable to the alignment though is that the sparse clouds have many more points initially (something like 20% in our case I think) - and if that represents including points in their implementation of the SIFT algorithm that would have been rejected (something like reducing the difference-of-gaussians threshold or changing the way that's calculated?) in previous versions, maybe that could be a factor? total guessing here.

I've tried to do a closer look at residual error in the camera model, but I'm having issues with calibration coefficients not reliably propagating through to the final aligned project, so that makes recovering those values hit-or-miss at the moment.

Are there deviations in the pure camera calibration / control points?
(Without a point cloud or an elevation map being calculated?).

Hi c-r-o-n-o-s,

If I understand correctly, you're wondering about differences in the GCP and camera error (lens model error too?) between 1.7.3 and 1.6.6? I haven't been able to fully evaluate the lens model errors because of problems propagating camera calibration coefficients through optimization, but I should be able to compare the camera and single-GCP errors and update on that - Although it probably would be good to do what I've been thinking about, and what Tom asked about, and manually place the other GCP points on all the images to use as checkpoints and compare the error in the alignment-only between the two versions.

The only things I can comment on at the moment re the "pure" alignment parameters are that after optimization:

1) RMS Reprojection Error was slightly lower in 1.6.6 vs 1.7.3 (0.289357 pix vs 0.297011 pix - test 2b and 0.30069 pix vs 0.306297 pix - test 2a), (max RE was higher in 1.6 in test 2b but higher in 1.7 in test 2a) (2b and 2a are different collections of flights - internal referencing system for my brain - processed identically).

2) 1.7 found about 20% more points than 1.6, removed about 3% less of the original points during optimization, and aligned about 0.05% more images in both test 2a and 2b

3) in test 2a, total camera error (m) is reported lower for 1.7 (0.14496) than 1.6 (0.14795), and computed average camera error (m) from individual camera error values is also lower in 1.7 (0.12472  than in 1.6 (0.12660). BUT interestingly, average camera error (pix) is higher in 1.7 (0.31083) than in 1.6 (0.30448),  Haven't looked in 2b yet.


Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13332
    • View Profile
Hello Andy,

If you can share the sample dataset (images) and the project processing results in 1.6 and 1.7 versions, where the accuracy difference can be observed, please send the download link to support@agisoft.com, so that we could analyze the internal behavior and investigate the reason of the observed difference.
Best regards,
Alexey Pasumansky,
Agisoft LLC

andyroo

  • Sr. Member
  • ****
  • Posts: 397
    • View Profile
I'll try to make a smaller project in the area where we have ground control - the existing project is several hundred thousand images.

Andy