Forum

Author Topic: *HELP* Poor alignment, massive GCP errors, useless endproducts  (Read 9924 times)

BobvdMeij

  • Full Member
  • ***
  • Posts: 157
    • View Profile
*HELP* Poor alignment, massive GCP errors, useless endproducts
« on: September 07, 2017, 04:28:29 PM »
Dear all,

Last week we mapped two water barrier structures at two seperate locations as a pilot project for one of our clients, this week was all about processing of the images we captured. Normally, given the small area of each site (only 2.5 and 5 hectares respectively) this shouldn't take longer than half a day of work. Unfortunately, however, we are facing serious troubles throughout the process starting with image alignment, let alone with respect to distilling of the desired products. In the past days we have been running an endless number of processing runs, each time minorly tweaking some parameters, for well over 90 hours continously ánd simultaneously.

Without any success, however. We have seen everything so far ranging from Alignment failing miserably or all together up to GCP and CP accuraties indicating errors of several meters in all three (XYZ) directions. For what it's worth, these are the specs for the smallest of the two areas (see also attached screenshot, WSZZ_StudyArea).

Site specs
Area = approx 2.5Ha (450m in length, 70m at its widest).
Surface = 50% low standing grass / 50% bare soil
Elevation difference = less than 3m
Ground Control: 11 markers as GCP, 7 as independent checkpoints. The markers were positioned and measured using RTK-GPS before the flights and measured again prior to removal after the flights. XYZ coordinates were then averaged before being imported to PS.

The data was captured using the following:

Device specs
UAV: DJI Inspire Pro
Camera: Zenmuse X5
Altitude: 40m AGL (GSD = approx. 1cm)
Flight speed: 4.3m/s
Number of flights: 2 (total of 321 images)
Overlap (F/S): 75% - 75%. The overlap between the two flights was approximately 200%, just to be sure.
GCPs: Leica GS08 Plus

Initially the following processing workflow was used, in accordance with settings that we have used across most of our other projects: Alignment accuracy was set to High, Generic+Reference preselection were turned ON and Key- and Tie Point limits were set to 40.000 and 4.000 respectively. After Alignment we would manually mark the GCPs on all images they are visible on (> 9) as well as the Checkpoints but keep the latter checked OFF in the reference pane. Then camera's are Optimized (checking all boxes, except P3 and P4) and the resulting errors for both Control- and Checkpoints studied in the Reference Pane.

Immediately after Initial Alignment, however, a large black area can be distuingished in the scene where the two flights met. No Tie Points were found here whatsoever, regardless of overlap being well over 200% here and the underlying scene being no different than elsewhere (see WSZZ_AlignmentBlack). Inclusion of GCPs and Re-Optimization of Camera's did not result in improvements. Neither did re-running of Alignment using 40k/10k, 120k/40k, 160k/40k or 160k/120k for Key- and Tiepoint limits or turning Generic and Reference Pair Preselection OFF or ON. Although allowing a higher number of Tiepoints did result in the bare spot becoming smaller, the Sparse Cloud remained notably different from other locations in the scene.

To make matters even more interesting I decided to process each of the two flights in two seperate chunks, mark the GCPs in each, and merge both chunks using the GCP markers thereafter. This time, the area that remained fully black in the previous attempts was actually provided with a decent level of tie points. It remained somewhat more sparsely populated than other parts of the scene, but it at least demonstrated that the images of the scene were find and DO allow for Tie points to be found. Merging of both chunks, however, failed miserably and left a large vertical gap between both flights, regardless of each being georeferenced using RTK-GPS GCPs and basing the merging on Markers (see WSZZ_ChunksMisaligned).
 
Also please have a look at the screendump attached (WSZZ_ImageShift) displaying the water barrier (sparse point cloud) underneath with the 'Aligned' images on top. The appears to be a massive horizontal shift between the scene on the one hand and the grid flown overhead on the other. This remains even when GCPs are added and MARKED in the scene! We are absolutely sure that all images were captured at nadir (gimbal angle varies between minus) 88-90 degrees) and the blue plane of each image after alignment confirms this as well, but the shift remains.

More importantly still, however, is the fact that the total error (of both GCPs and CPs) throughout each of the above attempts was appalling. The total error averaged at around 2 meters although individual GCP errors reached even larger values. This notion remained largely unchanged throughout the different attempts, regardless of the number of Key- and Tiepoints, processing the flights seperately and/ or changing any other Alignment parameters. Even when I decided to incorporate ALL markers (including Checkpoints) as Ground Control Points for Camera Optimization, resulting in 18 GCPs in a scene as small as 2.5 hectares (!), the error remained in the order of meters. I have never ever witnessed anything like this produces as we have easily produced and (externally) validated both horizontal and vertical accuracies during prior projects below 5 cm.

Without going into too much detail it should be mentioned that we run into similar issues processing the images from the other pilot location. This location measures slightly bigger (4-5 Ha) and contains more (571) images and approximately 30 GCPs/CPs. The output, however, is almost identical. Alignment either fails miserably or produces weird artificats that cannot be explained for. Just as important, however, model accuracy is appalling with several meters (XYZ) even after the inclusion of an ever growing number of GCPs. Likewise, processing each of the flights seperately and then merging them together results in two vertically transposed models. After many hours I have managed to bring the two closers together, but GCP accuracy is still well over 30cm and absolutely not matching either expectations or the specifications.

In short, what is going on? We are absolutely clueless, regardless of processing a multitude of somewhat similar projects (and even larger) projects in the past without any problems and decent accuracies. What may explain this strange model behavior? More important, are there any more things we might consider in an attempt to succesfully produce the desired outputs after all?

SAV

  • Hero Member
  • *****
  • Posts: 710
    • View Profile
Re: *HELP* Poor alignment, massive GCP errors, useless endproducts
« Reply #1 on: September 07, 2017, 06:20:16 PM »
Hi BobvdMeij,

Without going into too much detail...... that's what you were saying in your post. I think this is one of the most detailed posts I've ever read in this forum  ;)

Indeed, the behaviour/results are quite odd.

Here are a few things you might want to try:

1. Uncheck all images in the reference pane in order to force PhotoScan to only use your ground control points (not the camera locations). GPS altitude/elevation values coming from geotagged images of DJI drones can be problematic, especially for larger surveys which consist of several separate flights. Also use 4-6 GCPs (distributed all over the area) as check points and the rest of GCPs as control points. Also make sure that you run the image alignment step for ALL photos, NOT chunk by chunk.

2. Double-check the reference settings. What accuracy did you choose for the GCPs? It should match the accuracy of your RTK GPS (1-3cm, I guess). Best practice is to import the X, Y and Z error for each GCP.

3. Check if your RTK GPS was measuring in FIXED mode (achieving cm accuracy). If it was in FLOAT it won't have measured very accurately. This could explain the large error.

4. Check image quality (right click on images in photo pane > ESTIMATE IMAGE QUALITY). Maybe some of the images in the problematic area are blurry or out of focus.

5. You mentioned an image overlap of 75%. Double check if that's really the case, because on the DJI Inspire Pro / Zenmuse X5 setup different lenses can be used. If (accidentally) the wrong focal lenght is chosen in your mission planning/waypoint planning app, then you might not achieve the set image overlap. This could explain some of the problems.

6. After image alignment, use PhotoScan's Gradual Selection tool to remove all tie points with a reprojeciton error that is greater than 1. Probably choose a value between 0.5 and 0.8. Also find a 'natural' threshold for the reprojection uncertainty and delete points with high values. Then run OPTIMIZE (magic wand icon in reference pane). Your total error should now lie within a few cm.

I hope that helps you to fix the problem. Keep us/me updated.

Regards,
SAV
 
« Last Edit: September 07, 2017, 06:22:28 PM by SAV »

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14855
    • View Profile
Re: *HELP* Poor alignment, massive GCP errors, useless endproducts
« Reply #2 on: September 07, 2017, 06:24:11 PM »
Hello BobvdMeij,

Can you please share the processing report from this project?
In case it is possible, please re-save the project with the alignment results in PSZ format and send to support@agisoft.com, so that we could check it in detail.
Best regards,
Alexey Pasumansky,
Agisoft LLC

BobvdMeij

  • Full Member
  • ***
  • Posts: 157
    • View Profile
Re: *HELP* Poor alignment, massive GCP errors, useless endproducts
« Reply #3 on: September 07, 2017, 09:03:05 PM »
Hello BobvdMeij,

Can you please share the processing report from this project?
In case it is possible, please re-save the project with the alignment results in PSZ format and send to support@agisoft.com, so that we could check it in detail.

Sure thing, first thing I'll do tonorrow morning when I'm back at the office! The PSZ file is a bit of a 'mess' though with about eight chunks, each processed with different parameters, number of images, until OR beyond Alignment and with different numbers of GCPs or no GCP optimization at all (yet). But I'll reckon I'll send it all, assuming you will be able to make sense of it all.

If you like I can also send you the source images and GCPs in .csv format, so you can run Alignment yourself to see if the same 'black spot' appears?

SAV

  • Hero Member
  • *****
  • Posts: 710
    • View Profile
Re: *HELP* Poor alignment, massive GCP errors, useless endproducts
« Reply #4 on: September 08, 2017, 05:31:55 AM »
Hi BobvdMeij,

I think you just mentioned the issue in your last post. 8 chunks processed with different parameters and settings, some with GCPs some without. ???
This is probably causing the issues.

As mentioned in my previous post, you should run the image alignment step for ALL photos, NOT chunk by chunk. Split the project into smaller chunks AFTER you have computed your tie points (= photo alignment). Also include all the GCPs (use some as check points) during photo alignment.

Cheers
SAV


Hello BobvdMeij,

Can you please share the processing report from this project?
In case it is possible, please re-save the project with the alignment results in PSZ format and send to support@agisoft.com, so that we could check it in detail.

Sure thing, first thing I'll do tonorrow morning when I'm back at the office! The PSZ file is a bit of a 'mess' though with about eight chunks, each processed with different parameters, number of images, until OR beyond Alignment and with different numbers of GCPs or no GCP optimization at all (yet). But I'll reckon I'll send it all, assuming you will be able to make sense of it all.

If you like I can also send you the source images and GCPs in .csv format, so you can run Alignment yourself to see if the same 'black spot' appears?

BobvdMeij

  • Full Member
  • ***
  • Posts: 157
    • View Profile
Re: *HELP* Poor alignment, massive GCP errors, useless endproducts
« Reply #5 on: September 08, 2017, 08:17:08 AM »
Hi BobvdMeij,

I think you just mentioned the issue in your last post. 8 chunks processed with different parameters and settings, some with GCPs some without. ???
This is probably causing the issues.

As mentioned in my previous post, you should run the image alignment step for ALL photos, NOT chunk by chunk. Split the project into smaller chunks AFTER you have computed your tie points (= photo alignment). Also include all the
GCPs (use some as check points) during photo alignment.

Cheers
SAV

Sorry, I probably should have been more clear. They are not specifically seperate 'chunks' to be merged. They are just duplicates of the same projects and input data that have been run using different parameters, in order to evaluate any improvements occuring after finding out our default workflow did not work out as planned.

Only two chunks are actual chunks, representing the processing of the two flights seperately, which were then merged into a third chunk.

As for your comments on working with chunks I am not entirely following you. This is how the two actual chunks (i.e. seperate flights) were processed:

1. Import images of each flight in two seperate chunks.
2. Run Alignment for each chunk using the same parameters.
3. Mark all GCPs in EACH chunk (two GCPs are visible in BOTH chunks)
4. Optimize EACH chunk using markers (images checked OFF)
5. Align chunks based on Markers
6. Merge chunks, hereby merging BOTH the Sparse Clouds AND Markers of each chunk
7. Re-Optimize camera's in the resultantly merged project

You seem to suggest to first run alignment for all (approx. 360 images), including GCP marking and optimization thereafter, and then split it two seperate chunks. This is, as far as I can tell, NOT what the PS Manual suggest. The PS Manual actually seems to suggest the workflow I presented above. The above workflow is also what we have been using on previous projects using chunks, and which has Always resulted in XYZ accuracies well below 5cm.

Yoann Courtois

  • Sr. Member
  • ****
  • Posts: 316
  • Engineer in Geodesy, Cartography and Surveying
    • View Profile
Re: *HELP* Poor alignment, massive GCP errors, useless endproducts
« Reply #6 on: September 08, 2017, 09:28:18 AM »
Really interesting post !
As SAV said, never seen such amount of details before !

It looks you've followed a good workflow but something goes wrong at some point.

- Before all, I suggest you to keep your whole set of pictures into an unique chunk (both flight into one chunk)
- Check out image quality to highlight unusable pictures
- Usually when I worked with quality UAV data, I don't set any limit of keypoints and tiepoints (0, 0) and then I use gradual filters to keep the most accurate ones. But before that, you should check if several pictures present less points than others, or larger RMS error than others. Especially around point 2004 where it looks there is no matching at all.
- Double check your alignment process before handling any georeferencement process. One of the big problem is coming from alignment.
- If you try to go further without fix it, you won't be able to get something good.

May you be able to also share us your processing report ?
As well, as I'm really curious and always looking for errors anticipation, would it be possible to share your set of pictures so that I can handle some tests be myself ?

Regards

P.S. Re-optimize your merged project is useless because, if you align each flight in two exasperated chunk, there is no other link between them than markers you've used to georeferenced them. Then you won't get any improvement by doing that.
« Last Edit: September 08, 2017, 09:31:47 AM by Yoann Courtois »
--
Yoann COURTOIS
R&D Engineer in photogrammetric process and mobile application
Lyon, FRANCE
--

BobvdMeij

  • Full Member
  • ***
  • Posts: 157
    • View Profile
Re: *HELP* Poor alignment, massive GCP errors, useless endproducts
« Reply #7 on: September 08, 2017, 10:09:43 AM »
As SAV said, never seen such amount of details before !

I believe that the least I can do when asking for help is to provide as many details as possible ;)

Quote
- Check out image quality to highlight unusable pictures
I did, image quality ranges between 0.81 and 0.86 which should be more than sufficient. There is some White Balance inconsistency across the scene (WB set to auto), but this is a recurring pattern across the entire site. Hence this does not (fully) explain why Alignment seems to fail in one location specifically.

Quote
May you be able to also share us your processing report ?
Below you find a screendump of two sections of our report, from two runs. The first (top) one relates to our initial attempt running the full project using mostly default parameters. The second (bottom) one relates to the project after merging the chunks wherein each flight was processed seperately. To me the report indicates at least one thing; somethin MAJOR is going wrong during Alignment/Camera Calibration. The pattern of camera residuals is something I have never seen before, although we have used the same camera that we have used throughout the past year! Overlap on the other hand is decent (>9 at all critical locations) to say the least. However, the Survey/Overlap figure indeed indicates a horizontal shift between the studied surface on the one hand and the camera positions on the other. The pattern of camera positions should be fully aligned with the surveyd area, but it is far from it! It is also worth nothing that images were acquired at 40m AGL, although the two quality reports indicate otherwise!

Quote
P.S. Re-optimize your merged project is useless because, if you align each flight in two exasperated chunk, there is no other link between them than markers you've used to georeferenced them. Then you won't get any improvement by doing that.
That's exactly what I thought as well, until I gave it a try nonetheless. After merging the two seperate chunks there would be a significant vertical shift between the point clouds making up each (i.e. several meters). After re-optimizing camera's in the merged chunk this vertical shift was lowered significantly, although XYZ accuracy remained appalling. This is not to say that I believe optimization after merging helps, but rather that it does alter things significantly.

BobvdMeij

  • Full Member
  • ***
  • Posts: 157
    • View Profile
Re: *HELP* Poor alignment, massive GCP errors, useless endproducts
« Reply #8 on: September 08, 2017, 10:24:37 AM »
To make matters even more 'interesting', attached you find the same sections from the Quality Report of the other project we ran that day. The two projects were executed less than 3 hours apart, using the exact same camera settings. Likewise, lightning conditions were somewhat similar and this project TOO deals with some WB inconsistencies due to White Balancing set to auto.

This time, the camera residuals indicate a more 'traditional' radial pattern that we have seen before. Likewise, this time the grid of camera positions actually DOES align with the underlying overlap figure. It is worth nothing though, however, that this project deals with somewhat similar issues and outputted accuracies are appalling (several meters).

Yoann Courtois

  • Sr. Member
  • ****
  • Posts: 316
  • Engineer in Geodesy, Cartography and Surveying
    • View Profile
Re: *HELP* Poor alignment, massive GCP errors, useless endproducts
« Reply #9 on: September 08, 2017, 12:28:14 PM »
Hi !

Wow ... It looks something went really wrong !

- Image quality: OK (But still, all pictures with low quality factor are bad, but the reverse is not true)
- WB: Well it could reduce a bit the quality of the photogrammetric production, but not that much !

I will only focus on the first processing report, dealing with both your flights:
- On the fig. 1, it looks there is a shift between image overlap and camera locations. Was the camera looking perfectly down ? You should unchecked all camera coordinates in the reference panel, and use generic preselection instead of reference.
- Reprojection error should be less than 1 pix and you got 2.87 pix. It is a major problem !
- On the fig. 2, the scale is 5 pix ! There is indeed a massive shit that can also be highlight in the calibration values.

I see on your last post that you have no precalibrated cameras. It's a bit sad with such camera !
As this flight gave you quite well result, I would suggest you to adjusted it as best you can, and then save the calibration to be imported as precalibrated parameters in your project that failed.

In a simple way, the main problem really come from alignment step. So you should try again with precalibrated parameters and WITHOUT camera coordinates !

Regards

P.S. Is there any possibility to share your set of picture ?
--
Yoann COURTOIS
R&D Engineer in photogrammetric process and mobile application
Lyon, FRANCE
--

Yoann Courtois

  • Sr. Member
  • ****
  • Posts: 316
  • Engineer in Geodesy, Cartography and Surveying
    • View Profile
Re: *HELP* Poor alignment, massive GCP errors, useless endproducts
« Reply #10 on: September 08, 2017, 12:34:22 PM »
That's exactly what I thought as well, until I gave it a try nonetheless. After merging the two seperate chunks there would be a significant vertical shift between the point clouds making up each (i.e. several meters). After re-optimizing camera's in the merged chunk this vertical shift was lowered significantly, although XYZ accuracy remained appalling. This is not to say that I believe optimization after merging helps, but rather that it does alter things significantly.

Why is there a significant vertical shift if both flight were georeferenced onto the same GCPs  ??? ??? ???
--
Yoann COURTOIS
R&D Engineer in photogrammetric process and mobile application
Lyon, FRANCE
--

BobvdMeij

  • Full Member
  • ***
  • Posts: 157
    • View Profile
Re: *HELP* Poor alignment, massive GCP errors, useless endproducts
« Reply #11 on: September 08, 2017, 12:50:38 PM »
Wow ... It looks something went really wrong !
I'm glad you agree ;)

Quote
- On the fig. 1, it looks there is a shift between image overlap and camera locations. Was the camera looking perfectly down ? You should unchecked all camera coordinates in the reference panel, and use generic preselection instead of reference.
Yes they were all shot in nadir, with very minor deviations obviously. I Aligned the set with different parameters, with the images turned on AND off, with preselection on AND off, generic preselection on AND off. None, strangely enough, produced satisfactory outputs though.

Quote
- Reprojection error should be less than 1 pix and you got 2.87 pix. It is a major problem !

You are absolutely correct, we normally reach a reprojection error well below 0.5. Even without application of any form of Gradual Selection filtering! If we use Gradual Selection to filter out all TiePoints with a Reprojection Error over 1 we end up deleting more than 80% of our entire Sparse Pointcloud. That shouldn't be the case.

Quote
- On the fig. 2, the scale is 5 pix ! There is indeed a massive shit that can also be highlight in the calibration values.
What are you exactly referring to? I see no mention of 'scale' or '5 pix'.

Quote
I see on your last post that you have no precalibrated cameras. It's a bit sad with such camera !
As this flight gave you quite well result, I would suggest you to adjusted it as best you can, and then save the calibration to be imported as precalibrated parameters in your project that failed.
Does it make sense to import calibration parameters for a camera that has not been geometrically calibrated? What do you suggest? Open a previously ánd succesfully calibrated project with this camera and save/ export the camera calibration parameters (as acquried by PS during Alignment) and then import them to be used in this particular project? Would you then set these parameters to 'fixed' or 'initial'? I am a bit hestitant to fix camera calibration parameters of a consumer-grade camera that hasn't been properly geometrically calibrated in a laboratory.

Quote
P.S. Is there any possibility to share your set of picture ?
I'll see what I can do!

Quote
Why is there a significant vertical shift if both flight were georeferenced onto the same GCPs
You tell me. I am just as surprised (to say the very least!) as you are!

Yoann Courtois

  • Sr. Member
  • ****
  • Posts: 316
  • Engineer in Geodesy, Cartography and Surveying
    • View Profile
Re: *HELP* Poor alignment, massive GCP errors, useless endproducts
« Reply #12 on: September 08, 2017, 01:19:33 PM »
What are you exactly referring to? I see no mention of 'scale' or '5 pix'.

Please find attached your image residual graph with a 5 pix scale, and one of my last UAV survey with a 1 pix scale.

Does it make sense to import calibration parameters for a camera that has not been geometrically calibrated? What do you suggest? Open a previously ánd succesfully calibrated project with this camera and save/ export the camera calibration parameters (as acquried by PS during Alignment) and then import them to be used in this particular project? Would you then set these parameters to 'fixed' or 'initial'? I am a bit hestitant to fix camera calibration parameters of a consumer-grade camera that hasn't been properly geometrically calibrated in a laboratory.

Yes it does ! I'm not dealing with laboratory calibration for many reasons (hard to calibrate with the same focus at 40m, impossible to keep the same focus between two flights, hard to calibrate fisheye lens, and so on...)
But using correct adjusted camera parameters as precalibration will help PhotoScan during the estimation and the adjustment of the geometry (step 3 and 4 of SfM functioning). And it will help to converge to the correct values. Whereas each autocalibration start from 0 and it looks (if there is nothing else, which I have some doubts) this one didn't converge.
Results are really better using precalibrated parameters...and process goes faster !

To do it:
- Open the bigger and better project you did with the same camera (the most pictures, the more accurate internal parameters)
- Go to "Camera calibration" and save the adjusted values.
- Then each time you start a new PS project, go to "Camera calibration" and import those values in the initial panel.
- Don't fix the calibration ! (it always moves a bit, the focus is never the same, the precalibration is never perfect, and so on...)
- Go ahead !


I would be really interested by trying some test on your set of pictures. Every problem is a source of experience !
I would be glad if you could send me your two sets of pictures (both flights) be WeTransfer or something similar.
Here is my e-mail adress: yoann.courtois@insa-strasbourg.fr

Regards
--
Yoann COURTOIS
R&D Engineer in photogrammetric process and mobile application
Lyon, FRANCE
--