Depth Maps Turning Black with Large Drone Dataset

[jakobma]

Hello fellow Metashape users,

I'm dealing with a persistent issue in Metashape Professional 2.1.0 while processing a drone dataset of over 50,000 images captured using my Mavic 3 Enterprise. These images contain highly accurate RTK data, including a mix of low-altitude and handheld photos (4-5 thousand) alongside a considerable amount of oblique and nadir aerial shots (around 50k). I consistently use the highest and ultra settings for processing, yet I encounter a recurring issue with the depth maps.

Problem Description:
Throughout the dataset, the depth maps at a very specific location associated with images that overlap significantly either turn black or nearly black. Curiously, when I isolate this specific area (a pier) in a separate project with a smaller dataset but the same images, the results appear normal. Generally the model looks good at the alignment stage but falters during depth map generation. And those specific images that turn out black are the same images that don't align during the initial alignment, but then have a lot of matches at the end.

My Workflow:

• Verify image accuracy and quality.
• Create camera groups for all clusters of photos from one location / one flight / one type (low altitude, high altitude, nadir, etc.)
• Add all images to one chunk or just smaller sub-chunk regions and align with the following settings:
  
  • General Alignment Settings:
    
    • Accuracy: Highest
    • Generic preselection: Yes
    • Reference preselection: Disabled (I've disabled this because the dataset includes a mix of low-altitude and handheld photos which typically should align well with the many oblique aerial photos. My thought was, that this would have to be disabled, because the images should align with the aerial photos, but the distance might prevent them from doing so if enabled. However, I suspect this might also contribute to the issue as I've noticed some matches between completely unrelated photos across the entire island.)
    • Reset current alignment: No
  • Advanced Alignment Settings:
    
    • Key point limit: 90,000
    • Tic point limit: 6,000
    • Apply masks to: None
    • Exclude stationary tie points: Yes
    • Guided image matching: No
    • Adaptive camera model fitting: Yes
• Remove all poor tie points.
• Identify and realign any photos that didn't align properly initially (Usually didn't have to use this, but now many images with very obvious alignment possibility don't align...
• Either subdivide the chunk into smaller segments or use the Block Model approach to generate all depth maps once before building smaller meshes.

Model Building Settings:

• Source data: Depth maps
• Surface type: Arbitrary
• Depth maps quality: Ultra high
• Face count: High
• Advanced:
  
  • Depth filtering: Moderate
  • Interpolation: Enabled (default)
  • Point classes: All
  • Calculate vertex colors: No
  • Use strict volumetric masks: No
  • Reuse depth maps: No
  • Replace default model: No

Interestingly, no matter the settings I adjust, the same images consistently exhibit these issues. This problem persists even when I limit the processing area to just the pier. I tried also removing a lot of pictures from surrounding areas and just aligning a smaller subset, but it looks the same.

Attachments:

• Matches on a photo with a black depth map.
• Screenshot of the tie points at the location.
• Screenshot of the model and black depth maps for a specific point.

I hope to gain insights or suggestions on how to resolve this issue, as it appears consistently regardless of the varied settings or the scope of images processed.

Thank you in advance for any help you can provide!

[Bzuco]

Strange, did you try also only High alignment quality, which is 1:1 original photo resolution? I am not sure how many Mpix your photos have, but 90k key point limit coud be over the reasonable value and maybe in some case misleading for alignment algorithm... it depends on photo's resolution. In guided image matching option you can set key point limit per Mpix, so your photos would have more consistent amount of keypoints.

Did you saw some errors or strange unusual lines in console log during depth maps calculation?

[James]

Not sure if this is relevant, but see page 40 of the manual (if you haven't already):
https://www.agisoft.com/pdf/metashape-pro_2_1_en.pdf

Note

• Tie point limit parameter allows to optimize performance for the task and does not generally
  effect the quality of the further model. Recommended value is 10 000. Too high or too low tie
  point limit value may cause some parts of the point cloud model to be missed. The reason is that
  Metashape generates depth maps only for pairs of photos for which number of matching points is
  above certain limit. This limit equals to 100 matching points, unless moved up by the figure "10%
  of the maximum number of matching points between the photo in question and other photos,
  only matching points corresponding to the area within the bounding box being considered."
• The number of tie points can be reduced after the alignment process with Tie Points - Thin Point
  Cloud command available from Tools menu. As a results tie point cloud will be thinned, yet the
  alignment will be kept unchanged.

I can't quite get my head around what this really means, because if depth maps are generated for 'pairs of photos' then how is it that they appear to be assigned to individual photos in the UI, but without twisting my head around it too painfully I can imagine that maybe if a photo has an unreasonably large number of tie points with another photo, perhaps because it's actually taken from a very similar position, then that may raise the '10%' threshold too high for more suitable matching photos to be able to contribute to a proper depth map, or something.

So anyway, maybe a red herring, but if you're stuck you could try the suggestion in the note to use 'thin point cloud' to ensure a more uniform density of tie points to rule out this possibility.

[Alexey Pasumansky]

Hello jakobma,

How many tie points you are removing prior to the depth maps generation step?

Do you get better results, if you split the calibration groups prior to the processing according to each camera and flight used and run the image matching with High accuracy instead of Highest?

[jakobma]

Strange, did you try also only High alignment quality, which is 1:1 original photo resolution? I am not sure how many Mpix your photos have, but 90k key point limit coud be over the reasonable value and maybe in some case misleading for alignment algorithm... it depends on photo's resolution. In guided image matching option you can set key point limit per Mpix, so your photos would have more consistent amount of keypoints.

Did you saw some errors or strange unusual lines in console log during depth maps calculation?

Nothing out of the ordinary during depth maps calculation.

All pictures have 20MP and I have tried multiple values like 40k, 50k, 70k, 100k,....

[jakobma]

Not sure if this is relevant, but see page 40 of the manual (if you haven't already):
https://www.agisoft.com/pdf/metashape-pro_2_1_en.pdf

Note

• Tie point limit parameter allows to optimize performance for the task and does not generally
  effect the quality of the further model. Recommended value is 10 000. Too high or too low tie
  point limit value may cause some parts of the point cloud model to be missed. The reason is that
  Metashape generates depth maps only for pairs of photos for which number of matching points is
  above certain limit. This limit equals to 100 matching points, unless moved up by the figure "10%
  of the maximum number of matching points between the photo in question and other photos,
  only matching points corresponding to the area within the bounding box being considered."
• The number of tie points can be reduced after the alignment process with Tie Points - Thin Point
  Cloud command available from Tools menu. As a results tie point cloud will be thinned, yet the
  alignment will be kept unchanged.

I can't quite get my head around what this really means, because if depth maps are generated for 'pairs of photos' then how is it that they appear to be assigned to individual photos in the UI, but without twisting my head around it too painfully I can imagine that maybe if a photo has an unreasonably large number of tie points with another photo, perhaps because it's actually taken from a very similar position, then that may raise the '10%' threshold too high for more suitable matching photos to be able to contribute to a proper depth map, or something.

So anyway, maybe a red herring, but if you're stuck you could try the suggestion in the note to use 'thin point cloud' to ensure a more uniform density of tie points to rule out this possibility.

I also cannot get my head around this text. I can try thin point cloud!

[jakobma]

Hello jakobma,

How many tie points you are removing prior to the depth maps generation step?

Do you get better results, if you split the calibration groups prior to the processing according to each camera and flight used and run the image matching with High accuracy instead of Highest?

I am just removing tie points which are either floating hundreds of meters above the world (when it aligned clouds for example) or hundreds of meters beneath the sea surface for example.
Those tend to be a few thousand while I have over 28million tie points usually.

I usually split the photos into many calibration groups prior to aligning and right after importing. Sorted by location / flight too.

[jakobma]

Update:
I have found out that out of my 50k dataset, there were around 12,000 duplicate photos.

Usually I had different chunks for different photo folders and I have kept some photos twice in two folders since I wanted to have some overlap images. Of course, when importing all photos from all folders now, I would have differently named files with the exact same photo.

I used a duplicate checker and removed all duplicates and am trying alignment and model building on low now for a quick test.

The duplicate photos are from locations where the model was glitched. I assume that was the error, but am checking !!!

I'll keep you updated.
Thanks all