Forum

Author Topic: depthmap hole filling  (Read 1150 times)

HaydenMartin

  • Newbie
  • *
  • Posts: 23
    • View Profile
depthmap hole filling
« on: October 04, 2024, 10:01:28 PM »
https://github.com/atapour/depthComp

Something along these lines could be quite useful for for depthmap hole filling…

For example, slightly reflective or featureless shapes could potentially be accounted for during depthmap generation

There’s also DepthCrafter by TenCent which just came out, which whilst not physically accurate, had good temporal stability (sequential photo adherance)

Will we ever see some kind of enhancement like this implemented into Metashape?

HaydenMartin

  • Newbie
  • *
  • Posts: 23
    • View Profile
Re: depthmap hole filling
« Reply #1 on: October 08, 2024, 03:56:11 PM »
ok guess not!

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 15138
    • View Profile
Re: depthmap hole filling
« Reply #2 on: October 08, 2024, 04:57:56 PM »
Hello HaydenMartin,

Do you have any examples of the projects, where such feature may be helpful to improve the surface reconstruction results?
Best regards,
Alexey Pasumansky,
Agisoft LLC

HaydenMartin

  • Newbie
  • *
  • Posts: 23
    • View Profile
Re: depthmap hole filling
« Reply #3 on: October 08, 2024, 06:30:01 PM »
Hey Alex


Thanks for getting back to me


Like I mentioned earlier, featureless or inconsistent surfaces (like reflective glass/metal), typically appear as dead zones in depth map generation, or just introduce high amounts of noise.

Tools like TenCent's DepthCrafter have the ability to produce depth maps that are temporally stable (no flicker) and also are rarely negatively affected by reflections, etc...


So as an example, let's say we are having to work with 4K or 8K video frames, maybe for VFX purposes - we might only have a limited amount of data captured from the location rather than a full survey, so we have to try and make a reconstruction from just a couple of clips. These clips may have been filmed in a room with plain white walls. We are able to reconstruct *some* of the detail in the walls due to marks and texture, but there is a lot of data still missing in the depth maps. This is where a hole filling tool comes in handy, essentially content-aware filling in the depth maps based on the recognized scenery. Now we have full "complete" depth maps for every image which should hopefully combine together to make a clean consistent mesh even for featureless objects.


Another paper that just dropped was Apple's Depth Pro https://machinelearning.apple.com/research/depth-pro - Astonishingly detailed results. Whilst I don't know if it's temporally stable, it's amazing fidelity and accuracy...


In 2024, we really should have more advanced features to help us reclaim lost detail
« Last Edit: October 09, 2024, 11:38:58 AM by HaydenMartin »

HaydenMartin

  • Newbie
  • *
  • Posts: 23
    • View Profile
Re: depthmap hole filling
« Reply #4 on: October 09, 2024, 11:42:45 AM »
This just came out and  almost perfectly illustrates what I’m describing:

https://monst3r-project.github.io/

PolarNick

  • Full Member
  • ***
  • Posts: 103
    • View Profile
Re: depthmap hole filling
« Reply #5 on: October 09, 2024, 01:11:44 PM »
Hi, Martin!

1.1) Depth Pro - performs processing in 0.3 seconds on a Tesla V100 with 32 GB VRAM, and we're only talking about 2.25 megapixels here.
1.2) Metashape supports many video cards from different vendors (including those that do not support CUDA, including those with quite little VRAM), to process even large 100+ megapixel images as fast as possible, etc.

2.1) Such methods are well suited for VFX (bokeh effect, etc.), but poorly suited for photogrammetric tasks, that's why there are no measurements of accuracy in real units of measurement (centimeters/millimeters) in the article, besides such neural networks will not be able to solve the problem in unfamiliar conditions (for example, when shooting in a well, if there were no such frames in the training dataset).
2.2) Metashape relies on the laws of optics and geometry, which are universal and reliable in any environment (even underwater), have a certain predictability in accuracy and therefore allow the use of photogrammetry for construction/measurement tasks.

Indeed, ideas from such methods can be used, for example, to try to improve a depth map constructed by photogrammetric methods - by patching holes in it. But this requires a lot of work to bring an academic idea to a reliable industrial application, will slowdown processing a lot and have other problems. In particular, there is a risk that such a method will often spoil the depth map - by patching holes that are actually holes (as a mental experiment - for example, imagine a lattice in a window, and through the hole of this lattice you can see a perfectly clear sky - what is it? a hole in the lattice, or a white sheet of paper in the lattice?).

If you are talking about building depth maps from single frame (i.e. without taking parallax into account), then this is a task that should rather be solved in VFX programs for creating effects in video, while Metashape is focused on photogrammetry tasks.

HaydenMartin

  • Newbie
  • *
  • Posts: 23
    • View Profile
Re: depthmap hole filling
« Reply #6 on: October 09, 2024, 02:59:05 PM »
Thanks for the feedback!

I want to emphasize that photogrammetry is a key tool in VFX, and in this field absolute mm-precise accuracy is usually less critical than achieving passable visual results. VFX artists would benefit from more flexibility, and a tool like Agisoft could maybe help by introducing masking for hole-filling - this could allow users to designate areas for filling and exclude zones where precision is required, circumventing the very valid pitfall you described (I didn’t think of that!)

Instead of relying on heavy AI models like DepthCrafter, A custom Stable Diffusion LoRA or hypernetwork could fill holes in depth maps - these run effectively on RTX 30 series GPUs, offering a practical and resource-friendly way to improve outputs without needing crazy hardware.

This approach could really help those of us working in creative industries who don’t always need engineering-level accuracy but do need solid, reliable results.

HaydenMartin

  • Newbie
  • *
  • Posts: 23
    • View Profile
Re: depthmap hole filling
« Reply #7 on: October 29, 2024, 03:22:08 AM »
Any more thoughts on this?

Seeing more and more developments in depth estimation, pretty much weekly