1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5

[jedfrechette]

The current version of Metashape 1.7.1 build 11797 fails to use GPUs for depth map generation with the same settings that succeed with 1.6.5 build 11249. This was reported in the pre-release thread in the following post:

I can't seem to use GPU for depth map generation.  I get the following error:

Warning: Device GeForce GTX 1070 Ti ignored because it has 5958/8192 MB available, but 6870 MB required

We're using different GPUs (2080 Ti, driver version 461.40), but get the same error message, albeit with different VRAM values. Presumably, we're also using different size images than the original reporter. Nonetheless, opening the project files that produce this error in version 1.6.5 on the same machine allows us to utilize the GPU for depth map generation. Choosing a lower quality reconstruction in 1.7.1 also uses the GPUs for depth map generation.

The suggested workaround in the pre-release thread:

https://www.agisoft.com/forum/index.php?topic=12653.msg56367#msg56367

of setting main/depth_pm_max_neighbors = 40 does not seem to be possible any longer, however, from inspecting the logs it appears that processing is automatically limited to 40 neighbors in this version automatically.

[Mak11]

jedfrechette,

Are you using the latest nvidia drivers ?

Mak

[jedfrechette]

Yes, as of this writing the latest WHQL Nvidia drivers are 461.40, which is what we are using.

[Alexey Pasumansky]

Hello Jed,

What is the resolution and bit depth of the images that you are using?

In the pre-releae thread I have posted the tweak that will considerably reduce the VRAM consumption if non 8-bit RGB imagery is used:
BuildDepthMaps/pm_convert_16u_to_8u
With this tweak set to True value, the images will be internally downsampled to 8-bit for the depth maps generation.

[jedfrechette]

They are 40-60 megapixel half-float linear exrs so converting to 8 bit internally makes sense. I'll try the tweak and report back.

One question about the 8 bit conversion. Do I need to worry about the range of values in the exrs? Our images use a pretty conventional scale, normalized so that an 18% gray card would have a value of 0.18. The majority of values will be in the range 0-1.0 but it's not uncommon for some values to be greater than 1 depending on the scene and exposure level. In other places inside Metashape where the exrs get converted to 8-bit, e.g. thumbnails, point cloud colors, and view port texture maps, the images appear badly overexposed. We generally need to use Tools -> Set Brightness to compensate for this.  Will pm_convert_16u_to_8u do the color transform correctly to deal with the dynamic range and scene-linear input data?

Thanks,

[jedfrechette]

Adding the tweak didn't help. Meshing still fails with the following error message on my machine:

Code: [Select]

Warning: Device Geforce RTX 2080 TI ignored because it has 8929/11264 MB available, but 10249 MB required.
The amount of required memory was not decreased by adding the pm_convert_16u_to_8u tweak. Is that because the images I'm using are 16-bit float not 16-bit uint images?

[wojtek]

Adding the tweak didn't help. Meshing still fails with the following error message on my machine:

Code: [Select]

Warning: Device Geforce RTX 2080 TI ignored because it has 8929/11264 MB available, but 10249 MB required.
The amount of required memory was not decreased by adding the pm_convert_16u_to_8u tweak. Is that because the images I'm using are 16-bit float not 16-bit uint images?

In my case it also depended on what I was meshing, can you show your camera alignment?

[jedfrechette]

In my case it also depended on what I was meshing, can you show your camera alignment?

I imagine VRAM usage also depends on how much overlap there is between images? Most of my datasets have a very high degree of overlap and judging by the logs are almost always hitting the maximum of 40 neighbors.

Attached is a screenshot of a test data set I shot using a 24 megapixel camera. With this data set I'm able to trigger the error on an older machine with a 6 GB GPU by attempting a reconstruction at Ultra High Resolution.

[Matioupi]

same behaviour here with heavy phase one images. Whatever VRAM the GPU has, the software alway try to allocate too much. This fail, and then back to CPU.

[Alexey Pasumansky]

Hello Matioupi,

can you share the related log from the depth maps generation stage?

[Matioupi]

Hello, I attached a screenshot of the messages in the monitor. Do you need more ? (it may take a little time to be able to produce more detailled logs)
Images come from airborne mission with PhaseOne MPix (14204x10652) jpeg images. Overlap is about 80% sidelap and alongside.

The   red message is similar on all nodes, whatever the GPU, even when they have less / more VRAM, the system always try to allocate slightly more RAM than the GPU has.

[Matioupi]

Any news on this ? do you need more details logs or the previous screenshot gives enought hints ?

[Alexey Pasumansky]

Hello Matioupi,

The provided information is sufficient.

We are currently trying to lower the VRAM consumption for the depth maps generation in 1.7 (especially for the high-resolution images).

[DavidD]

We're having the same issue with our 100MP images. It seems the VRAM required is always just above what is available...
Our images are 8bit by the way. Current fallback solution is using 1.6.5 but I'd clearly prefer using 1.7.x with the new algorithm. The speed improvements are impressive!

[Alexey Pasumansky]

Hello DavidD,

Which GPU are you using and how big is the difference between available VRAM and required for the processing?