Forum

Author Topic: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5  (Read 5695 times)

DavidD

  • Newbie
  • *
  • Posts: 21
    • View Profile
Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
« Reply #15 on: March 03, 2021, 09:21:49 PM »
In our workstation there is a 2080 and a 2080 super, both have 8 Gigabyte of VRAM.
I currently have the job running with Metashape 1.6.5, but once it's done I can reproduce the error in 1.7.1 and send you the logs and exact numbers of required RAM.

The project contains ~1800 images with 100 Megapixels each.
When I start the 'build DEM from Depth Maps' processing (in 1.7.1) it fails right away - interestingly It didn't matter if I had the ultra setting or high or medium. It always happended.
When I removed a part of the images the error did not occur right away. It did start to work fine and used both GPUs. However over time during the night the VRAM consumption grew and at some point it was saturated again and defaulted back to the CPU processing.

DavidD

  • Newbie
  • *
  • Posts: 21
    • View Profile
Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
« Reply #16 on: March 05, 2021, 12:14:46 AM »
I attached the logs of a recent run of a smaller project. This contains only 45 cameras but one of the GPU fails anyways.
This runs in ultra settings.

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13150
    • View Profile
Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
« Reply #17 on: March 05, 2021, 12:25:53 AM »
Hello DavidD,

We are working on new depth maps generation procedure improvement in order to optimize the VRAM consumption and avoid CPU fallbacks (at least on GPUs with 4+ GB VRAM).

As a workaround you can reduce the max_neighbors limit for the depth maps via tweaks, or see, if any external applications can be closed, that are using extra VRAM on 2080 Super GPU. You can use NVIDIA SMI to see, which processes are using GPU.
Best regards,
Alexey Pasumansky,
Agisoft LLC

DavidD

  • Newbie
  • *
  • Posts: 21
    • View Profile
Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
« Reply #18 on: March 09, 2021, 07:36:59 PM »
Perfect, glad to hear that! Looking forward to the next point release then! You probably can't give an ETA right now, can you?

sapere-aude

  • Newbie
  • *
  • Posts: 1
    • View Profile
Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
« Reply #19 on: March 14, 2021, 04:18:21 AM »
Hello DavidD,

We are working on new depth maps generation procedure improvement in order to optimize the VRAM consumption and avoid CPU fallbacks (at least on GPUs with 4+ GB VRAM).

As a workaround you can reduce the max_neighbors limit for the depth maps via tweaks, or see, if any external applications can be closed, that are using extra VRAM on 2080 Super GPU. You can use NVIDIA SMI to see, which processes are using GPU.

Hi,

I've noticed something about this bug on my machine.

If I align a set of images, and then try to build the dense cloud afterwards I get the error even though my GPU have more ram than the dense cloud generation requires. If I then cancel the dense cloud generation and save my project, and then reboot the computer, open the project again and try to build the dense cloud it works!

It seems like the issue is related to clearing the VRAM or perhaps the program for some reason thinks we have less free VRAM than we actually have. Rebooting obviously clears the VRAM.

Just thought I'd mention this.

My PC:
1660 ti
64 GB ram
Ryzen 5 2600X

PS: Will the planned optimizatiom of the VRAM consumption reduce the quality/accuracy of the depth maps/dense cloud and/or texture? Converting the images to 8-bit obviously discards information, no? Quality is one of the strong points of Metashape, please now and in the future don't make changes that sacrifice quality over speed unless we can also override it! Thanks!

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13150
    • View Profile
Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
« Reply #20 on: March 17, 2021, 03:33:10 PM »
Perfect, glad to hear that! Looking forward to the next point release then! You probably can't give an ETA right now, can you?
Hello David,

Hopefully, the VRAM consumption optimization will be included in 1.7.3 release.
Best regards,
Alexey Pasumansky,
Agisoft LLC

itzhalpepe

  • Newbie
  • *
  • Posts: 13
    • View Profile
Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
« Reply #21 on: March 31, 2021, 08:43:44 PM »
Hello Alexey,
we have the same issue, that Agisoft ignore our GPU during depth maps processing.

We are working with 6 RTX 2080 TI.

Warning: Device GeForce RTX 2080 Ti ignored because it has 8177/11264 MB available, but 56739 MB required
Warning: Device GeForce RTX 2080 Ti ignored because it has 7992/11264 MB available, but 56739 MB required
Warning: Device GeForce RTX 2080 Ti ignored because it has 8177/11264 MB available, but 56739 MB required
Warning: Device GeForce RTX 2080 Ti ignored because it has 8177/11264 MB available, but 56739 MB required
Warning: Device GeForce RTX 2080 Ti ignored because it has 8177/11264 MB available, but 56739 MB required
Warning: Device GeForce RTX 2080 Ti ignored because it has 8177/11264 MB available, but 56739 MB required

How we can fix that problem? Without the GPUs, the processing time increased by the factor 5.

/Leon

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13150
    • View Profile
Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
« Reply #22 on: March 31, 2021, 09:43:19 PM »
Hello Leon,

We are in process of GPU memory utilization. I have sent the pre-release version of 1.7.3 to a few users from this thread, but it seems that some other problem has appeared. Once we fix it, I will update the pre-release links. If you wish, then I can also send it to you later.

Meanwhile, please specify the dimensions of the images that you are working with, number of bands and bit depth for each band.
Best regards,
Alexey Pasumansky,
Agisoft LLC

itzhalpepe

  • Newbie
  • *
  • Posts: 13
    • View Profile
Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
« Reply #23 on: March 31, 2021, 09:50:21 PM »
Hello Alexey,
that would be great.

I am working with around 600 images with a resolution of 26000 x 17000 pixel. Each image has 4 bands with 16bit.

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13150
    • View Profile
Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
« Reply #24 on: April 01, 2021, 05:47:35 PM »
Hello Leon,

I have just sent the link to the pre-release of 1.7.3 via PM. Hopefully it would work as expected now. Would be interesting to compare the log for the same project, where 56739 MB were required.
Best regards,
Alexey Pasumansky,
Agisoft LLC

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13150
    • View Profile
« Last Edit: May 06, 2021, 10:16:09 PM by Alexey Pasumansky »
Best regards,
Alexey Pasumansky,
Agisoft LLC

RHenriques

  • Full Member
  • ***
  • Posts: 197
    • View Profile
Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
« Reply #26 on: April 24, 2021, 02:17:24 AM »
Hi Alexey

The generation of a mesh from depth maps remains problematic. Here is an image with the results from the classic dense cloud approach and with mesh from depth maps.
The test was done in 1.7.3 - 12248.
Best regards

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13150
    • View Profile
Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
« Reply #27 on: April 24, 2021, 01:28:30 PM »
Hello Renato,

Can you share the sample dataset (project with the alignment results and source images) to support@agisoft.com for our internal testing?
Best regards,
Alexey Pasumansky,
Agisoft LLC

RHenriques

  • Full Member
  • ***
  • Posts: 197
    • View Profile
Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
« Reply #28 on: April 24, 2021, 11:35:13 PM »
Hi Alexey

Just sent a PCloud invitation to support@agisoft.com with the acesso to the RAW photos used. The work is splitted in two sessions called SetA and SetB that must be merged in the same chunk.

Meanwhile, I'm starting the same project from the ground up in the latest version. The previous test was made from a project started in the last version. Judging by the sparse cloud, it seems that the latest beta produces a better alignment. I'm going to see how the rest of the processing goes.


PS: Depth maps seems to work great now! I suspect that using a previous project might have injected bias in the mesh generation in the latest version. If we start up fresh, things seem to be right now (a lot!!). Now I'll try to test Strict Volumetric Masks.

PS2: From the successful previous project, I've used the generate masks command. Masks were generated perfectly. After tried to produce the mesh using Strict Volumetric Masks - very incomplete mesh. The problem with SVM still remains.

PS3: Despite SVM not working well, the Mesh generation from Depth Maps (without SVM activated) seems to have improved a lot. Mesh is more detailed and sharpen now. I've already tested in two complex projects and it seems to be consistently better than in previous versions. 
 

Best Regards


« Last Edit: April 26, 2021, 06:40:14 PM by RHenriques »

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 13150
    • View Profile
Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
« Reply #29 on: April 27, 2021, 02:26:52 PM »
Hello Renato,

If the masks are not accurate and cover the parts of the actual object, using them for Strict Volumetric Masking will lead to the missing parts of the object, as Volumetric Masks defines the regions where mesh cannot be generated.
Best regards,
Alexey Pasumansky,
Agisoft LLC