Agisoft Metashape

Agisoft Metashape => Bug Reports => Topic started by: jedfrechette on February 03, 2021, 12:37:05 AM

Title: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: jedfrechette on February 03, 2021, 12:37:05 AM
The current version of Metashape 1.7.1 build 11797 fails to use GPUs for depth map generation with the same settings that succeed with 1.6.5 build 11249. This was reported in the pre-release thread in the following post:

I can't seem to use GPU for depth map generation.  I get the following error:

Warning: Device GeForce GTX 1070 Ti ignored because it has 5958/8192 MB available, but 6870 MB required

We're using different GPUs (2080 Ti, driver version 461.40), but get the same error message, albeit with different VRAM values. Presumably, we're also using different size images than the original reporter. Nonetheless, opening the project files that produce this error in version 1.6.5 on the same machine allows us to utilize the GPU for depth map generation. Choosing a lower quality reconstruction in 1.7.1 also uses the GPUs for depth map generation.

The suggested workaround in the pre-release thread:

https://www.agisoft.com/forum/index.php?topic=12653.msg56367#msg56367

of setting main/depth_pm_max_neighbors = 40 does not seem to be possible any longer, however, from inspecting the logs it appears that processing is automatically limited to 40 neighbors in this version automatically.
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: Mak11 on February 03, 2021, 01:04:55 AM
jedfrechette,

Are you using the latest nvidia drivers ?

Mak
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: jedfrechette on February 03, 2021, 04:45:51 AM
Yes, as of this writing the latest WHQL Nvidia drivers are 461.40, which is what we are using.
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: Alexey Pasumansky on February 05, 2021, 12:18:39 AM
Hello Jed,

What is the resolution and bit depth of the images that you are using?

In the pre-releae thread I have posted the tweak that will considerably reduce the VRAM consumption if non 8-bit RGB imagery is used:
BuildDepthMaps/pm_convert_16u_to_8u
With this tweak set to True value, the images will be internally downsampled to 8-bit for the depth maps generation.
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: jedfrechette on February 05, 2021, 09:51:32 AM
They are 40-60 megapixel half-float linear exrs so converting to 8 bit internally makes sense. I'll try the tweak and report back.

One question about the 8 bit conversion. Do I need to worry about the range of values in the exrs? Our images use a pretty conventional scale, normalized so that an 18% gray card would have a value of 0.18. The majority of values will be in the range 0-1.0 but it's not uncommon for some values to be greater than 1 depending on the scene and exposure level. In other places inside Metashape where the exrs get converted to 8-bit, e.g. thumbnails, point cloud colors, and view port texture maps, the images appear badly overexposed. We generally need to use Tools -> Set Brightness to compensate for this.  Will pm_convert_16u_to_8u do the color transform correctly to deal with the dynamic range and scene-linear input data?

Thanks,
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: jedfrechette on February 06, 2021, 03:50:03 AM
Adding the tweak didn't help. Meshing still fails with the following error message on my machine:

Code: [Select]
Warning: Device Geforce RTX 2080 TI ignored because it has 8929/11264 MB available, but 10249 MB required.
The amount of required memory was not decreased by adding the pm_convert_16u_to_8u tweak. Is that because the images I'm using are 16-bit float not 16-bit uint images?
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: wojtek on February 06, 2021, 01:37:27 PM
Adding the tweak didn't help. Meshing still fails with the following error message on my machine:

Code: [Select]
Warning: Device Geforce RTX 2080 TI ignored because it has 8929/11264 MB available, but 10249 MB required.
The amount of required memory was not decreased by adding the pm_convert_16u_to_8u tweak. Is that because the images I'm using are 16-bit float not 16-bit uint images?

In my case it also depended on what I was meshing, can you show your camera alignment?
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: jedfrechette on February 07, 2021, 01:34:57 AM
In my case it also depended on what I was meshing, can you show your camera alignment?

I imagine VRAM usage also depends on how much overlap there is between images? Most of my datasets have a very high degree of overlap and judging by the logs are almost always hitting the maximum of 40 neighbors.

Attached is a screenshot of a test data set I shot using a 24 megapixel camera. With this data set I'm able to trigger the error on an older machine with a 6 GB GPU by attempting a reconstruction at Ultra High Resolution.
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: Matioupi on February 16, 2021, 01:48:13 PM
same behaviour here with heavy phase one images. Whatever VRAM the GPU has, the software alway try to allocate too much. This fail, and then back to CPU.
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: Alexey Pasumansky on February 17, 2021, 05:42:32 PM
Hello Matioupi,

can you share the related log from the depth maps generation stage?
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: Matioupi on February 18, 2021, 11:10:14 AM
Hello, I attached a screenshot of the messages in the monitor. Do you need more ? (it may take a little time to be able to produce more detailled logs)
Images come from airborne mission with PhaseOne MPix (14204x10652) jpeg images. Overlap is about 80% sidelap and alongside.

The   red message is similar on all nodes, whatever the GPU, even when they have less / more VRAM, the system always try to allocate slightly more RAM than the GPU has.
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: Matioupi on February 27, 2021, 09:16:40 PM
Any news on this ? do you need more details logs or the previous screenshot gives enought hints ?
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: Alexey Pasumansky on February 27, 2021, 09:56:03 PM
Hello Matioupi,

The provided information is sufficient.

We are currently trying to lower the VRAM consumption for the depth maps generation in 1.7 (especially for the high-resolution images).
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: DavidD on March 03, 2021, 04:51:15 PM
We're having the same issue with our 100MP images. It seems the VRAM required is always just above what is available...
Our images are 8bit by the way. Current fallback solution is using 1.6.5 but I'd clearly prefer using 1.7.x with the new algorithm. The speed improvements are impressive!
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: Alexey Pasumansky on March 03, 2021, 06:27:04 PM
Hello DavidD,

Which GPU are you using and how big is the difference between available VRAM and required for the processing?
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: DavidD on March 03, 2021, 09:21:49 PM
In our workstation there is a 2080 and a 2080 super, both have 8 Gigabyte of VRAM.
I currently have the job running with Metashape 1.6.5, but once it's done I can reproduce the error in 1.7.1 and send you the logs and exact numbers of required RAM.

The project contains ~1800 images with 100 Megapixels each.
When I start the 'build DEM from Depth Maps' processing (in 1.7.1) it fails right away - interestingly It didn't matter if I had the ultra setting or high or medium. It always happended.
When I removed a part of the images the error did not occur right away. It did start to work fine and used both GPUs. However over time during the night the VRAM consumption grew and at some point it was saturated again and defaulted back to the CPU processing.
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: DavidD on March 05, 2021, 12:14:46 AM
I attached the logs of a recent run of a smaller project. This contains only 45 cameras but one of the GPU fails anyways.
This runs in ultra settings.
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: Alexey Pasumansky on March 05, 2021, 12:25:53 AM
Hello DavidD,

We are working on new depth maps generation procedure improvement in order to optimize the VRAM consumption and avoid CPU fallbacks (at least on GPUs with 4+ GB VRAM).

As a workaround you can reduce the max_neighbors limit for the depth maps via tweaks, or see, if any external applications can be closed, that are using extra VRAM on 2080 Super GPU. You can use NVIDIA SMI to see, which processes are using GPU.
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: DavidD on March 09, 2021, 07:36:59 PM
Perfect, glad to hear that! Looking forward to the next point release then! You probably can't give an ETA right now, can you?
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: sapere-aude on March 14, 2021, 04:18:21 AM
Hello DavidD,

We are working on new depth maps generation procedure improvement in order to optimize the VRAM consumption and avoid CPU fallbacks (at least on GPUs with 4+ GB VRAM).

As a workaround you can reduce the max_neighbors limit for the depth maps via tweaks, or see, if any external applications can be closed, that are using extra VRAM on 2080 Super GPU. You can use NVIDIA SMI to see, which processes are using GPU.

Hi,

I've noticed something about this bug on my machine.

If I align a set of images, and then try to build the dense cloud afterwards I get the error even though my GPU have more ram than the dense cloud generation requires. If I then cancel the dense cloud generation and save my project, and then reboot the computer, open the project again and try to build the dense cloud it works!

It seems like the issue is related to clearing the VRAM or perhaps the program for some reason thinks we have less free VRAM than we actually have. Rebooting obviously clears the VRAM.

Just thought I'd mention this.

My PC:
1660 ti
64 GB ram
Ryzen 5 2600X

PS: Will the planned optimizatiom of the VRAM consumption reduce the quality/accuracy of the depth maps/dense cloud and/or texture? Converting the images to 8-bit obviously discards information, no? Quality is one of the strong points of Metashape, please now and in the future don't make changes that sacrifice quality over speed unless we can also override it! Thanks!
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: Alexey Pasumansky on March 17, 2021, 03:33:10 PM
Perfect, glad to hear that! Looking forward to the next point release then! You probably can't give an ETA right now, can you?
Hello David,

Hopefully, the VRAM consumption optimization will be included in 1.7.3 release.
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: itzhalpepe on March 31, 2021, 08:43:44 PM
Hello Alexey,
we have the same issue, that Agisoft ignore our GPU during depth maps processing.

We are working with 6 RTX 2080 TI.

Warning: Device GeForce RTX 2080 Ti ignored because it has 8177/11264 MB available, but 56739 MB required
Warning: Device GeForce RTX 2080 Ti ignored because it has 7992/11264 MB available, but 56739 MB required
Warning: Device GeForce RTX 2080 Ti ignored because it has 8177/11264 MB available, but 56739 MB required
Warning: Device GeForce RTX 2080 Ti ignored because it has 8177/11264 MB available, but 56739 MB required
Warning: Device GeForce RTX 2080 Ti ignored because it has 8177/11264 MB available, but 56739 MB required
Warning: Device GeForce RTX 2080 Ti ignored because it has 8177/11264 MB available, but 56739 MB required

How we can fix that problem? Without the GPUs, the processing time increased by the factor 5.

/Leon
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: Alexey Pasumansky on March 31, 2021, 09:43:19 PM
Hello Leon,

We are in process of GPU memory utilization. I have sent the pre-release version of 1.7.3 to a few users from this thread, but it seems that some other problem has appeared. Once we fix it, I will update the pre-release links. If you wish, then I can also send it to you later.

Meanwhile, please specify the dimensions of the images that you are working with, number of bands and bit depth for each band.
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: itzhalpepe on March 31, 2021, 09:50:21 PM
Hello Alexey,
that would be great.

I am working with around 600 images with a resolution of 26000 x 17000 pixel. Each image has 4 bands with 16bit.
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: Alexey Pasumansky on April 01, 2021, 05:47:35 PM
Hello Leon,

I have just sent the link to the pre-release of 1.7.3 via PM. Hopefully it would work as expected now. Would be interesting to compare the log for the same project, where 56739 MB were required.
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: Alexey Pasumansky on April 02, 2021, 09:39:33 PM
Hello,

Here are the links for public pre-release 1.7.3:

Professional edition:
http://download.agisoft.com/metashape-pro_1_7_3_x64.msi 
http://download.agisoft.com/metashape-pro_1_7_3.dmg 
http://download.agisoft.com/metashape-pro_1_7_3_amd64.tar.gz 

Standard edition:
http://download.agisoft.com/metashape_1_7_3_x64.msi 
http://download.agisoft.com/metashape_1_7_3.dmg 
http://download.agisoft.com/metashape_1_7_3_amd64.tar.gz 

Updated to build 12337.
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: RHenriques on April 24, 2021, 02:17:24 AM
Hi Alexey

The generation of a mesh from depth maps remains problematic. Here is an image with the results from the classic dense cloud approach and with mesh from depth maps.
The test was done in 1.7.3 - 12248.
Best regards
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: Alexey Pasumansky on April 24, 2021, 01:28:30 PM
Hello Renato,

Can you share the sample dataset (project with the alignment results and source images) to support@agisoft.com for our internal testing?
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: RHenriques on April 24, 2021, 11:35:13 PM
Hi Alexey

Just sent a PCloud invitation to support@agisoft.com with the acesso to the RAW photos used. The work is splitted in two sessions called SetA and SetB that must be merged in the same chunk.

Meanwhile, I'm starting the same project from the ground up in the latest version. The previous test was made from a project started in the last version. Judging by the sparse cloud, it seems that the latest beta produces a better alignment. I'm going to see how the rest of the processing goes.


PS: Depth maps seems to work great now! I suspect that using a previous project might have injected bias in the mesh generation in the latest version. If we start up fresh, things seem to be right now (a lot!!). Now I'll try to test Strict Volumetric Masks.

PS2: From the successful previous project, I've used the generate masks command. Masks were generated perfectly. After tried to produce the mesh using Strict Volumetric Masks - very incomplete mesh. The problem with SVM still remains.

PS3: Despite SVM not working well, the Mesh generation from Depth Maps (without SVM activated) seems to have improved a lot. Mesh is more detailed and sharpen now. I've already tested in two complex projects and it seems to be consistently better than in previous versions. 
 

Best Regards


Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: Alexey Pasumansky on April 27, 2021, 02:26:52 PM
Hello Renato,

If the masks are not accurate and cover the parts of the actual object, using them for Strict Volumetric Masking will lead to the missing parts of the object, as Volumetric Masks defines the regions where mesh cannot be generated.
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: RHenriques on April 27, 2021, 05:49:27 PM
Hi Alexey

I was careful enough to increase the depth value to 5 and did not select the blurred areas option. This way the masks were perfectly and accurately generated around the object of interest, without cropping nothing important. Only the more distant background was eliminated. Even so, in the object from the previous post (Rock with cross on top), by using SVM enabled, only the upper part of the cross appeared in the model. The rest were indistinguishable small fragments. See if you can download the RAW images and replicate. However, as I said in the previous post, the process os generating a mesh from depth maps is now a lot better, without strange crops in the geometry, and it seems more sharpen and accurate than before.
Best Regards
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: Andreas1206 on May 07, 2021, 07:44:31 PM
Hello Alexey,

the latest build 12337 of 1.73 imediately breaks down and closes with a bug report window, when importing .dng files. JPEG and TIFF files work.

best regards,
Andreas
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: Alexey Pasumansky on May 08, 2021, 03:15:59 PM
Hello Andreas,

Thank you for reporting. The problem will be fixed in the next update.
Title: Re: 1.7.1 fails to build depth maps with same amount of VRAM as 1.6.5
Post by: Eric Baird on September 24, 2021, 12:16:03 AM
I was having the same issue with 1.7.3. With a GPU card with 2GB memory (Quadro K620), sometimes a sizeable proportion of the memory would report as unavailable midway through processing, causing a fallback to CPU processing. My workaround was to go back to using 1.6.6 which didn't show have the problem. Unfortunately, once 1.7.3 has modified a project file, 1.6.6 can have trouble reading it.