Metashape 1.6.1 using CPU instead of GPU during depth maps based reconstruction?

[Mak11]

Alexey,

I'm encountering an issue in Metashape 1.6.1 with a project which is getting stuck exactly at the same stage each time (no way to finish it or even cancel it without terminating Metashape in Task Manager.

The project seems to switch to using the CPU halfway through during mesh reconstruction using Depth Maps and nothing happens besides the CPU being stuck at 14% usage with only one CPU Core working for 10 seconds (then another core for 10s, then another one etc..)

Same project worked fine in previous 1.6 builds.



UPDATE: Just ran another project and the exact same thing happened. This is while reconstruction on Ultra High settings. No issue on High. Doesn't make sense why the CPU is being invoked halfway through even though the kernels have just been loaded on the GPU (and CPU OpenCL is disabled in the settings tab)


Cheers

Mak

[Alexey Pasumansky]

Hello Mak,

Thank you for reporting.

Do you have "Use CPU" flag enabled or disabled in the Preferences window?

[Mak11]

Alexey,

No it's obviously disabled in the preference tab. Just tried another project and same thing happened half way through. This is on Ultra High settings. No issue when reconstructing on High.




Mak

[Alexey Pasumansky]

Hello Mak,

Thank you for providing the additional information.

We have checked the procedure and  the use of CPU may be involved, if the currently processed block cannot be allocated in VRAM, however, it should work in multithreaded mode. Unfortunately, the 1.6.1 build 10009 doesn't contain the multithreaded implementation for such CPU usage cases. However, if you would like to compare the behavior, I can send you a next version pre-release with the fix implemented.

[Mak11]

Alexey,

Thanks for the info. Feel free to send me the pre-release to my usual email address (or via PM here?).

Did anything change between 1.6.0 & 1.6.1 that causes this? Given that it worked before & that the GPU should have plenty of VRAM available (16GB)?

EDIT: In this case I have 13.8GB of VRAM available when this is happening in 1.6.1. While its perfectly fine in 1.6. It is also always happening around 50% progress no matter the project (tried 3 different projects and its always the same thing). Ironically the Metashape.exe process is actually using less RAM than there's free VRAM available..so something isn't right.




Cheers

Mak

[Alexey Pasumansky]

Hello Mak,

According to the log the AMD drivers returns the following information: "max mem alloc size = 4048", so the blocks of bigger size cannot be allocated, even if you have plenty of VRAM available.  It's quite a common case for AMD GPU though, having the max alloc size about 25% of total VRAM.

[Mak11]

Alexey,

But the same projects are all processing fine in Metashape 1.6 . 1.6.1 is generating bigger blocks ? Why ?

Mak

[Mak11]

Alexey,

I've now tested 10 diffèrent projects in 1.6.1 and only 1 was able to complete on Ultra High settings. Having to fall back on the CPU (even if it's multi-threaded in a future build) would be horrendous given that it will theorically be an order of a magnitude slower than on the GPU (unless the block can be computed concurrently on the CPU & GPU ?). I wouldn't mind testing the pre-release of the upcoming build though.

All those projects are computing fine in 1.6 where the block size never goes beyond 4048MB unlike 1.6.1. (I've attached the Log of the relevant section the 1.6 run of the same project as above)

Mak

[Mak11]

Alexey,

I've sent you a DM with my log of 1.6.2 pre-release.

Mak

[scanp_0]

Hello,

I've been having the same issue over multiple projects. Log is attached for one of them. Project has 432 cameras and 213k points (which is a thinned sparse cloud from the original 1.5million). UHigh, High as well as Medium depth map mesh generation fails around 60% and switches to CPU, which uses the amount of only one core. I've tried disabling my second gpu (1060gtx), "Use cpu" in prefs is off and Nvidia 441.87 drivers are installed (clean installation)..still no luck. 

[Mak11]

scanp_0,

The Dev team is working on fixing/modifying this behavior for the next release.

Mak

[Mak11]

Alexey,
What's the estimated release date of 1.6.2?

Mak

[Alexey Pasumansky]

Hello Mak,

1.6.2 is expected to be released next week.

[Mak11]

Alexey,

Thanks

Mak

[Alexey Pasumansky]

Hopefully the long processing time for the depth maps generation related to the additional corrections in calibration model (reported in another thread: https://www.agisoft.com/forum/index.php?topic=11722.0) will be also fixed in the upcoming version.