Depth Map Generation Time

[CPryor_YT]

Hello All,

I'm a long time user but a new poster to the forum.  I have been using Metashape for a number of years to process drone imagery datasets of generally 300 to 1500 images.  I have had no problems and processing times have seemed on par when considering computing power.

We have recently build a new processing computer that was primarily put together for processing large LiDAR data sets but I thought it would do okay for processing Photogrammetry data sets as well.

Computer Specs:
Pro WS WRX80E-SAGE SE WIFI Board
AMD Ryzen Threadripper Pro 5975WX 32-core CPU
256 GBytes DDR4-3200 (1600 MHz) Kingston  RAM
NVIDIA GeForce RTX 3090
I have not disabled SMT

Project Specs
4588 Images
432 MB Each
14204X10652 Pixels
Overlap was flown at 85% along track and 85% across track

Depth Map generation is estimated at 13 days!
Process has currently been running for 2 days.

This doesn't seem reasonable to me....or maybe I'm missing something.  Does anyone have any suggestions on what could be going on here?

Thanks,

Cort

[Bzuco]

Hi Cort,

those Images are pretty big. Can you check console log during processing, if your GPU is receiving depth maps tasks?

You should see something about peak VRAM usage:
e.g.  "group 1/1: expected peak VRAM usage: 619 MB (225 MB max alloc, 4864x5472 mipmap texture, 16 max neighbors)"

...and plenty of lines starting with:
[GPU1] Camera #...
[GPU2] group 1/1...

Maybe there are some limits which needs to be tweaked in preferences, otherwise your big depth maps are calculated on CPU only...
You can also check how is your GPU utilized...at this size of images you should see all the time @100% if everything is OK.

[CPryor_YT]

Hey Bzuco....Yes they are large they were collected with a Phase One iXM-RS150F.

My GPU is receiving the depth maps tasks and I set the OpenGL rendering GPU to the NVIDIA card in the NVIDIA Control Panel 3D settings=>Program Setting and set the Power Management Mode to maximum but it is only being utilized up to a maximum of 66% (Typically between 8% and 14% according to the MSI Afterburner hardware monitor and the task manager.

I am seeing the lines that you reference in the console and the process is moving forward....just extremely slow.

CPU usage is running between 23% and 97%.

What limits are you referring to that may need to be tweaked?

Thanks!

[Bzuco]

Can you check disk read/write and differences in RAM memory usage? CPU temps. are OK?..it is 280W processor.
CPU is quite heavily utilized during depth maps gen. even on 18Mpix JPEGs and 8core CPU. Maybe 32c threadripper is still not enough for RTX3090 when preparing 151Mpix photos.
I was thinking about some limits of allocating VRAM, but if GPU is receiving data, then it is probably OK, and no tweaks are necessary(not enough knowledge in this area).
Can you post log? maybe there will be some useful info for devs.

EDIT: what frequency is your CPU able keep? it should be around 3.6GHz

[Alexey Pasumansky]

Hello Cort,

Can you please provide the processing log from the beginning of the task (or at least the current part, if the log is not saved to the external file) and also specify the processing task parameters and Metashape version used.

[CPryor_YT]

Alexey,

I have attached a portion of the processing log (file is large).  I have noted some strange behavior.....I canceled the process and restarted and immediately the the GPU started processing at 80% to 90%, twice as much utilization as previous.  I canceled task and restarted and GPU then was only utilized at the 20% to 40% level.  CPU consistently utilized between 70% and 100% and read/write on the NVME RAID 0 volume seems normal.  I am not having any problem with GPU or CPU temps.

MetaShape Professional 1.8.4
Build Dense Cloud:
     Quality High
     Depth Filtering Mild
     Caculate Point Colors
     Calculate Point Confidence

Curious to hear your thoughts.

[Bzuco]

Just discovered that newer task manager in win8/10 seems to not correctly monitor CPU utilization. It shows much higher % values than in other monitoring tools like old win7 task manager or HWiNFO / core temp utils.
The difference is 25-30%.
Maybe newer TM does not count virtual threads HT/SMT. Monitoring GPU seems to be correct.

@CPryor_YT
Are your times now better when you restarted the process, or still the same?
Did you just stop/start depth maps calculation, or restarted metashape or whole PC?