Slow pointcloud generation on nVidia Tesla K80 with Metashape v.1.5.0

[jansawicki]

Hello,

Me and my colleagues noticed a significant slowdown in pointcloud processing since we updated to Metashape v.1.5.0
We used to work on Photoscan 1.4.2 which as far as we remember worked a lot faster, but we also tried downgrading back to this version and it doesn't appear to be any faster...


Workstation:
OS Windows Server 2016 (10.0.14393)
CPU: Intel Xeon E5-2620 v4 @ 2.1GHz
128GB RAM
GPU: 2x Nvidia Tesla K80 (driver 397.44)

We tested above configuration on a job consisting of approx. 500 photos 20MP each (oblique photos with a lot overlap)

Of course we turn on GPU acceleration in Preferences window.
"Align photos" took 11 minutes (matching time 4m26 + alignment 6m21s) which seems OK to us, although "Build Dense Cloud" on medium quality and moderate filtering took almost 24 hours (4h32min of reconstructing depth and 19h point cloud generation), which from our experience is about 5 times slower than it used too be on numerous similar projects. On previous jobs we also noticed that depth reconstruction was much more time consuming than point cloud generation, which now seems to be opposite.

We are now trying various settings and observing CPU and GPU usage - at some stages both seem to be working at its maximum power.

We are running out of ideas on what is causing this problem.

Has anyone else experienced slowdown in processing since the latest Metashape update?
What processing time would you expect on such configuration?
Any advices on workstation settings, which I may not be aware of? 
Is Nvidia Tesla K80 fully supported by Metashape?

[Alexey Pasumansky]

Hello jansawicki,

Do you have the processing logs related to the runs in the version 1.4 and 1.5? Or at least the PDF reports from the both projects that contain the processing settings used and timing for the stages?

The dense cloud filtering process (second phase of the Build Dense Cloud stage) is CPU-only sage, so GPU is not involved here (both for the versions 1.4 and 1.5). Usually such long processing time is related to the excessive image overlap where almost every camera has overlap with any other.
For such cases it is possible to use "main/dense_cloud_max_neighbors" tweak (see the following thread how to set it up: https://www.agisoft.com/forum/index.php?topic=9636.msg44442#msg44442)

[jansawicki]

Hello Alexey,
Thank you for your answer.
I've just tried using suggested tweak - generating pointcloud seems to be A LOT faster, although hasn't finished yet so I cannot share any numbers.
Can you give some more explanation on how this tweak works? Where can I find a list of other tweaks that can be used?

I also switched depth filtering to "disabled", which I believe reduced the time of depth maps generation, am I correct?

I will run some additional tests to share 1.4/1.5 processing reports as soon as our workstation will not be busy.

Thanks again,
Jan

[Alexey Pasumansky]

Hello Jan,

Depth filtering mode is related to the depth maps generation - first phase of the dense cloud generation. However, disabling the filtering is not suggested, as the result will be noisy and, actually, it may even lead to the longer dense cloud generation due to that noise, but it shouldn't be that considerable.

The mentioned tweak limits the number of pairs considered for each camera when the dense cloud is generated. The speed up from using this tweak would be greater for the projects with very high overlap ratio. By default this number is not limited (you can switch to default behavior, if necessary either by removing the tweak, or by setting its value to -1).

The tweaks, actually, are not considered to be used in regular cases, otherwise they will be added as parameters to GUI, so there is no list of tweaks available, as it may lead to confusion and to invalid result, if the internal parameters are changed carelessly.

[cadm8]

Check last sentence with Alexey's quote

Hi, I've had speed processing problems with Photoscan versions above 1.3.0 (about double or more the processing time for the same projects), so I tested 1.4.0 with dissapointing results... About 35hr to produce a 4mil. point cloud on an i7 6700k with 32gb ram and an r7 radeon. I'm attaching the log https://drive.google.com/file/d/0BzGCwRIMzzYsM0ZXTTBfUHdZcTg/view?usp=sharing in case something comes out

Do you have the processing log or at least Chunk info with the parameters and time spent from the previous version, where the processing time was adequate?

According to the log the depth filtering took very long time and it could be related to the excessive overlap, when most of the cameras are overlapping with a lot of neighbors.

I will provide this, since I can reprocess the job with 1.3.0. You're right there is probably much unneeded overlap in this project (it was an early one)

Hello cadm8,

In the version 1.3.0 the number of pairs for the depth filtering has a strict threshold: 50 pairs, in the later updates the limit has been removed.

[jansawicki]

Hello again,

I attach some processing reports:
-old one, generated a couple of months ago, processed with PhotoScan 1.43 (normal time of processing)
- new one, generated with Metashape 1.5 (long depth maps generation time)
- new one, generated after downgrading back to Photoscan 1.43 (normal time again)

I think last two reports show that depth maps generation time increased drastically after updating to Metashape 1.5.
Every time in Preferences window I've had GPU devices enabled and "Use CPU..." option unticked.

Please help me if you have any ideas.

Greetings,

 

[Alexey Pasumansky]

Hello jansawicki,

Do you have the processing log saved from the version 1.5?

Also have you been using any "tweaks" in the older versions (1.4) of PhotoScan? Like max_neighbors for the depth maps or dense cloud?

[jansawicki]

Unfortunately not. Will it be OK if i re-run just the "build dense cloud" process and share the logs? I didn't use any tweaks during these tests.

[Alexey Pasumansky]

Hello Jan,

Yes, if you have time to run the test again, then ensure that Write log to file option is enabled in the General Preferences tab. I can suggest to re-start the application after enabling writing the log to file.

[adam_s]

Hello Alexey,

I'm cooperating with Jan. The log file, which you asked about, is too big to attaching it below.

I uploaded this file to zippyshare.

https://www10.zippyshare.com/v/Wry5xIMX/file.html

Best regards,
Adam

[jansawicki]

Any updates on this?

[Alexey Pasumansky]

Currently we are trying to reproduce the issue on similar size datasets.

[Alexey Pasumansky]

Hello Jan, hello Adam,

Are you observing the similar processing time difference on the smaller sets of images?

For example, if you run the dense cloud generating in High quality and Aggressive filtering on the alignment from the following project:
https://www.dropbox.com/s/hh5yg0fmpr4bpn3/benchmarking_1.3.zip?dl=0
What is the time difference for 1.4 and 1.5 versions run?

[jansawicki]

Hello Alexey,
Thank you for sharing benchmark project. I just ran it on both 1.4 and 1.5 versions. Version 1.4 was still faster than 1.5, but the difference was "only" about 20-25% (1.4 total time = 27minutes  compared to 1.5 total time= 35minutes) so not that huge as in larger projects. I attach log files from both runs.

I've noticed that the name of GPU driver changed slightly in the beginning of the file, and "Depth reconstruction devices performance" was different. On the other hand, 1.4 was faster on both stages of the process - not only reconstructing depth but also generating pointcloud.

Of course I didn't do any GPU driver updates between these tests, and I made sure no other processes were running in the background.

I don't think it is the issue, but I would also like to mention that I am using the computer with remote desktop access, without full OpenGL support. This is what I see in the console at the start:

Code: [Select]

2019-02-18 13:09:10 Agisoft Metashape Professional Version: 1.5.0 build 7492 (64 bit)
2019-02-18 13:09:10 Platform: Windows
2019-02-18 13:09:10 CPU: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (server)
2019-02-18 13:09:10 CPU family: 6 model: 79 signature: 406F1h
2019-02-18 13:09:10 RAM: 127.9 GB
2019-02-18 13:09:11 OpenGL Vendor: Google Inc.
2019-02-18 13:09:11 OpenGL Renderer: ANGLE (Microsoft Basic Render Driver Direct3D9Ex vs_3_0 ps_3_0)
2019-02-18 13:09:11 OpenGL Version: OpenGL ES 2.0 (ANGLE 2.1.99f075dade7c)
2019-02-18 13:09:11 Maximum Texture Size: 16384
2019-02-18 13:09:11 Quad Buffered Stereo: not enabled
2019-02-18 13:09:11 ARB_vertex_buffer_object: supported
2019-02-18 13:09:11 ARB_texture_non_power_of_two: supported
I have no direct access to this machine, but this worked fine in the past.

[Alexey Pasumansky]

Hello Jan,

Thank you for additional tests, it would be helpful for us, while we are working on the issue.