We are encountering unexpectedly slow dense cloud generation on a high-performance cluster. Any help working out the solution would be greatly appreciated!
We have two computer systems:
Workstation: (OS Windows 64; CPU i7-7800X 6-core 3.5 GHz; GPU 1 x NVIDIA GeForce GTX1080; RAM 128GB DDR4 2133 MHz; Photoscan-Pro 1.4.2 build 6205)
Cluster: (OS Linex; CPU 2x Intel Haswell E5-2640v3 16-core 2.6GHz; GPU 4x NVIDIA Tesla K80; RAM 256 GB DDR4; Photoscan-Pro/1.4.3)
Our test job consists of
489 x 20 MP photos, processed through the initial steps (alignment, precise markers, tie point filtering, optimisation, etc.). All available GPUs are enabled on both systems (x1 on the workstation, x4 on the cluster; either via the GUI or with "
PS.app.gpu_mask = 2 ** len(PS.app.enumGPUDevices()) - 1 # activate all available GPUs"). We will be using Python scripts for (mostly) headless processing. Dense Cloud Generation was attempted at ultrahigh quality with mild depth filtering on both systems.
On the workstation, depth map generation takes 9 hours and dense cloud generation takes 28 hours (37 hours total).
On the cluster, depth map generation takes 5 hours and dense cloud generation takes >28 hours (not complete after 33 hours).
(We thought that there might be unexpected overheads with using the 4 GPUs on the cluster, but our tests with 10 and 100 image sets show processing time consistently decreases (slightly) as the number of GPUs increases).
Questions:
(1) Would anyone comment on our expectation that this Cluster should complete dense cloud processing substantially (e.g. more than 40%) faster than this Workstation?
(2) Could there be an issue with the GPU not being configured for optimal performance from PhotoScan? (e.g. the compute mode, as hinted at by Alexey here
http://www.agisoft.com/forum/index.php?topic=9199.0)
We have no experience configuring GPUs, but suggestions are welcome, especially if it might be possible to better optimise the K80s via the Python script.
Topic: Slow dense cloud generation with Tesla K80 GPUs (Read 2385 times)
