Increasing Distributed Processing Speed

[Corensia]

Dear Alexey,

We've recently been looking into integrating distributed local network processing for faster processing times and as a result we did some tests in a computer lab using up to 17 computers connected in a LAN. However the results were somewhat unexpected and I'll share the summary below.

Project: 1400 images
Align Photo: Highest
Dense Cloud: High, Aggressive
DEM, Ortho: default

All PC Specs
CPU: I7-7700
GPU: 1050
RAM: 16 GB

17 PCs LAN: Total time - 9hr 21min
8 PCs LAN: Total time - 9hr 41min
5 PCs LAN (**PC where project file is saved also added as a node**): Total time - 5hr 53min ()
1 PC: Total time - 16hr 47min

We also compared this to our office's processing PC (i9-9900K, P4000, 64GB): Total time - 11hr 41min

From the results above it seems clear that using the PC where the project was saved as a node made the most difference and adding 17 PCs but not using the PC where the project was saved didn't even halve the processing time of using just 1 PC. It seems like there is a bottleneck occurring with the processing due to data transfer speed limits across the network. Do you have any tips on how we could get better results using 10+ computers?

Thank you in advance,
David

[Alexey Pasumansky]

Hello David,

It would be helpful to get the PDF report from the processed project and the server log (available in the bottom frame of the Network Monitor window) to have the better understanding, which steps took longer time.

The possible issues, apart from the slow network connection (we recommend to have at least 1 GBit connection) could be insufficient RAM for the operations on the 16 GB nodes - that leads to the disk swap utilization and therefore longer processing time. Have you noticed, if the nodes are using all the RAM available during the operation?

I can also suggest to make sure that all nodes have properly configured GPU mask to enable additional speed-up for the image matching and depth maps generation operations.

[Corensia]

Hello Alexey,

I think you've nailed it, it was the ethernet speed. We assumed it was at least 1Gbps since this was a campus computer lab but it turns out that it was only 100Mbps! That would explain the apparent slow transfer speeds that we noticed.

For your information, the steps that had almost no difference or took longer over network processing were Photo Matching and Orthomosaic Generation. The processing times were greatly reduced during Depth map and Dense cloud generation when using network processing.

While monitoring the process, it didn't seem like insufficient RAM was a problem (most of the time usage was below 50%). And the GPU mask was configured at 1 since they all only had one GPU.

Thank you for your timely response! Hopefully we will get to test this in a faster setting.
David