Odd failures over network processing

[adalta]

I recently started a quite large project using network processing hooked up to 8 machines. The biggest of this machine being a 256gb ram, dual Intel Xeon E5-2690 V4 processors and a quadro M600 graphics card this machine is setup as the server and as a node in the cluster, the other PC are all 32gb ram quadro k4000 graphics cards and dual Intel Xeon E5-2690 V4 processors all setup as nodes. The project that I started has 4 chunks total (2 chunks are micasense data added as multispectral photos, the other 2 chunks are DJI X3 photos) the total photos combined across all chunks is about 2800 photos. Each chunk is the same area just flown pre-fire 100m and 90m and post-fire 100m and 90m, total area covered is approx. 2.5 hectares. I often found that the large PC in the cluster would be sitting idly by while the other 7 PC processed the node they were working on even though the largest PC was set to highest priority. To build depth maps on all these chunks it took approx. 132hr to complete, then to build dense clouds it took about another 124hrs and it was at 91.4% on chunk 3/4 again I noticed that the largest PC was sitting idly by while the smaller machines were still busy on the last 2 nodes. I decided to manually kick out one of the smaller machines so the large machine would pickup the node which it did and finished the node before the smaller machine was able to even get 10% done on the last remaining node. So I again manually kicked it out of the cluster and the larger machine finished the last node in no time at all. Also while doing depth maps and dense clouds I am constantly getting bad allocation errors on the smaller machines which to my knowledge is due to running out of memory?? Needless to say after I removed all the smaller nodes from the cluster and was just left with the larger pc the processing time dropped by days and the project finished most of the batch process except, I keep getting DEM's that are blank when I use the classify ground pints and then try and build DEM from ground points, which in turn makes the build orthos process fail also. But if I individually classify ground and build DEM and ortho by individually selecting which chunk to process this on these failures of empty extent cease to exist. Also is there any handy way to export a processing report on multiple chunks at one time or do you have to manually select each chunk and export report. I have Included a report of just one of the chunks that was processed in this project here. I am using the most current stable version of photoscan pro.

[adalta]

I have a general question(s) about network processing.

1. does it matter what PC you use as a server.
For example if I setup our most powerful PC as a server and node will that effect the lower performing PC's in the cluster, I have found instances where it seems to slow down processing.

2. What does GPU mask mean on network monitor. I have been able to set them from 0 to 50 what does that mean, if one is 0 and the rest are 1,2,3,4.... doe those values mean anything and if so what?

3. Bad allocation errors seems to be a constant error on 2 of the 7 machines that are in the cluster (all of which have 32gb of ram except the server has 256gb. I have tried adding to the VRAM, still constantly get bad allocation.

4. Are nodes based on the server? For my example server is our big pc 256gb ram and a very nice graphics card (it fails me to what it is, it was new 2 years ago and quite powerful at the time), the other 6 machine all have 32 gb of ram and identical graphics cards and processors but only 2 of the machines give bad allocation errors consistently .

5. I am processing very small projects (area) with extreme overlap and sidelap (600photos\3 hectares), if you read the earlier post you will understand the project, if you don't please ask.