Cluster Processing - RAM consumption of nodes

[HyperFox]

Hello,

I have got a project, that ends up in 60 packages (nodes) and the processing on each single node consumes 54 GB RAM. So the node start swapping (windows or linux with the same behaviour).

Is ist possible to define the max amount of ram, that is allowed for a node in a cluster?
Fine level task distribution is already activated for all steps...

Best regards
Ansgar

[Wishgranter]

how much imgs per project ?

[HyperFox]

The project has 1000 oblique images, each 15 MPix.
Split in chunks is no option due to the oblique imagery.
As I mentioned, i tried to process it as cluster.
In fine level task distribution, the server creates 60 tasks.
However, each task needs about 54 GB to process the data.
My cluster nodes have only 32 GB per node, so they start swapping.

I also tried to switch off the swap file. The result is the same, but now the nodes crashed instead of swapping (which is expected).

So my question is: Does the server take the node's memory into account, when he is defining the tasks? Can I increase or decrease the number of tasks? Or can I specify the "max memory" for a task in cluster processing?

It's no "windows" issue, a linux setup of my cluster showed the same behaviour...

[Wishgranter]

Hmm read PM or email, will talk about it...

[Alexey Pasumansky]

Hello Ansgar,

PhotoScan doesn't take into account the hardware specification of the individual nodes during task distribution. Also it is not possible to limit the mem usage on the nodes.

At what processing step do you have these problems?

[HyperFox]

Hello Alexey,

It is the "filtering depth Images" - step in the command "Build dense Cloud".
The first step is done by GPU and works pretty well on all nodes.
Afterwards, on the CPU, the filtering is done.
And during this process, right at the beginning, each node loads it's data into RAM (I think the depth Images) and starts swapping due to the high memory consumption (54GB, 32 available).