Runaway RAM usage on first processing node causing processing failures

[RVC-Juan]

Hi all,

I have recently been testing the network processing feature on our array of company modelling PCs.

I am encountering an error in which the first processing node seems to parse/preload data and run out of RAM, causing a Unexpected Process Termination error message. Please see attached image. The system will then repeat the process, failing at the point at which it cannot be allocated any more memory.

Each processing node is equipped with 64GBs RAM & dual RTX 2080 supers (or similar).

I am aware that 64GBs of RAM is not by any means enough for the type of projects I am doing here, but I had assumed that the network processing feature would share the load in a way that would not cause errors like this.

Will network processing always parse/preload data on the first node before allocating tasks to the remaining nodes?

If this is the case, would it make sense to upgrade the 'main' node to 128GB ram or similar and leave the remaining nodes as they are? Or will I see this problem replicated on other nodes despite upgrading the main one?

I hope this makes sense! I would appreciate any input, cheers! 

Kind regards,

Juan Shimmin

Graduate Survey Data Analyst - Rovco

[Alexey Pasumansky]

Hello Juan Shimmin,

Can you please provide the log from the node, which failed with the error? Via the Network Monitor you can open the list of disconnected nodes and through Details command of the context menu open the node log up to the node termination.

[RVC-Juan]

Hi Alexey,

On the right hand side of the image I attached is what the node was logging at the time of the error, as you can see it's in a loop trying to preload data on the 2/6 part of the BuildDenseCloud task. Unfortunately I can't get the actual log file for this, as the particularly node was not set to write one. (oops)

I will try and repeat this error today and produce a a log file for you to have a look at.

Kind regards,

Juan S

[RVC-Juan]

Hi Alexey,

Find attached the full log from this node.

The same error occurred:

(2022-02-28 13:38:46 [PC NAME AND PORT REDACTED] failed #0 BuildDenseCloud.buildDenseCloud (2/6): Unexpected process termination

I look forward to your response. 

Kind regards,

Juan S

[RVC-Juan]

Hi Alexey,

Is this the detail you needed? Or can I provide any further info?

Kind regards,

Juan S

[RVC-Juan]

Bump 

[jenkinsm]

Alexey, could you post here (not just email Juan directly) when you have figured out the problem or found a solution? I (and likely others) would like to learn as much as possible about networked rendering and the situation Juan posted about is something that I will need to consider when I eventually move towards networked rendering for my projects.

Thanks!

Hello Juan Shimmin,

Can you please provide the log from the node, which failed with the error? Via the Network Monitor you can open the list of disconnected nodes and through Details command of the context menu open the node log up to the node termination.

[Roofy_g]

+1 Please 

[RVC-Juan]

Hi,

Just returning to this to see if there has been any movement/information come to light over this?

Kind regards,

Juan S

[RVC-Juan]

Has there been any more development here?

This is becoming more and more of an issue.

We have recently purchased a server system with > 1TB of RAM and are still encountering issues/errors, could this please be fixed!