Forum

Author Topic: RAM Memory issue when processing in Cluster Mode  (Read 5273 times)

Sebas80

  • Newbie
  • *
  • Posts: 31
    • View Profile
RAM Memory issue when processing in Cluster Mode
« on: July 21, 2015, 07:20:09 PM »
Hello everyone,
I usually process huge datasets with photoscan pro in clustering mode by using three PC connected on the same network, but I have some consideration to do based on the problems I'm finding by using it, and I would like to know your opinions and experiences.

First of all I need to clarify my cluster network configuration: three workstations with different RAM memory size installed on, as 2 PC have 16 GB, the 3rd one has 128 GB installed on.

When you process a dataset in this mode, as you know, photoscan splits the process in several "packets" and every node of the network processes a single packet and starts to process the next one when the actual packet has been completed.

Now my problem:
I'm processing, as above described, a huge dataset (in my case I'm building a dense cloud in High resolution) but when the "Filtering data" step starts (after the depth maps have been calculated), the nodes with less memory (in my case the two PC's with 16GB) procede to process the packets very slow, as the progress value stays on the same value for a lot of time. By checking the TASK MANAGER on these two machines, I've realized I've NO free ram, as totaly occupied by photoscan (the process is not in a "not responding" status, it's just like a "congestion" of the pc).

1st  question:
how photoscan decides to split the project is packets? I mean...how photoscan decide how many packets must be done? By checking the ram memory size of the nodes of the cluster?

2nd question:
If photoscan checks the ram memory size, it should adapt the packet size to the nodes' ram memory. Isn't it?
Anyhow, believing the packets are equal each other, whit ram size value is used? The lowest one? The Biggest one?

Consideration:
What I think, but I would like a confirm (maybe from agisoft itself) is that the single packet size is computed based on the first node that has been connected to the server, and that could explain the fact that I have two PC freezed, while the PC with 128GB proceedes without problems (in my case the first machine I've connected to the server is the biggest one). Anyhow, this behaviour compromise the conclusion in a reasonable time of the whole process, as there will be always two machine almost freezed on two packets, and even if the biggest one will process all the remaining packets, the process itself will be completed very slowly. The only solution is to disconnect the freezed nodes and to let the biggest pc process the remaining packets.
Could some confirm this thesis?

Thank you.

« Last Edit: July 22, 2015, 09:23:58 PM by Sebas80 »

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 15067
    • View Profile
Re: RAM Memory issue when processing in Cluster Mode
« Reply #1 on: July 23, 2015, 05:42:38 PM »
Hello Sebas80,

At the moment PhotoScan doesn't use the information regarding available RAM on the nodes, when the tasks are distributed by server.
Best regards,
Alexey Pasumansky,
Agisoft LLC