Not Enough Memory, Again!

[Kars-10]

Hello All,

So I am a novice with both photoscan and programming, but I will do my best to supply as much information as possible.

I am modelling surface texture and micro-topography of bedrock samples. Typically my models have around 55-60 photos to them shot on a full frame Nikon camera, usually 7378x4924 pixels. However, I have gone through and masked off around 60% of the photo area to ensure that photoscan processes the least amount of data necessary.

I can generate my dense point cloud on all settings except "Ultra High", which is unfortunately the only resolution that I actually need. I am using our 32-core high performance cluster with 128gb of RAM.

Each model will run for varying periods of time, some up to 100 hours, and always fail just before completion with the same message "Not Enough Memory". Looking at the records, photoscan utilizes around 110gb of the cluster's RAM while it is processing, and suddenly in the last portion of processing it spikes and tries to consume more than 200% of the available memory for the system. I have it recorded that photoscan attempted to use 287gb of ram on our system that only has 128.

My question:
Is there any way to assign a maximum RAM allocation threshold for photoscan?

If needed I can offer the program several terabytes of virtual memory, though I do not know if that would solve anything, nor do I currently know how to do so. All other processes seem to work no problem, but every single model regardless of the photoset and level of masking fails, without fail (pun intended) on the dense cloud generation. I have tried at least a dozen "Ultra High" res dense clouds over the past two and a half weeks with no success.

Please let me know if you have any tips, all advice is very much appreciated!
Thank you.

[Alexey Pasumansky]

Hello Kars-10,

Can you provide the full processing log related to the failed operation and also the overview screenshot of the model view with the camera positions?

[Kars-10]

Alex,

Thank you for your reply, and my apologies for the tardiness of my own.
I will capture and upload those requested logs and screenshots this weekend.

Due to the extreme consumption of the school's cluster resources, I have been politely forbidden to utilize it for photoscan until after business/working hours. This weekend will provide enough time to process another model on high resolution and receive a failure message without interrupting anyone else's work.

All the best,
-Karsten

[Kars-10]

My apologies for the extremely late reply, but I am still running into the same issues each time I try to render a model on Ultra High resolution.

Here is the processing log, and the screenshots will follow:

[Kars-10]

Here is the requested screenshot of the model and the camera positions. Please let me know if there is any more information that I could supply to help.

[Alexey Pasumansky]

Hello Karsten,

Ultra high reconstruction may be very resource demanding operation in case of excessive overlap.
Can you try to generate DEM in High and notice the memory consumption?

Also I can suggest to use version 1.3.2 for processing.