Yet again out of Memory

[olihar]

3 Days straight processing, comes to the last step and I get this.



There has to be a way to fix this, it has finished processing everything but in the last step it does not allow to cache more then certain amount and crash.

I have 64GB of RAM and 512GB cache space, "Photoscan does not seem to be able to use it all as it will crash when reaching up to a certain value" "Cache is a stripe of SSD disks and I could see way managed to keep up with Photoscan pretty good, not as RAM would but close"

This is a 3D object not ground so using height field is not an option.

[olihar]

Logs

[Matt]

Hi,  Any solutions for the bad allocation at the end of the processing yet.  I have been getting the same result from a project with 80 20 MP photos on a Quad CPU, 32 core, GTX 680 machine with 256 GB of RAM. 

[FoodMan]

wow Matt, what kind of Monster Machine you have ...? I mean 256 GB Ram..? I would call this a bug then..?

Thanks

[Matt]

Its a number cruncher for sure but I have crashed it a few times and out of memory fairly often.
We need another 256 GB of RAM but it will cost about another $7000 US.

Specs

Dell Poweredge
4 * AMD Opteron 6128 (=32 cores)
RAM: 256 GB ECC

I have now managed to process the 81 photos on ultra high.  It took 10 hours on the above machine. Errors possibly caused by high user traffic on the server.

[FoodMan]

awesome Matt, thanks for the answer..

f/

[Wishgranter]

matt can send us bench of the card ? open just 3 photos, align and the process on LOW, jump to the menu CONSOLE (under menu VIEW)  and there shold be few lines up numbers like this and post it

finished depth reconstruction in 2.529 seconds
Device 1 performance: 29.1235 million samples/sec (CPU)
Device 2 performance: 144.011 million samples/sec (GeForce GTX 560 Ti)
Total performance: 173.134 million samples/sec
Generating mesh...
resolution: 0.0128725
grid size: 753x425x126
integrating depth maps... *** done
429794 faces, 229951 vertices extracted
filtering mesh (429776 -> 419885)
Calculating vertex colors...
blending textures... *** done in 0.58 sec
Finished processing in 5.96 sec (exit code 1)

[glennn]

I too constantly run out of memory at the end of each process.  I am pushing it very hard with 112 images at 5000x4000.  Sometimes I get lucky and it works.  Other times it just falls over at end of processing and sits there all night processing at 46% going no further.
I am processing 2million polys from 112 images at high quality./

[Alexey Pasumansky]

Hello glennn,

Could you please specify the reconstruction method you are using and the type of object or scene?

Also we recommend to process your dataset in version 0.9.0 since there are some memory and speed optimizations for Smooth methods.

[FoodMan]

mmmm.. not sure but I don't think it's a memory problem... I have just made one... 154 photos, Tif, 5616 X 3744, all masked.. each photo is 87 mb..

I used arbitrary sharp.. high setting.

BUT it stopped without any message while making the vertex color.. I cancelled, saved the project, reloaded it, and did a rebuild (with "rebuild depth maps ticked off) and it worked perfect..

I have an (now) old Intel R Xeon R CPU X5482 3.20GHz with only 14 GB ram..

final mesh is around 15 millions polys..

so it's got to be something else...

btw, I am using the last Pscan build 0.9 - 1530

f/

[WickedAndy]

I as well get frequent out of memory errors, with 196 gigs of ram..

Here's a few suggestions based on the 0.9.0 version..

the issue I have, is since the longest part of the process is the depth generation, it's great that now you can re-use depth data, but the problem is you CAN'T re-use it if you change reconstruction resolution, for example. I do a Ultra High reconstruction, takes 2 days to process the depth, then fails on geometry build... great so I really can't do much except switch to High setting, or lower but switching to another level  photoscan has to re-build the depth data... would be great if you could preserve the "ultra high" depth data since it's there, and now worthless...

few other things that would be nice, is full picture display of depth, right now it's on the thumbnails only.. and even that there is depth data generated when you right click and want to export depth, photoscan once again has to rebuild the depth..... shouldn't it just export what took 2 days to generate in the first place?

[glennn]

I am processing geo with arbitrary, smooth High 600k polys filter at .2
Also using the latest build 0.9.0 wich i tried last night.
I have reduced it too 97 cameras

Im processing a tree where I have just stepped around the object.

I had done a number of trees before and they all worked great.  This scene was a little more complicated and needed more pictures. 

[glennn]

I was considering running out to buy more Ram today but even WickedAndy is running out of ram with much higher quality images and 196gb of ram.  So I dont think buying more ram will help me.

[FoodMan]

I was considering running out to buy more Ram today but even WickedAndy is running out of ram with much higher quality images and 196gb of ram.  So I dont think buying more ram will help me.

as I said, I don't think it's a ram issue.. I have only 14gb

[Dmitry Semyonov]

Hello glennn,

Please make sure that bounding box of reconstruction volume is not too large. Most likely it was selected by default to include some distant objects visible in the background. Reducing it to a smaller size covering just an object of interest will greatly reduce memory consumption in this case.

Reconstruction volume can be adjusted using Resize Region and Rotate Region toolbar buttons.