Mesh generation stuck, no disk swap occuring

[dcobra]

Hello,

I'm building a fairly large model that has been split into chunks and am having issues generating meshes.  Project is ~5000 20 Mpix images.  Hardware is a 2008 Mac Pro 2.8 octo, 42 GB Ram.  Using photoscan 1.3.4 professional version.  Project file is saved on an internal drive dedicated for storage, not the system drive.

I aligned all of the images in one chunk and after adding CPs split the project into many chunks using the split code provided by Agisoft. I want a very detailed models and prefer smaller models for use in a visualization tool once the models are complete so I split into ~260 chunks.  The dense point cloud processed without any problem.   I am now trying to mesh the dense point clouds and am having varying degrees of success.  Meshing will start off well and then after processing 2 - 6 chunks it gets stuck and I can leave it over the weekend without any progress.  Progress bar just sits on a percentage completed and elapsed time increases.  CPU has essentially no load.  There does not appear to be any swap occurring and total RAM usage tends to only reach ~20 GB so it doesn't seem like a RAM issue, which is what I have read is the common problem with mesh generation getting stuck.  The dense point clouds are 10-20 million points, and I would estimate that there are <50 photos in each bounding box.  The process doesn't necessarily fail on the largest point clouds.  The only remedy is to force the application to close and restart the machine.  If I force close but don't restart the problem happens again right away on the first mesh.  After I restart I can process a few more meshes.  I have deleted most of the cameras that are not within the bounding box in each chunk as a potential fix, although I did leave some on either end of each chunk.  This hasn't made any difference. 

Any ideas as to why this is happening would be greatly appreciated.  Could chunks that are too small be causing the problem? 

[Alexey Pasumansky]

Hello dcobra,

Can you please provide the processing log related to the failed operation, also please specify, how many chunks are there in the project and whether you are using PSZ or PSX format.

[dcobra]

Alexey,

Thank you for the reply.  I am using PSX format.  Will get back to you with a processing log.

[dcobra]

Alexey,

I let processing run over night to generate a new log.  See attached.  When successful, these meshes typically take 1.5 hrs to process.  I was able to process 3 and the current one has been running for almost 11 hrs.  There is no swap, as can be seen in the attached screen shot.

[dcobra]

Attached is another log where processing has stopped but at a different step.

[dcobra]

I dug into the console to check for system errors and found these logs that happen when Photoscan stops processing. There are Buring CPU, spindump and excessive wakeups errors.  Logs attached.

[dcobra]

As an update, I booted into safe mode and was able to process 12 models in a row which is a significant improvement.  If safe mode would have completely resolved my problem I would have thought that it had something to do with login items or disk cache.  Not sure why it improved processing but the problem still persists. 

Alexey - what are your thoughts after reviewing the logs?

[dcobra]

Thought I would post the outcome of this in case anyone else has similar problems.  I managed to figure out the problem after a significant amount of trouble shooting, both hardware and software. 

The problem ended up being with the tree depth that is selected during mesh generation.  Looking back at the logs I noticed that the tree depth had increased significantly from the meshes that I was successfully processing previously.  Meshes that I was able to process previously of a similar model had a tree depth of 11, whereas the meshes that I was having problems with had a tree depth of 15.  From my research 15 is extremely high.  I ended up figuring out that by changing the bounding box to be as tight to the model as possible, the tree depth would decrease to 11. 

It would be nice to have the ability to manually control tree depth, or at least have an upper limit.  I don't think the problem was hardware related as memory and cpu usage were not at capacity when problems occurred.