1.0.0 and 1.0.2 "stuck" at ca. 95% of dense cloud computing

[Heinrich]

HELP, please!

I used 1.0.0 for a long time and it worked fine. Three days ago, when calculating a batch process (dense cloud, high, 6 chuncks with ~40 cameras each, expected number of points around 8 million) the program got "stuck" at 95% of the first dense cloud in the reconstructing depth step, no remaining time shown. The elapsed time kept increasing to 145 hours. At that point I tried to the job via "cancel", which had no effect. Memory usage waas a ~2 GB of 8 available, CPU power was at 100% for one, with the remaining 15 idling.
I had to end photoscan via the task manager (end task).

I rebooted and tried again, this time with only one chunk. Same result, this time stuck at 92%, no time remaining shown.
I tried again, using a smaller box for the point cloud, and that worked.
I tried again, with a somewhat larger box, and it failed again.

I upgraded to 1.0.2 and keep having the same problem.

The size of the jobs is so small that normally Photoscan handled them within less than an hour without problems. It is not a memory issue (never more than 66% of available memory used). it is not a CPU issue, either.

I tried uninstalling and reinstalling - no change.

The "critical" size seems to be anything that calculates the depth reconstruction for more than 40 minutes.

Any suggestions how I can get this to work again? If needed I can post files.

[Alexey Pasumansky]

Hello Heinrich,

Could you please provide us the project file and source photos to reproduce the problem on our side? Please also specify the processing settings used.

In case you have the log file saved for the operation, it could be also helpful for understanding the cause of the problem.

[Heinrich]

here you go: http://wp.me/a1v8tM-1AV (shortlink; I uploaded it to my blog)

rename the file to *.zip!

I can calc any dense cloud up to ~ 1.5 million polygons points, but anything bigger leads to the "stuck" problem. The last attempt got stuck at 0% of point cloud generation, again with one CPU under 100% load.

Alignment was performed in an earlier version (possibly 0.9.0), and now I want a dense point cloud in high, aggressive filtering.
Note that I had to mask parts of the pics, because repetitive architecture gave false matches between points.

Please also note: the problem is NOT limited to THIS file! I have dozens of files that get stuck in the exact same way. I remember a similar issue with the pre-release of 1.0.0 (IIRC), which was gone after a build update the very next day.

EDITED to correct mistake.

[Alexey Pasumansky]

Hello Heinrich,

Thank you for the dataset, we'll check it on our side. If you have any log related to this problem it will be also helpful for us.

But do I understand correctly, that problem appears only during dense cloud generation after depth maps have been already calculated? You've mentioned 1.5 polygons, and it is a little bit confusing.

[Heinrich]

Sorry, Alexey! My mistake: it is 1.5 million POINTS, not polygons.
Sorry, no log file.

[Alexey Pasumansky]

Hello Heinrich,

We have no issues during processing of the dataset provided (High quality, Aggressive filtering).

Could you please specify the computer configuration and OS used?

[Heinrich]

Hi Alexey,
it is a virtual machine,
Windows Server 2008 R2 Datacenter, SP1
Intel XEON X5670 @2.93 GHz, 16 processors (hyperthreaded)
16 GB RAM
TESLA M2070

I have been running models the exact same way for the last two weeks, some 20 in all I guess, and it was always fine.

[Alexey Pasumansky]

Hello Heinrich,

Just for case, could you please check of 64-bit version of PhotoScan is installed. If possible, I also ask you to run the "long"processing with log file turned on in the Preferences window, so that we could at least know the sub-step where the problem occurs.

[Heinrich]

Yes, it should be the x64 bit version (will check). Will start the run now and post log tomorrow.

[Heinrich]

here's a log file (finally); this is with a very small box and I was surprised that this run got stuck.

[Alexey Pasumansky]

Hello Heinrich,

Thank you for providing the log file. Looks like the depth maps estimation on GPU is stuck by some reason.
Could I ask you to perform another short test, to run the same data on CPU only (without any OpenCL devices enabled)?
Also if possible I recommend to update the graphic card drivers for Tesla, in case it is possible for virtual machine.

[Heinrich]

hello Alexey,
without openGL it ran a small box just fine. Currently, it is running a huge model; I'll let you know.

We deliberately did not update the drivers, because we were glad to have found, after much trouble, a configuration that worked for all software that is installed. Thus, if Photoscan is fine without openGL, we will leave things as they are.

[Heinrich]

Alexey, consider this solved as far as I am concerned: it was indeed the openGL-use!
Many thanks!

Once the current very urgent work is done I will ask our IT to update the driver and try again. if the problem persist I'll let you know.

 

[Alexey Pasumansky]

Hello Heinrich,

Thank you for this information. Hope the problem will be solved with the new driver installed.

P.S. but it is OpenCL and not OpenGL