Forum

Author Topic: 1.1.3 and 1.1.6 causes PC to reboot!  (Read 6751 times)

eriks

  • Newbie
  • *
  • Posts: 34
    • View Profile
1.1.3 and 1.1.6 causes PC to reboot!
« on: May 05, 2015, 11:57:56 AM »
Photoscan Pro 1.1.6 gives the error shown attached and a and makes the PC reboot! when building a  pointcloud out of 566 images of 36Mpix each.
It is a new PC(intel i7-5930) with windows 8.1 and 2x GeForce GTX 980 (NVIDIA driver 344.48 and 350.12 tested)
It has 64Gb ram, 10 of 12 cpu cores active, and the two graphics cards active as OpenCL.
The PC have really been stresstested for a few months including OpenCL and autoupdate is switched off, but I can see Windows is configured to automatically reboot with a critical sustem fault.

Is it software bug or a hard way of saying "PC out of memory"?
How much memory is required to compute the pointcloud from 566 images of 36Mpix each?

The last part of the Photoscan logfile says:

ocl_engine.cpp line 231: clEnqueueWriteBuffer failed, CL_MEM_OBJECT_ALLOCATION_FAILURE
GPU processing failed, switching to CPU mode
[CPU] estimating 2115x2458x256 disparity using 1058x1229x8u tiles, offset 0
timings: rectify: 0.484 disparity: 2.523 borders: 0.343 filter: 0.376 fill: 0
[GPU] estimating 3119x3174x256 disparity using 1040x1058x8u tiles, offset 0
timings: rectify: 0.578 disparity: 19.134 borders: 0.296 filter: 0.204 fill: 0
[CPU] estimating 2022x3369x192 disparity using 1011x1123x8u tiles, offset 0
timings: rectify: 0.718 disparity: 4.364 borders: 0.454 filter: 0.421 fill: 0
[GPU] estimating 2193x3160x288 disparity using 1097x1054x8u tiles, offset -35
timings: rectify: 0.318 disparity: 12.198 borders: 0.11 filter: 0.187 fill: 0
[GPU] estimating 1456x1959x256 disparity using 1456x980x8u tiles, offset -106
ocl_engine.cpp line 231: clEnqueueWriteBuffer failed, CL_MEM_OBJECT_ALLOCATION_FAILURE
GPU processing failed, switching to CPU mode
[CPU] estimating 1456x1959x256 disparity using 1456x980x8u tiles, offset -106
timings: rectify: 0.547 disparity: 3.078 borders: 0.234 filter: 0.203 fill: 0
[GPU] estimating 1262x3129x352 disparity using 1262x1043x8u tiles, offset -171
timings: rectify: 0.281 disparity: 1.787 borders: 0.203 filter: 0.157 fill: 0
[GPU] estimating 1931x2337x416 disparity using 966x1169x8u tiles, offset -107
timings: rectify: 0.391 disparity: 12.991 borders: 0.186 filter: 0.157 fill: 0
timings: rectify: 0.219 disparity: 6.028 borders: 0.157 filter: 0.046 fill: 0
[CPU] estimating 1443x3104x192 disparity using 1443x1035x8u tiles, offset -26
[GPU] estimating 1049x2886x352 disparity using 1049x962x8u tiles, offset -62
ocl_engine.cpp line 231: clEnqueueWriteBuffer failed, CL_MEM_OBJECT_ALLOCATION_FAILURE
GPU processing failed, switching to CPU mode
[CPU] estimating 1049x2886x352 disparity using 1049x962x8u tiles, offset -62
timings: rectify: 0.235 disparity: 2.694 borders: 0.109 filter: 0.25 fill: 0
[GPU] estimating 2461x2323x416 disparity using 1231x1162x8u tiles, offset -77
timings: rectify: 0.213 disparity: 3.658 borders: 0.218 filter: 0.188 fill: 0
[GPU] estimating 2353x2404x384 disparity using 1177x1202x8u tiles, offset 0
timings: rectify: 0.296 disparity: 8.59 borders: 0.109 filter: 0.141 fill: 0
[CPU] estimating 1956x2265x192 disparity using 978x1133x8u tiles, offset 0
timings: rectify: 0.344 disparity: 2.984 borders: 0.313 filter: 0.203 fill: 0
[GPU] estimating 1873x2389x384 disparity using 937x1195x8u tiles, offset -40

--
In the PC Event viewer, I can find these errors before the automatic restart:

Cannot find a description on Event-ID 13 from source nvlddmkm.....
\Device\UVMLiteProcess1
Graphics Exception: ESR 0x505224=0x80000041 0x505228=0x180010 0x50522c=0xd94e2b 0x505234=0x0

Cannot find a description on Event-ID 13 from source nvlddmkm......
\Device\UVMLiteProcess1
NVRM: Graphics TEX Exception on (GPC 0, TPC 2):     TEX NACK / Page Fault


Cannot find a description on Event-ID 13 from source nvlddmkm.....
\Device\UVMLiteProcess1
Graphics Exception: ESR 0x505224=0x80000000 0x505228=0x0 0x50522c=0x0 0x505234=0x0

--
AND when reducing the amount of images to 367, I can build my dense cloud, but Generating mesh... :
This error appears:

Grid size: 32768 x 13442 x 3964
Tree depth: 15
Error: vector<T> too long
"Not enough memory"
Finished processing in 37730.3 sec (exit code 0)
Saving project...

I tried to fix this "Error: vector<T> too long" by reducing the region a bit, it didn\t help :(
« Last Edit: May 06, 2015, 11:05:45 PM by eriks »

Arie

  • Full Member
  • ***
  • Posts: 134
    • View Profile
Re: 1.1.3 and 1.1.6 causes PC to reboot!
« Reply #1 on: May 06, 2015, 01:06:35 PM »
My first guess would be that it's a temperature problem. Grab something like GPU-Z and log the gpu temperatures.
Building the dense cloud from 566 images using both cards is quite a workload, and air-cooled multi-gpu setups can become quite hot.
Good luck troubeshooting!

eriks

  • Newbie
  • *
  • Posts: 34
    • View Profile
Re: 1.1.3 and 1.1.6 causes PC to reboot!
« Reply #2 on: May 08, 2015, 01:21:51 PM »
Thanks Arie!
I have now split the project into two chunks and now it seems to work, and GPU-Z says the temperatures have been maximum 57 degrees C. Previously I have experienced temperatures upto 88-90 deg C and it was still running ok, so at least this is not my problem now.


Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 14813
    • View Profile
Re: 1.1.3 and 1.1.6 causes PC to reboot!
« Reply #3 on: May 08, 2015, 03:59:56 PM »
Hello eriks,

System reboot is usually related to the hardware issues, so if you have some time for testing, we can suggest to run memory and GPU tests.
Best regards,
Alexey Pasumansky,
Agisoft LLC