Forum

Author Topic: cudastreamdestroy failed error  (Read 1517 times)

Toll

  • Newbie
  • *
  • Posts: 16
    • View Profile
cudastreamdestroy failed error
« on: October 24, 2017, 11:34:05 AM »
hello

running agisoft 1.3.4  build 5067 on a xeon 2687w v4, 3x1080.

I am getting the attached error as soon as I started 'align photos'

2017-10-24 18:24:39 Warning: cudaStreamDestroy failed: all CUDA-capable devices are busy or unavailable (46)
2017-10-24 18:24:45 Finished processing in 12.54 sec (exit code 0)
2017-10-24 18:24:45 Error: unspecified launch failure (4) at line 119

and a windows 10 notification of "application has been blocked from accessing Graphics hardware"

I have enabled the 3 x gpu's in preferences and disabled the CPU.

disabling GPU's and enabling CPU's allows the align photos to start. It is still processing, so no comments yet on the dense cloud generation.

I have tried multiple versions of agisoft and multiple version of Nvidia driver between 385.28 to 388.0

we have another system with different specs that doesn't have any issues.
can you help please Alexey!?

thanks

Toll

  • Newbie
  • *
  • Posts: 16
    • View Profile
Re: cudastreamdestroy failed error
« Reply #1 on: October 24, 2017, 12:10:38 PM »
For some extra info,

I have downgraded to 1.2.6 and when on 'reconstructing depth'

I get ocl_engine.cpp line 231: clenqueuewritebuffer failed, cl_out_of_resources
Gpu processing failed, switching to cpu mode
« Last Edit: October 24, 2017, 12:24:26 PM by Coalisgood »

Toll

  • Newbie
  • *
  • Posts: 16
    • View Profile
Re: cudastreamdestroy failed error
« Reply #2 on: October 24, 2017, 12:13:37 PM »
back to 1.3.1 and:

2017-10-24 19:12:16 photo 22: 24922 points
2017-10-24 19:12:17 photo 1: 27382 points
2017-10-24 19:12:17 photo 23: 26058 points
2017-10-24 19:12:17 Warning: unspecified launch failure
2017-10-24 19:12:17 Warning: unspecified launch failure
2017-10-24 19:12:17 Warning: unspecified launch failure
2017-10-24 19:12:17 Finished processing in 7.754 sec (exit code 0)
2017-10-24 19:12:17 Error: Kernel failed: unspecified launch failure at line 222

1.3.1 and NVidia version 387.92 (the screen flashed black just as it crashed)

2017-10-24 19:21:46 photo 11: 16798 points
2017-10-24 19:21:46 photo 12: 16205 points
2017-10-24 19:21:47 photo 13: 17233 points
2017-10-24 19:21:53 photo 2: 24814 points
2017-10-24 19:21:53 photo 1: 26094 points
2017-10-24 19:21:53 Finished processing in 11.088 sec (exit code 0)
2017-10-24 19:21:53 Error: all CUDA-capable devices are busy or unavailable at line 91

as above, settings was low alignment, low densecloud, error when processing densecloud (attached)
« Last Edit: October 24, 2017, 12:28:14 PM by Coalisgood »

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 8951
    • View Profile
Re: cudastreamdestroy failed error
« Reply #3 on: October 24, 2017, 12:47:51 PM »
Hello Coalisgood,

This seems to be a known problem and there is no reliable workaround.  Have you tried to reinstall drivers? For example someone workarounded problem with this - https://forums.lenovo.com/t5/ThinkCentre-A-E-M-S-Series/application-has-been-blocked-from-accessing-graphics-hardware/m-p/3609674/highlight/true#M19593
Also can you please test with only one GPU (each one from your configuration) enabled after rebooting the system, as the driver may be crashed now and all the consecutive runs will be affected?
Best regards,
Alexey Pasumansky,
AgiSoft LLC

Toll

  • Newbie
  • *
  • Posts: 16
    • View Profile
Re: cudastreamdestroy failed error
« Reply #4 on: October 24, 2017, 01:31:14 PM »
Thanks Alexey

I have reinstalled the 387.92 drivers and restarted between each of the following

204, 16mp pictures

Lowest align/Lowest dense cloud cpu+gpu#0  = success

Lowest align/Lowest dense cloud cpu+gpu#1  = error on depth reconstruction, launch timed out and was terminated at line 156

Lowest align/Lowest dense cloud cpu+gpu#2  = success

Lowest align/Lowest dense cloud cpu+gpu#1 (again) = success

Lowest align/Lowest dense cloud cpu+gpu#0+gpu#1  = success

Lowest align/Lowest dense cloud cpu+gpu#0+gpu#2  = success

Lowest align/Lowest dense cloud cpu+gpu#1+gpu#2  = success

Lowest align/Lowest dense cloud cpu+gpu#0+gpu#1+gpu#2  = error on depth reconstruction, program crashed, gpu rectifying failed, kernel failed. Launch timeouts on 164,749 and 156.

Lowest align/Lowest dense cloud cpu+gpu#0+gpu#1+gpu#2 (again) = success, but noticeable pauses in the depth reconstruction. Also slowest alignment time... Same depth time as 2gpu.

Lowest align/Lowest dense cloud cpu+gpu#0+gpu#2  = success

Lowest align/Lowest dense cloud cpu+gpu#0  = success

Lowest align/Lowest dense cloud gpu#0+gpu#1+gpu#2  = fastest alignment, no pauses, success

Med align/med dense cloud, gpu0+1+2 = kernel failed: launch timeouts at line 184,749,156

Still testing...

Med align/med dense cloud, gpu0+1 = software crashed, kernel failed: launch timeouts at line 194,191

Med align/med dense cloud, gpu0+2 = success

Med align/med dense cloud, gpu0+1+2= Bsod dpc watchdog violation

Med align/med dense cloud, gpu0+2 = success

med align/med dense cloud, gpu 1+2 = fail, 2017-10-24 21:39:51 GPU rectifying failed: unknown error at line 212Error: Kernel failed: unknown error at line 749, 801,156

med align/med dense cloud, gpu 1+2 = fail, kernel failed, lines 194,307,749

is there something wrong with gpu1?? temps only got to 70degrees..
« Last Edit: October 24, 2017, 02:52:07 PM by Coalisgood »