Agisoft Metashape > Bug Reports

CUDA Crash during dense point cloud processing

(1/4) > >>

edtriplett:
I have a  very new multi-GPU machine that has been crashing during the dense point cloud processing due to an error you can see below from the console. It does not happen with a smaller 200 photo process, but this set with 421 photos is causing this error. Does anyone know a good fix?


--- Quote ---2018-02-19 13:15:46 [GPU] estimating 2339x1537x832 disparity using 1170x769x8u tiles
2018-02-19 13:15:46 timings: rectify: 0.03 disparity: 0.402 borders: 0.031 filter: 0.088 fill: 0
2018-02-19 13:15:46 [GPU] estimating 1889x1945x192 disparity using 945x973x8u tiles
2018-02-19 13:15:50 timings: rectify: 0.058 disparity: 0.377 borders: 0.018 filter: 3.42 fill: 0
2018-02-19 13:15:50 [GPU] estimating 1224x924x896 disparity using 1224x924x8u tiles
2018-02-19 13:15:50 GPU rectifying failed: the launch timed out and was terminated (6) at line 216
2018-02-19 13:15:50 using CPU implementation...
2018-02-19 13:15:50 Error: the launch timed out and was terminated (6) at line 216
2018-02-19 13:15:50 GPU processing failed, switching to CPU mode
2018-02-19 13:15:50 [CPU] estimating 1224x924x896 disparity using 1224x924x8u tiles
2018-02-19 13:15:50 Error: Kernel failed: the launch timed out and was terminated (6) at line 199
2018-02-19 13:15:50 GPU processing failed, switching to CPU mode
2018-02-19 13:15:50 [CPU] estimating 2339x1537x832 disparity using 1170x769x8u tiles
2018-02-19 13:15:50 Error: Kernel failed: the launch timed out and was terminated (6) at line 821
2018-02-19 13:15:50 GPU processing failed, switching to CPU mode
2018-02-19 13:15:50 [CPU] estimating 1889x1945x192 disparity using 945x973x8u tiles
2018-02-19 13:15:50 Error: the launch timed out and was terminated (6) at line 198
2018-02-19 13:15:50 GPU processing failed, switching to CPU mode
2018-02-19 13:15:50 [CPU] estimating 1472x1562x128 disparity using 1472x781x8u tiles
2018-02-19 13:15:50
2018-02-19 13:15:50 Depth reconstruction devices performance:
2018-02-19 13:15:50  - 35%    done by GeForce GTX 1080 Ti
2018-02-19 13:15:50  - 32%    done by GeForce GTX 1080 Ti
2018-02-19 13:15:50  - 33%    done by GeForce GTX 1080 Ti
2018-02-19 13:15:50 Total time: 202.378 seconds
2018-02-19 13:15:50
2018-02-19 13:15:50 Warning: cudaStreamDestroy failed: all CUDA-capable devices are busy or unavailable (46)
2018-02-19 13:15:50 Warning: cudaStreamDestroy failed: all CUDA-capable devices are busy or unavailable (46)
2018-02-19 13:15:50 Warning: cudaStreamDestroy failed: all CUDA-capable devices are busy or unavailable (46)
2018-02-19 13:15:50 Warning: cudaStreamDestroy failed: all CUDA-capable devices are busy or unavailable (46)
2018-02-19 13:15:50 Warning: cudaStreamDestroy failed: all CUDA-capable devices are busy or unavailable (46)
2018-02-19 13:15:53 Warning: cudaStreamDestroy failed: all CUDA-capable devices are busy or unavailable (46)
2018-02-19 13:15:53 Finished processing in 231.347 sec (exit code 0)
2018-02-19 13:15:53 Error: the launch timed out and was terminated (6) at line 198
--- End quote ---

Alexey Pasumansky:
Hello edtriplett,

Can you please specify the NVIDIA driver version installed, OS version and PhotoScan version used?

edtriplett:
Thanks for the fast reply. Here are the specs:

NVIDIA DRIVERS: 23.21.13.8813 on 3x GeForce 1080Ti
Windows 10 Enterprise 64bit
Photoscan 1.4.0 Build 5650

Alexey Pasumansky:
Hello edtriplett,

Can you please try to make a clean driver install of the package from NVIDIA web-site:
http://www.nvidia.com/download/driverResults.aspx/130633/en-us
And then check, if the processing works fine after the system reboot.

edtriplett:
Hi Alexey,
I was able to do a fresh driver install and test it successfully when I ran the dense point cloud stage on "Mediu." but when I switched it toi high, I get the same CUDA error you see below:

--- Quote ---2018-02-22 15:32:22 [GPU] estimating 1126x1722x288 disparity using 1126x861x8u tiles
2018-02-22 15:32:22 timings: rectify: 0.015 disparity: 0.145 borders: 0.012 filter: 0.038 fill: 0
2018-02-22 15:32:22 [GPU] estimating 1572x1238x224 disparity using 786x1238x8u tiles
2018-02-22 15:32:22 [GPU] estimating 1064x1889x384 disparity using 1064x945x8u tiles
2018-02-22 15:32:24 [GPU] estimating 1157x1962x384 disparity using 1157x981x8u tiles
2018-02-22 15:32:24 GPU borders filtering failed: the launch timed out and was terminated (6) at line 216
2018-02-22 15:32:24 using CPU implementation...
2018-02-22 15:32:24 Error: Kernel failed: the launch timed out and was terminated (6) at line 769
2018-02-22 15:32:24 GPU processing failed, switching to CPU mode
2018-02-22 15:32:24 [CPU] estimating 1064x1889x384 disparity using 1064x945x8u tiles
2018-02-22 15:32:24 Error: Kernel failed: the launch timed out and was terminated (6) at line 821
2018-02-22 15:32:24 GPU processing failed, switching to CPU mode
2018-02-22 15:32:24 [CPU] estimating 1157x1962x384 disparity using 1157x981x8u tiles
2018-02-22 15:32:24 timings: rectify: 0.021 disparity: 0.292 borders: 0.028 filter: 0.108 fill: 0
2018-02-22 15:32:24 Error: Kernel failed: the launch timed out and was terminated (6) at line 821
2018-02-22 15:32:24 GPU processing failed, switching to CPU mode
2018-02-22 15:32:24 [CPU] estimating 1381x7424x160 disparity using 1381x1061x8u tiles
2018-02-22 15:32:24 Error: Kernel failed: the launch timed out and was terminated (6) at line 170
2018-02-22 15:32:24 GPU processing failed, switching to CPU mode
2018-02-22 15:32:24 [CPU] estimating 1126x1722x288 disparity using 1126x861x8u tiles
2018-02-22 15:32:25
2018-02-22 15:32:25 Depth reconstruction devices performance:
2018-02-22 15:32:25  - 36%    done by GeForce GTX 1080 Ti
2018-02-22 15:32:25  - 32%    done by GeForce GTX 1080 Ti
2018-02-22 15:32:25  - 32%    done by GeForce GTX 1080 Ti
2018-02-22 15:32:25 Total time: 664.673 seconds
2018-02-22 15:32:25
2018-02-22 15:32:25 Warning: cudaStreamDestroy failed: all CUDA-capable devices are busy or unavailable (46)
2018-02-22 15:32:25 Warning: cudaStreamDestroy failed: all CUDA-capable devices are busy or unavailable (46)
2018-02-22 15:32:25 Warning: cudaStreamDestroy failed: all CUDA-capable devices are busy or unavailable (46)
2018-02-22 15:32:25 Warning: cudaStreamDestroy failed: all CUDA-capable devices are busy or unavailable (46)
2018-02-22 15:32:25 Warning: cudaStreamDestroy failed: all CUDA-capable devices are busy or unavailable (46)
2018-02-22 15:32:26 Warning: cudaStreamDestroy failed: all CUDA-capable devices are busy or unavailable (46)
2018-02-22 15:32:26 Finished processing in 691.457 sec (exit code 0)
2018-02-22 15:32:26 Error: the launch timed out and was terminated (6) at line 198
--- End quote ---

Navigation

[0] Message Index

[#] Next page