All cuda-capable devices are busy or unavailable

[xiyanguva]

Hello all,

We are experiencing this error when PS is working on dense point clouds processing of ~800 images. We use NVIDIA Quadro P6000. It is a Windows 10 system. The GPU first works fine for a while and then it gave the error that:

Warning: cudaStreamDestroy failed: all CUDA-capable devices are busy or unavailable (46)

I've added the depth gpu multiplier in the regedit. And I have turned off the CPU in the GPU tab.

Then the time it takes to finish the processing increased an order of magnitude. Here a short summary of the log:

2018-05-20 23:40:13 [GPU] estimating 988x1798x160 disparity using 988x899x8u tiles
2018-05-20 23:40:14 timings: rectify: 0.014 disparity: 0.179 borders: 0.01 filter: 0.037 fill: 0
2018-05-20 23:40:14 [GPU] estimating 1343x1887x128 disparity using 1343x944x8u tiles
2018-05-20 23:40:14 timings: rectify: 0.014 disparity: 0.213 borders: 0.015 filter: 0.05 fill: 0
2018-05-20 23:40:14 [GPU] estimating 1544x1529x128 disparity using 772x1529x8u tiles
2018-05-20 23:40:14 timings: rectify: 0.016 disparity: 0.199 borders: 0.015 filter: 0.051 fill: 0
2018-05-20 23:40:15 [GPU] estimating 1774x1773x128 disparity using 887x887x8u tiles
2018-05-20 23:40:15 timings: rectify: 0.017 disparity: 0.243 borders: 0.016 filter: 0.058 fill: 0
2018-05-20 23:40:15 [GPU] estimating 1084x1793x160 disparity using 1084x897x8u tiles
2018-05-20 23:40:15 timings: rectify: 0.014 disparity: 0.176 borders: 0.012 filter: 0.041 fill: 0
2018-05-20 23:40:15 [GPU] estimating 1320x1792x128 disparity using 1320x896x8u tiles
2018-05-20 23:40:15 timings: rectify: 0.016 disparity: 0.185 borders: 0.013 filter: 0.05 fill: 0
2018-05-20 23:40:15 [GPU] estimating 1331x1802x128 disparity using 1331x901x8u tiles
2018-05-20 23:40:16 timings: rectify: 0.017 disparity: 0.203 borders: 0.015 filter: 0.054 fill: 0
2018-05-20 23:40:16 [GPU] estimating 1185x1789x160 disparity using 1185x895x8u tiles
2018-05-20 23:40:16 timings: rectify: 0.015 disparity: 0.212 borders: 0.013 filter: 0.043 fill: 0
2018-05-20 23:40:16 [GPU] estimating 1579x1208x160 disparity using 790x1208x8u tiles
2018-05-20 23:40:17 timings: rectify: 0.012 disparity: 0.173 borders: 0.012 filter: 0.049 fill: 0
2018-05-20 23:40:17 [GPU] estimating 1523x1361x128 disparity using 1523x1361x8u tiles
2018-05-20 23:40:17 timings: rectify: 0.015 disparity: 0.169 borders: 0.012 filter: 0.045 fill: 0
2018-05-20 23:40:17 [GPU] estimating 1470x1525x128 disparity using 1470x1525x8u tiles
2018-05-20 23:40:17 timings: rectify: 0.014 disparity: 0.179 borders: 0.012 filter: 0.046 fill: 0
2018-05-20 23:40:17 [GPU] estimating 1068x1795x128 disparity using 1068x898x8u tiles
2018-05-20 23:40:17 timings: rectify: 0.015 disparity: 0.177 borders: 0.011 filter: 0.04 fill: 0
2018-05-20 23:40:17 [GPU] estimating 1315x1809x128 disparity using 1315x905x8u tiles
2018-05-20 23:40:18 timings: rectify: 0.016 disparity: 0.184 borders: 0.014 filter: 0.052 fill: 0
2018-05-20 23:40:18 [GPU] estimating 1324x1814x128 disparity using 1324x907x8u tiles
2018-05-20 23:40:18 timings: rectify: 0.017 disparity: 0.186 borders: 0.014 filter: 0.052 fill: 0
2018-05-20 23:40:18 [GPU] estimating 1084x1793x160 disparity using 1084x897x8u tiles
2018-05-20 23:40:18 timings: rectify: 0.015 disparity: 0.179 borders: 0.011 filter: 0.042 fill: 0
2018-05-20 23:40:19 [GPU] estimating 1384x1503x128 disparity using 1384x1503x8u tiles
2018-05-20 23:40:19 timings: rectify: 0.013 disparity: 0.173 borders: 0.015 filter: 0.049 fill: 0
2018-05-20 23:40:19 [GPU] estimating 1716x1227x128 disparity using 858x1227x8u tiles
2018-05-20 23:40:19 timings: rectify: 0.015 disparity: 0.177 borders: 0.014 filter: 0.046 fill: 0
2018-05-20 23:40:19 [GPU] estimating 1381x1662x128 disparity using 1381x831x8u tiles
2018-05-20 23:40:20 timings: rectify: 0.016 disparity: 0.198 borders: 0.012 filter: 0.045 fill: 0
2018-05-20 23:40:20 [GPU] estimating 988x1798x160 disparity using 988x899x8u tiles
2018-05-20 23:40:20 timings: rectify: 0.014 disparity: 0.179 borders: 0.01 filter: 0.04 fill: 0
2018-05-20 23:40:20 [GPU] estimating 1323x1814x128 disparity using 1323x907x8u tiles
2018-05-20 23:40:20 timings: rectify: 0.016 disparity: 0.2 borders: 0.014 filter: 0.051 fill: 0
2018-05-20 23:40:20 [GPU] estimating 1320x1792x128 disparity using 1320x896x8u tiles
2018-05-20 23:40:20 timings: rectify: 0.016 disparity: 0.2 borders: 0.014 filter: 0.051 fill: 0
2018-05-20 23:40:20 [GPU] estimating 1135x1788x160 disparity using 1135x894x8u tiles
2018-05-20 23:40:21 timings: rectify: 0.014 disparity: 0.199 borders: 0.011 filter: 0.043 fill: 0
2018-05-20 23:40:21
2018-05-20 23:40:21 Depth reconstruction devices performance:
2018-05-20 23:40:21  - 100%    done by Quadro P6000
2018-05-20 23:40:21 Total time: 928.687 seconds
2018-05-20 23:40:21
2018-05-20 23:40:22 Warning: cudaStreamDestroy failed: all CUDA-capable devices are busy or unavailable (46)
2018-05-20 23:40:22 Generating dense cloud...
2018-05-20 23:40:22 Generating dense point cloud...
2018-05-20 23:41:12 selected 800 cameras in 50.594 sec
2018-05-20 23:41:12 working volume: 13395x5441x1845
2018-05-20 23:41:12 tiles: 3x1x1
2018-05-20 23:41:12 selected 304 cameras in 0.003 sec
2018-05-20 23:41:12 preloading data... done in 17.637 sec
2018-05-20 23:41:30 filtering depth maps... done in 2376.28 sec
2018-05-21 00:21:07 preloading data... done in 9.995 sec
2018-05-21 00:21:19 working volume: 4465x5441x1845
2018-05-21 00:21:19 tiles: 1x1x1
2018-05-21 00:21:19 accumulating data... done in 9.479 sec
2018-05-21 00:21:38 building point cloud... done in 2.201 sec
2018-05-21 00:21:41 selected 573 cameras in 0.003 sec
2018-05-21 00:21:41 preloading data... done in 23.293 sec
2018-05-21 00:22:04 filtering depth maps... done in 5021.75 sec
2018-05-21 01:45:47 preloading data... done in 19.623 sec
2018-05-21 01:46:10 working volume: 4465x5441x1845
2018-05-21 01:46:10 tiles: 1x1x1
2018-05-21 01:46:10 accumulating data... done in 16.337 sec
2018-05-21 01:46:41 building point cloud... done in 3.582 sec
2018-05-21 01:46:47 selected 299 cameras in 0.002 sec
2018-05-21 01:46:47 preloading data... done in 12.93 sec
2018-05-21 01:47:00 filtering depth maps... done in 2128.28 sec
2018-05-21 02:22:29 preloading data... done in 10.241 sec
2018-05-21 02:22:41 working volume: 4465x5441x1845
2018-05-21 02:22:41 tiles: 1x1x1
2018-05-21 02:22:41 accumulating data... done in 11.015 sec
2018-05-21 02:23:00 building point cloud... done in 2.141 sec
2018-05-21 02:23:07 130079497 points extracted
2018-05-21 02:23:09 Finished processing in 10725.3 sec (exit code 1)

[Alexey Pasumansky]

Hello xiyanguva,

Which NVIDIA driver version you are using and whether it is the only GPU installed on this machine?

[xiyanguva]

Hi Alexey,

Thank you for the prompt reply! We are using the latest version of the driver (just updated a few days ago). I think it is 391.58 for quadro.

It is the only GPU we have.

[xiyanguva]

Hello,

Has the PhotoScan team fixed this issue? It happens all the time when we align the pictures. The GPU worked when it was finding the points, which take about 230 seconds. Then we get the CUDA warning. PhotoScan then started on finding the alignment, and it usually takes about 30 hours at least.