Crash: Processing dense cloud

[The_smiley]

hello

i keep getting a crash when trying to process a dense cloud

Win 10, i9-9880H with a RTX 4000 (Notebook)

Code: [Select]

2020-04-21 12:47:13 BuildDenseCloud: quality = Medium, depth filtering = Mild, point colors = 1
2020-04-21 12:47:13 Generating depth maps...
2020-04-21 12:47:13 Initializing...
2020-04-21 12:47:13 sorting point cloud... done in 0.039 sec
2020-04-21 12:47:13 processing matches... done in 0.02 sec
2020-04-21 12:47:13 initializing...
2020-04-21 12:47:14 selected 89 cameras from 89 in 0.332 sec
2020-04-21 12:47:14 groups: 17 18 18 18 18
2020-04-21 12:47:14 330 of 89 used (370.787%)
2020-04-21 12:47:14 scheduled 5 depth map groups
2020-04-21 12:47:14 saved depth map partition in 0.01 sec
2020-04-21 12:47:15 loaded depth map partition in 1.041 sec
2020-04-21 12:47:15 Initializing...
2020-04-21 12:47:15 Found 2 GPUs in 0.001 sec (CUDA: 0.001 sec, OpenCL: 0 sec)
2020-04-21 12:47:15 Using device: Quadro RTX 4000, 40 compute units, free memory: 6605/8192 MB, compute capability 7.5
2020-04-21 12:47:15   driver/runtime CUDA: 10020/8000
2020-04-21 12:47:15   max work group size 1024
2020-04-21 12:47:15   max work item sizes [1024, 1024, 64]
2020-04-21 12:47:15 Using CUDA device 'Quadro RTX 4000' in concurrent. (2 times)
2020-04-21 12:47:15 Loading photos...
2020-04-21 12:47:18 loaded photos in 3.053 seconds
2020-04-21 12:47:18 Generating depth maps...
2020-04-21 12:47:19 [GPU] estimating 1083x1109x160 disparity using 1083x1120x8u tiles
2020-04-21 12:47:19 [GPU] estimating 948x957x160 disparity using 948x960x8u tiles
2020-04-21 12:47:19 timings: rectify: 0.358 disparity: 0.304 borders: 0.182 filter: 0.18 fill: 0
2020-04-21 12:47:19 [GPU] estimating 857x940x192 disparity using 857x944x8u tiles
2020-04-21 12:47:20 timings: rectify: 0.729 disparity: 0.628 borders: 0.181 filter: 0.166 fill: 0
2020-04-21 12:47:20 [GPU] estimating 1093x1065x192 disparity using 1093x1088x8u tiles
2020-04-21 12:47:20 timings: rectify: 0.141 disparity: 0.524 borders: 0.107 filter: 0.205 fill: 0
2020-04-21 12:47:20 [GPU] estimating 1003x915x128 disparity using 1003x928x8u tiles
2020-04-21 12:47:21 timings: rectify: 0.112 disparity: 0.543 borders: 0.23 filter: 0.112 fill: 0
2020-04-21 12:47:21 timings: rectify: 0.16 disparity: 0.712 borders: 0.285 filter: 0.207 fill: 0
2020-04-21 12:47:22 [GPU] estimating 860x700x96 disparity using 860x704x8u tiles
2020-04-21 12:47:22 [GPU] estimating 759x705x96 disparity using 759x720x8u tiles
2020-04-21 12:47:22 timings: rectify: 0.132 disparity: 0.25 borders: 0.143 filter: 0.046 fill: 0
2020-04-21 12:47:22 timings: rectify: 0.069 disparity: 0.219 borders: 0.151 filter: 0.142 fill: 0
2020-04-21 12:47:22 [GPU] estimating 935x1887x128 disparity using 935x960x8u tiles
2020-04-21 12:47:22 [GPU] estimating 818x1834x192 disparity using 818x928x8u tiles
2020-04-21 12:47:24 timings: rectify: 0.289 disparity: 0.961 borders: 0.417 filter: 0.2 fill: 0
2020-04-21 12:47:24 timings: rectify: 0.353 disparity: 0.896 borders: 0.417 filter: 0.306 fill: 0
2020-04-21 12:47:24 [GPU] estimating 707x1827x320 disparity using 707x928x8u tiles
2020-04-21 12:47:24 [GPU] estimating 623x1734x160 disparity using 623x896x8u tiles
2020-04-21 12:47:25 timings: rectify: 0.296 disparity: 0.679 borders: 0.249 filter: 0.155 fill: 0
2020-04-21 12:47:26 [GPU] estimating 1061x1214x256 disparity using 1061x1216x8u tiles
2020-04-21 12:47:26 timings: rectify: 0.296 disparity: 0.859 borders: 0.273 filter: 0.249 fill: 0
2020-04-21 12:47:26 [GPU] estimating 1081x880x224 disparity using 1081x896x8u tiles
2020-04-21 12:47:27 timings: rectify: 0.217 disparity: 0.597 borders: 0.23 filter: 0.231 fill: 0
2020-04-21 12:47:27 timings: rectify: 0.143 disparity: 0.549 borders: 0.19 filter: 0.284 fill: 0
2020-04-21 12:47:27 [GPU] estimating 1470x1092x416 disparity using 1470x1120x8u tiles
2020-04-21 12:47:27 [GPU] estimating 1351x1054x384 disparity using 1351x1056x8u tiles
2020-04-21 12:47:29 timings: rectify: 0.26 disparity: 1.575 borders: 0.42 filter: 0.114 fill: 0
2020-04-21 12:47:30 timings: rectify: 0.332 disparity: 1.617 borders: 0.394 filter: 0.268 fill: 0
2020-04-21 12:47:30 [GPU] estimating 1302x1150x384 disparity using 1302x1152x8u tiles
2020-04-21 12:47:30 [GPU] estimating 854x1179x192 disparity using 854x1184x8u tiles
2020-04-21 12:47:31 timings: rectify: 0.219 disparity: 1.044 borders: 0.266 filter: 0.131 fill: 0
2020-04-21 12:47:31 timings: rectify: 0.291 disparity: 0.974 borders: 0.366 filter: 0.317 fill: 0
2020-04-21 12:47:31 [GPU] estimating 716x1124x192 disparity using 716x1152x8u tiles
2020-04-21 12:47:32 timings: rectify: 0.145 disparity: 0.266 borders: 0.185 filter: 0.16 fill: 0
2020-04-21 12:47:32 [GPU] estimating 702x945x192 disparity using 702x960x8u tiles
2020-04-21 12:47:33 timings: rectify: 0.109 disparity: 0.571 borders: 0.179 filter: 0.176 fill: 0
2020-04-21 12:47:33 [GPU] estimating 935x1887x128 disparity using 935x960x8u tiles
2020-04-21 12:47:33 [GPU] estimating 953x1955x160 disparity using 953x992x8u tiles
2020-04-21 12:47:35 timings: rectify: 0.293 disparity: 0.996 borders: 0.352 filter: 0.283 fill: 0
2020-04-21 12:47:35 timings: rectify: 0.237 disparity: 1.011 borders: 0.469 filter: 0.373 fill: 0
2020-04-21 12:47:35 [GPU] estimating 876x1948x256 disparity using 876x992x8u tiles
2020-04-21 12:47:36 [GPU] estimating 768x1893x160 disparity using 768x960x8u tiles
2020-04-21 12:47:38 Error: Error: Kernel failed: the launch timed out and was terminated (6) at line 333Kernel failed: the launch timed out and was terminated (6) at line 115
2020-04-21 12:47:38
2020-04-21 12:47:38 GPU processing failed, switching to CPU mode
2020-04-21 12:47:38 GPU processing failed, switching to CPU mode
2020-04-21 12:47:38 [CPU] estimating 768x1893x160 disparity using 768x947x8u tiles
2020-04-21 12:47:38 [CPU] estimating 876x1948x256 disparity using 876x974x8u tiles
2020-04-21 12:47:40
2020-04-21 12:47:40 Depth reconstruction devices performance:
2020-04-21 12:47:40  - 100% done by Quadro RTX 4000
2020-04-21 12:47:40 Total time: 22.145 seconds
2020-04-21 12:47:40
2020-04-21 12:47:40 Warning: cudaStreamDestroy failed: the launch timed out and was terminated (6)
2020-04-21 12:47:40 Warning: cudaStreamDestroy failed: the launch timed out and was terminated (6)
2020-04-21 12:47:41 Finished processing in 27.276 sec (exit code 0)
2020-04-21 12:47:41 Error: cudaMemGetInfo(&free_mem_size, &total_mem_size): the launch timed out and was terminated (6) at line 211
>>>

[Alexey Pasumansky]

Hello The_smiley,

Do you have the latest NVIDIA drivers installed on your system? I can suggest a clean driver install and system reboot:
https://www.nvidia.com/Download/driverResults.aspx/159682/en-us

[The_smiley]

Hello The_smiley,

Do you have the latest NVIDIA drivers installed on your system? I can suggest a clean driver install and system reboot:
https://www.nvidia.com/Download/driverResults.aspx/159682/en-us

Hi

I updated the driver and ran a clean install. still getting the same error.

[Alexey Pasumansky]

Hello The_smiley,

Please then try to create a new tweak via Advanced Preferences tab -> Tweaks button with the following name:
main/depth_max_gpu_multiplier
and set its value to 1 (the default is 2).

Then try to re-start the processing again and check if it works.

[The_smiley]

Hi

That seems to have sorted out my issues,

Thank you!