Crash at dense cloud processing - CUDA ERROR

[zapotek]

Hello,

I'm attempting to run a dense cloud using GPU. I have a nice setup, however, it crashes when I have the GPU checked in the preferences. The error is "CUDA_ERROR_UNKNOWN (999) at line 123." Please see screenshots attached.

Thank you,

[Alexey Pasumansky]

Hello zapotek,

Have you installed the drivers directly from NVIDIA web-site or via Windows tools?

Also can you provide the full log for the failed step (just copy from the Console pane)?

[zapotek]

Hi Alexy,

I installed the drivers via NVIDIA GeForce Experience.

Here is the log file:

2018-05-02 07:47:41 BuildDenseCloud: quality = Medium, depth filtering = Aggressive, point colors = 1
2018-05-02 07:47:41 Generating depth maps...
2018-05-02 07:47:41 Initializing...
2018-05-02 07:47:41 Using device: GeForce GTX 980 Ti, 22 compute units, 6144 MB global memory, compute capability 5.2
2018-05-02 07:47:41   driver version: 9020, runtime version: 5050
2018-05-02 07:47:41   max work group size 1024
2018-05-02 07:47:41   max work item sizes [1024, 1024, 64]
2018-05-02 07:47:41 Using CUDA device 'GeForce GTX 980 Ti' in concurrent. (2 times)
2018-05-02 07:47:41 Initializing...
2018-05-02 07:47:41 sorting point cloud... done in 1.98 sec
2018-05-02 07:47:43 processing matches... done in 0.035 sec
2018-05-02 07:47:43 initializing...
2018-05-02 07:47:44 selected 236 cameras from 236 in 0.731 sec
2018-05-02 07:47:45 Loading photos...
2018-05-02 07:48:04 loaded photos in 18.839 seconds
2018-05-02 07:48:04 Reconstructing depth...
2018-05-02 07:48:04 [CPU] estimating 1413x673x128 disparity using 1413x673x8u tiles
2018-05-02 07:48:05 timings: rectify: 0.02 disparity: 0.889 borders: 0.009 filter: 0.026 fill: 0
2018-05-02 07:48:05
2018-05-02 07:48:05 Depth reconstruction devices performance:
2018-05-02 07:48:05  - 100%    done by CPU
2018-05-02 07:48:05  - 0%    done by GeForce GTX 980 Ti
2018-05-02 07:48:05 Total time: 1.051 seconds
2018-05-02 07:48:05
2018-05-02 07:48:05 Finished processing in 24.15 sec (exit code 0)
2018-05-02 07:48:05 Error: CUDA_ERROR_UNKNOWN (999) at line 123
>>>

Thank you,

[abigailk]

I am also having similar issues when I run the dense cloud processing. I am only trying to process about 60 images at the lowest quality setting, and the processor immediately crashes each time. Is this a CPU issue?

Thanks


Here is my crash report:
ARB_texture_non_power_of_two: Yes
ARB_vertex_buffer_object: Yes
CPU: Intel(R) Core(TM) i5-3210M CPU @ 2.50GHz
Email: tuj23083@temple.edu
OpenGLMaxTextureSize: 16384
OpenGLRenderer: Intel HD Graphics 4000 OpenGL Engine
OpenGLStereo: No
OpenGLVendor: Intel Inc.
OpenGLVersion: 2.1 INTEL-8.28.37
ProductName: PhotoScan Pro
System: Mac OS X 64bit
SystemMemory: 8.00 GB
Throttleable: 1
Vendor: Agisoft
Version: 1.4.2.6205
XLog: ] processing 4008x2672 image using 1336x1336 tiles
[CPU] photo 27: 80000 points
[GPU] photo 29: 79998 points
[GPU] processing 4008x2672 image using 1336x1336 tiles
[GPU] photo 31: 79998 points
[GPU] processing 4008x2672 image using 1336x1336 tiles
[CPU] photo 30: 80000 points
[GPU] photo 32: 79998 points
[GPU] processing 4008x2672 image using 1336x1336 tiles
[GPU] photo 34: 79998 points
[GPU] processing 4008x2672 image using 1336x1336 tiles
[CPU] photo 33: 80000 points
[GPU] photo 35: 79998 points
[GPU] processing 4008x2672 image using 1336x1336 tiles
[GPU] photo 37: 79998 points
[CPU] photo 36: 80000 points
points detected in 87.3777 sec
Selecting pairs...
Using device: HD Graphics 4000, 16 compute units, 1024 MB global memory, OpenCL 1.2
  driver version: 1.2(Jan 11 2016 18:59:21), platform version: OpenCL 1.2 (Nov  2 2015 15:02:14)
  max work group size 512
  max work item sizes [512, 512, 512]
  max mem alloc size 256 MB
8806 matches found in 9.1477 sec
matches combined in 0.001568 sec
matches filtered in 0.008664 sec
56 pairs selected in 1.2e-05 sec
Matching points...
Using device: HD Graphics 4000, 16 compute units, 1024 MB global memory, OpenCL 1.2
  driver version: 1.2(Jan 11 2016 18:59:21), platform version: OpenCL 1.2 (Nov  2 2015 15:02:14)
  max work group size 512
  max work item sizes [512, 512, 512]
  max mem alloc size 256 MB
12718 matches found in 139.74 sec
matches combined in 0.001847 sec
matches filtered in 0.036072 sec
finished matching in 236.36 sec
setting point indices... 4656 done in 0.012891 sec
generated 4656 tie points, 2.24055 average projections
removed 28 multiple indices
removed 3 tracks
Estimating camera locations...
Estimating camera locations...
processing matches... done in 0.000906 sec
selecting camera groups... done in 0.000479 sec
processing block: 10 photos
pair 23 and 24: 228 robust from 511
pair 22 and 23: 204 robust from 382
pair 24 and 25: 132 robust from 267
adding photos 23 and 24 (228 robust)
adding 511 points, 0 far (6.68 threshold), 4 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxx 0.182883 -> 0.182868
adding 4 points, 0 far (6.68 threshold), 4 inaccurate, 0 invisible, 0 weak
optimized in 0.048556 seconds
adding camera 25 (3 of 10), 89 of 89 used
adding camera 22 (4 of 10), 87 of 87 used
adding 520 points, 0 far (6.68 threshold), 3 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxx 0.267408 -> 0.26448
adding 3 points, 0 far (6.68 threshold), 2 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxx 0.26447 -> 0.262278
adding 2 points, 0 far (6.68 threshold), 2 inaccurate, 0 invisible, 0 weak
optimized in 0.146359 seconds
adding camera 26 (5 of 10), 58 of 58 used
adding 97 points, 0 far (6.68 threshold), 2 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxx 0.260692 -> 0.258505
adding 2 points, 0 far (6.68 threshold), 2 inaccurate, 0 invisible, 0 weak
optimized in 0.079312 seconds
adding camera 27 (6 of 10), 31 of 31 used
adding 82 points, 0 far (6.68 threshold), 2 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxx 0.286923 -> 0.284518
adding 2 points, 0 far (6.68 threshold), 2 inaccurate, 0 invisible, 0 weak
optimized in 0.080151 seconds
adding camera 28 (7 of 10), 15 of 15 used
adding 58 points, 0 far (6.68 threshold), 2 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxx 0.287728 -> 0.286871
adding 2 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxx 0.286739 -> 0.282856
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.160901 seconds
adding camera 29 (8 of 10), 11 of 11 used
adding 34 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxx 0.283463 -> 0.275964
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.08323 seconds
3 sigma filtering...
adjusting: xxxxxxxxxxxxxxxxxxxx 0.274732 -> 0.254754
point variance: 0.205344 threshold: 0.616033
adding 0 points, 23 far (0.616033 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxxxxxxxx 0.234257 -> 0.1741
point variance: 0.139912 threshold: 0.419736
adding 4 points, 28 far (0.419736 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxxxxxxxxxxxx 0.157491 -> 0.157357
point variance: 0.126411 threshold: 0.379234
adding 1 points, 15 far (0.379234 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxxxxxxxxxxxx 0.150993 -> 0.150791
point variance: 0.120902 threshold: 0.362707
adding 1 points, 8 far (0.362707 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxxxxxxxxxxxx 0.148022 -> 0.147013
point variance: 0.117573 threshold: 0.352719
adding 0 points, 5 far (0.352719 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.732586 seconds
f 3939.55, cx 0, cy 0, k1 0, k2 0, k3 0
finished sfm in 1.42039 seconds
loaded projections in 4.3e-05 sec
tracks initialized in 7.3e-05 sec
adding 1292 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
processing block: 21 photos
pair 17 and 18: 685 robust from 685
adding photos 17 and 18 (685 robust)
adding 685 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxx 0.245626 -> 0.245473
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.051136 seconds
adding camera 19 (3 of 21), 165 of 165 used
adding camera 16 (4 of 21), 139 of 139 used
adding 799 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxx 0.323339 -> 0.247197
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.072038 seconds
adding camera 20 (5 of 21), 171 of 172 used
adding 255 points, 1 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxx 0.302336 -> 0.291938
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.094808 seconds
adding camera 21 (6 of 21), 79 of 79 used
adding 68 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxx 0.31499 -> 0.258886
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.085065 seconds
adding camera 15 (7 of 21), 11 of 11 used
adding 64 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxx 0.262994 -> 0.259239
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.07736 seconds
adding camera 14 (8 of 21), 10 of 10 used
adding 84 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxx 0.275821 -> 0.258963
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.065925 seconds
adding camera 13 (9 of 21), 28 of 28 used
adding 102 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxx 0.265229 -> 0.256543
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.096857 seconds
adding camera 12 (10 of 21), 17 of 17 used
adding 74 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxx 0.259847 -> 0.257216
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.082988 seconds
adding camera 11 (11 of 21), 18 of 18 used
adding 134 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxx 0.291702 -> 0.259931
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.099222 seconds
adding camera 10 (12 of 21), 28 of 28 used
adding 170 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxx- 0.274209 -> 0.260883
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.12611 seconds
adding camera 9 (13 of 21), 35 of 35 used
adding 112 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxx 0.271645 -> 0.260825
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.094991 seconds
adding camera 8 (14 of 21), 12 of 12 used
adding 104 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxx 0.262261 -> 0.260002
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.079426 seconds
adding camera 7 (15 of 21), 8 of 8 used
adding 74 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxx 0.331956 -> 0.263662
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.122331 seconds
adding camera 6 (16 of 21), 5 of 5 used
adding 83 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxx 0.278927 -> 0.26316
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.123324 seconds
adding camera 5 (17 of 21), 15 of 15 used
adding 143 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxx- 0.26674 -> 0.261016
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.149326 seconds
adding camera 4 (18 of 21), 14 of 14 used
adding 134 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxx- 0.263146 -> 0.260076
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.108287 seconds
adding camera 3 (19 of 21), 9 of 9 used
adding 153 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxx 0.264209 -> 0.25898
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.133063 seconds
adding camera 2 (20 of 21), 5 of 5 used
adding 30 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxx 0.325623 -> 0.259635
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.140119 seconds
3 sigma filtering...
adjusting: xxxxxxxxxx 0.259532 -> 0.259531
point variance: 0.208515 threshold: 0.625545
adding 0 points, 67 far (0.625545 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxxxxxxxxxxxx 0.2176 -> 0.216742
point variance: 0.172819 threshold: 0.518457
adding 0 points, 60 far (0.518457 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxxxxxxxxxxxx 0.20034 -> 0.198384
point variance: 0.157621 threshold: 0.472863
adding 1 points, 29 far (0.472863 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxxxxxxxxxxxx 0.192883 -> 0.192565
point variance: 0.152508 threshold: 0.457525
adding 0 points, 10 far (0.457525 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxx 0.190537 -> 0.190408
point variance: 0.150761 threshold: 0.452282
adding 1 points, 7 far (0.452282 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 1.2091 seconds
f 4177.25, cx 0, cy 0, k1 0, k2 0, k3 0
finished sfm in 3.12177 seconds
loaded projections in 5e-05 sec
tracks initialized in 0.000266 sec
adding 3267 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
processing block: 2 photos
pair 32 and 33: 11 robust from 11
adding photos 32 and 33 (11 robust)
adding 11 points, 0 far (6.68 threshold), 0 inaccurate, 1 invisible, 0 weak
adjusting: xxxxxxxxxx 1.04392 -> 0.861071
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.012088 seconds
3 sigma filtering...
adjusting: xxxxxxxxxxxxxxxxxxxx 0.861071 -> 0.81436
point variance: 0.397367 threshold: 1.1921
adding 0 points, 0 far (1.1921 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxxxxxxxxxxxx 0.81436 -> 0.796022
point variance: 0.388419 threshold: 1.16526
adding 0 points, 0 far (1.16526 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxxxxxxxxxxxx 0.796022 -> 0.770688
point variance: 0.376057 threshold: 1.12817
adding 0 points, 0 far (1.12817 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxxxxxxxxxxxx 0.770688 -> 0.751686
point variance: 0.366785 threshold: 1.10035
adding 0 points, 0 far (1.10035 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxxxxxxxxxxxx 0.751686 -> 0.738741
point variance: 0.360469 threshold: 1.08141
adding 0 points, 0 far (1.08141 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.146788 seconds
f 7110.4, cx 0, cy 0, k1 0, k2 0, k3 0
finished sfm in 0.160758 seconds
loaded projections in 4.6e-05 sec
tracks initialized in 4e-06 sec
adding 10 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
processing block: 1 photos
loaded projections in 3.9e-05 sec
tracks initialized in 0 sec
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
processing block: 1 photos
loaded projections in 4.2e-05 sec
tracks initialized in 0 sec
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
processing block: 1 photos
loaded projections in 3.6e-05 sec
tracks initialized in 0 sec
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
processing block: 1 photos
loaded projections in 4.1e-05 sec
tracks initialized in 0 sec
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
processing block: 1 photos
loaded projections in 4.4e-05 sec
tracks initialized in 1e-06 sec
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
8 blocks: 10 21 2 1 1 1 1 1
calculating match counts... done in 9.9e-05 sec
overlapping groups selected in 5e-06 sec
8 blocks: 21 10 2 1 1 1 1 1
block: 2 sensors, 31 cameras, 4642 points, 10379 projections
block_sensors: 0.000869751 MB (0.000869751 MB allocated)
block_cameras: 0.00969696 MB (0.0131378 MB allocated)
block_points: 0.212494 MB (0.375 MB allocated)
block_tracks: 0.0177078 MB (0.0177078 MB allocated)
block_obs: 0.197964 MB (0.197964 MB allocated)
block_ofs: 0.0354233 MB (0.0354233 MB allocated)
block_fre: 0 MB (0 MB allocated)
adding 4559 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxxxxxxxxxxxx 0.248165 -> 0.245704
adding 0 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 0.463171 seconds
f 4173.39, cx 0, cy 0, k1 0, k2 0, k3 0
f 3731.5, cx 0, cy 0, k1 0, k2 0, k3 0
adjusting: xxxxxxxxxxxxxxxxxxxx 18.6533 -> 0.243285
final block size: 31
adding 4559 points, 0 far (6.68 threshold), 0 inaccurate, 0 invisible, 0 weak
3 sigma filtering...
adjusting: xxxxxxxxxxxxxxxxx 0.244615 -> 0.243285
point variance: 0.195551 threshold: 0.586653
adding 0 points, 95 far (0.586653 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxxxxxxxxxxxx 0.204168 -> 0.202932
point variance: 0.162055 threshold: 0.486166
adding 0 points, 72 far (0.486166 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxxxxxxxxxxxx 0.18958 -> 0.189002
point variance: 0.150428 threshold: 0.451284
adding 0 points, 41 far (0.451284 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxxxxx 0.18284 -> 0.181803
point variance: 0.144496 threshold: 0.433489
adding 0 points, 33 far (0.433489 threshold), 0 inaccurate, 0 invisible, 0 weak
adjusting: xxxxxxxxxxxxxxxxxx 0.177474 -> 0.177287
point variance: 0.140678 threshold: 0.422035
adding 1 points, 18 far (0.422035 threshold), 0 inaccurate, 0 invisible, 0 weak
optimized in 1.80489 seconds
coordinates applied in 0 sec
Finished processing in 243.834 sec (exit code 1)
BuildDenseCloud: quality = Lowest, depth filtering = Mild, point colors = 1
Generating depth maps...
Initializing...
Using device: HD Graphics 4000, 16 compute units, 1024 MB global memory, OpenCL 1.2
  driver version: 1.2(Jan 11 2016 18:59:21), platform version: OpenCL 1.2 (Nov  2 2015 15:02:14)
  max work group size 512
  max work item sizes [512, 512, 512]
  max mem alloc size 256 MB
Using OpenCL device 'HD Graphics 4000' in concurrent. (2 times)
Initializing...
sorting point cloud... done in 0.000128 sec
processing matches... done in 0.000763 sec
initializing...
selected 26 cameras from 28 in 0.018445 sec
Loading photos...
loaded photos in 3.84343 seconds
Reconstructing depth...
Building kernels for HD Graphics 4000...
[CPU] estimating 152x271x96 disparity using 152x271x8u tiles

[Alexey Pasumansky]

Hello abigailk,

Your problem is related to Intel HD card. The impact of the integrated graphics is usually insignificant, so if the driver support for Intel HD cards on Mac OS X is poor, just disable it in the GPU Preferences tab of PhotoScan.

[Alexey Pasumansky]

Hello zapotek,

Do you experience the same (or similar) issue during the image matching operation with GPU acceleration?

[zapotek]

Yes, I just tried the alignment process and the same error occurred. I even upgraded to the latest version of Photoscan.

[andyroo]

I just got the same CUDA_ERROR_UNKNOWN (999) at line 123 message. Only thing I did different from previous models was upgrade from 1.4.1 (build 5125) to 1.4.2 (build 6205) failed in batch but didn't crash photoscan. Nvidia driver version 390.77, two GTX 980Ti and one GTX TITAN X.

They only notable difference in my log from previous poster is that mine shows 3% of the work is done by CPU with the rest mostly evenly divided among GPUs. Total dense cloud reconstruction reported at ~3 seconds.

Machine is not in network and I'm typing this on my cell sorry.

[andyroo]

#Update# I just re-ran outside of batch to see if I got the same results and they're is an additional error reported after the time total and card breakout. There is one empty line in the log after "Total time 3.703 seconds" then:

Warning: cudastreamdestroy failed: all CUDA-capable devices are busy or unavailable (46)

Then there's finished processing in 81.537 sec (exit code 0)

No crash reports in AppData.

[Alexey Pasumansky]

Hello andyroo,

Does the problem persists, if you disable CPU utilization for GPU-supported tasks?

Also are you observing the same behavior for image matching and depth maps generation procedures?

[andyroo]

Hi Alexey,

The problem still occures if I turn off CPU utilization during GPU. I just did that, and got the same errors (46) and (999).

I did not get any error during initial alignment or optimization and filtering with gradual selection, only during dense cloud reconstruction.

Happy to run something in debug if you want, (I'm on build 6205). Or I can provide more details on anything. The project is about 1700 aligned photos. Probably 10-15GB if I upload it.

Andy

[xiyanguva]

We got the same error with the latest version 1.4.2 of Agisoft. We are using NVIDIA Quadro P6000. Interestingly, 1.3.5 cannot even “see” the GPU

[phillyphanatic]

We got the same error with the latest version 1.4.2 of Agisoft. We are using NVIDIA Quadro P6000. Interestingly, 1.3.5 cannot even “see” the GPU

Seconded. Having the exact same issue with a Quadro M6000 24GB GPU.

[chort]

This seems to be referencing the same error and they've found and fixed the culprit as part of Python libs.
https://github.com/pytorch/pytorch/issues/20635

I still have this happening randomly on 8 different workstations with RTX2080Ti, GTX1080Ti and TitanX cards, all running version 1.4.5 and latest Stable Studio drivers (431.70).

It is not extremely common, but I have a crash every cca 1000 projects or so.

[Alexey Pasumansky]

Hello chort,

It would be helpful, if you could provide the processing log related to the failed operation and submit the crash report (with the link to this thread in the comments field), so that we could check it.