GPU error CUDA_ERROR_OUT_OF_MEMORY (2) at line 123

[eiscar]

Hi,

I am running Photoscan through the Python API and trying to process some images acquired with a MAVIC drone. The script has worked without issues with other datasets from different cameras, but fails with the images from the MAVIC with a CUDA_ERROR_OUT_OF_MEMORY (2) at line 123 message. I have tried reducing the image resultion to a quarter, but the error persists. I am using version 1.4.3, build 6529.

I have attached the complete error message:

Code: [Select]

MatchPhotos: accuracy = High, preselection = generic, reference, keypoint limit = 40000, tiepoint limit = 4000, apply masks = 0, filter tie points = 0
Using device: TITAN V, 80 compute units, 12066 MB global memory, compute capability 7.0
  driver version: 9020, runtime version: 5050
  max work group size 1024
  max work item sizes [1024, 1024, 64]
Using device: TITAN V, 80 compute units, 12066 MB global memory, compute capability 7.0
  driver version: 9020, runtime version: 5050
  max work group size 1024
  max work item sizes [1024, 1024, 64]
Using device: TITAN V, 80 compute units, 12066 MB global memory, compute capability 7.0
  driver version: 9020, runtime version: 5050
  max work group size 1024
  max work item sizes [1024, 1024, 64]
Using device: TITAN V, 80 compute units, 12066 MB global memory, compute capability 7.0
  driver version: 9020, runtime version: 5050
  max work group size 1024
  max work item sizes [1024, 1024, 64]
[GPU] photo 1: 40000 points
[GPU] photo 3: 40000 points
Warning: cudaStreamDestroy failed: out of memory (2)
Warning: cudaStreamDestroy failed: out of memory (2)
Traceback (most recent call last):
  File "/media/bucket/tmp_code/photoscan-drono-monocular.py", line 133, in <module>
    main()
  File "/media/bucket/tmp_code/photoscan-drono-monocular.py", line 126, in main
    process(args.input_path, output_path)
  File "/media/bucket/tmp_code/photoscan-drono-monocular.py", line 67, in process
    chunk.matchPhotos(accuracy = accuracy, generic_preselection = generic_preselection, reference_preselection = reference_preselection, filter_mask = False, keypoint_limit = keypoints, tiepoint_limit = tiepoints)
RuntimeError: CUDA_ERROR_OUT_OF_MEMORY (2) at line 123


Any ideas? Thanks!

[Alexey Pasumansky]

Hello eiscar,

Which NVIDIA driver version you are using?

Does the processing work, if you have only one GPU enabled?

[eiscar]

Looking at the output of NVIDIA-SMI command I found out that two of the GPUs were being used by another process. Disabling those GPUs through the mask solved the out of memory error. The reconstruction is now running and seems to be working on the other two GPUs, but I have seen a few of the  "Warning: cudaStreamDestroy failed: out of memory (2)" warnings pop up.

Just in case, the driver version is 396.37.

[chadfx]

Hitting the exact same error. Nothing fancy in my setup, but I did swap out a GTX 780 for a 1070ti recently.
If I disable the 780, leaving just the 1070ti, it runs fine.

The project is a small number of moderate res TIF images and has run fine in the past.

NVidia driver is 398.36 in Win764

 AlignPhotos: accuracy = Highest, preselection = generic, keypoint limit = 80000, tiepoint limit = 20000, apply masks = 0, filter tie points = 0, adaptive fitting = 1
2018-08-20 15:16:25 Matching photos...
2018-08-20 15:16:25 Detecting points...
2018-08-20 15:16:25 Using device: GeForce GTX 1070 Ti, 19 compute units, 8192 MB global memory, compute capability 6.1
2018-08-20 15:16:25   driver version: 9020, runtime version: 5050
2018-08-20 15:16:25   max work group size 1024
2018-08-20 15:16:25   max work item sizes [1024, 1024, 64]
2018-08-20 15:16:25 Using device: GeForce GTX 780, 12 compute units, 6144 MB global memory, compute capability 3.5
2018-08-20 15:16:25   driver version: 9020, runtime version: 5050
2018-08-20 15:16:25   max work group size 1024
2018-08-20 15:16:25   max work item sizes [1024, 1024, 64]
2018-08-20 15:16:26 [GPU] photo 0: 80000 points
2018-08-20 15:16:26 Finished processing in 1.529 sec (exit code 0)
2018-08-20 15:16:26 Error: CUDA_ERROR_OUT_OF_MEMORY (2) at line 123

[chadfx]

I've since updated the NVidia driver to 398.82 and I am still getting the error if both graphics cards are enabled.

[Alexey Pasumansky]

Hello chadfx,

Are you observing the similar errors during the depth maps calculation stage?

Also does the problem persists for GTX 780 card when alignment is run on High accuracy instead of Highest?

[chadfx]

Hello Alexey, thanks for following up on this.

I just tried to run the Build Dense Cloud phase which also failed, but it did seem to try and do a few steps of reconstructing depth before it failed. The console said that the 780 did 100% of the calculation and the 1070ti 0%.

It also gave the same error, but with an additional error; "Warning: cudaStreamDestroy failed: out of memory (2)

I'm running 1.4.3 build 6529

[chadfx]

I think there must be some issue between these two graphics cards. I have another piece of software which also is having semi-cryptic error messages with both cards. I actually have to pull out the 780 in order to get the other software to work.

I am not much of a PC guy, so I feel very limited in my troubleshooting abilities in this area. I'm happy to follow instructions if you have some ideas of where I can look for issues.

Cheers, -Chad

[chadfx]

Hi, just one last update. With the latest graphics driver update from Nvidia last week (v399.07) the problem seems to have been resolved.

Cheers, -Chad

[Alexey Pasumansky]

Hello Chad,

That's good to hear, we'll try to suggest driver update for other users who might be experiencing similar problems.

The mentioned update seems to resolve also the visualization problems that some Windows 10 users are experiencing.