cudaMemGetInfo time out error

[Alexey Pasumansky]

Hello Filip,

There shouldn't be any noticeable increase of the processing time, when switching to OpenCL implementation.

[ttsesm]

Hi Alexey, I am getting the same error with RTX2080 super on my linux machine. I would like to test the OpenCL option, is there a way to set this in my script through python?

Moreover, adding the suggested tweak on the gui seems to work for the Dense Point cloud creation, but then on the Mesh creation I am getting the following error:

Code: [Select]

ciErrNum: CL_UNKNOWN_ERROR_CODE_-9999 (-9999) at line 209
Which I guess is related to opencl.

[Alexey Pasumansky]

Hello ttsesm,

To switch to OpenCL implementation via Python, you need to add the following line to the beginning of your script:

Code: [Select]

Metashape.app.settings.setValue("main/gpu_enable_cuda", "0")
As for the error that you are observing, please provide the related log  corresponding to the failed operation. Also specify Linux distribution used and NVIDIA driver version.

[ttsesm]

Thanks Alexey, setting the parameter worked.

However after switching to OpenCL the error that I get is the following:

Code: [Select]

...
...
Using device: NVIDIA GeForce RTX 2080 SUPER, 48 compute units, 7980 MB global memory, OpenCL 3.0
  driver version: 510.54, platform version: OpenCL 3.0 CUDA 11.6.110
  max work group size 1024
  max work item sizes [1024, 1024, 64]
  max mem alloc size 1995 MB
  warp size 32
Building OpenCL kernels for NVIDIA GeForce RTX 2080 SUPER...
Kernels compilation done in 2.83562 seconds
Building OpenCL kernels for NVIDIA GeForce RTX 2080 SUPER...
Kernels compilation done in 0.646988 seconds
Traceback (most recent call last):
  File "/home/ttsesm/Development/metashape_project/bundler_extractor.py", line 68, in <module>
    main()
  File "/home/ttsesm/Development/metashape_project/bundler_extractor.py", line 47, in main
    chunk.matchPhotos()
Exception: Kernel locatePoints: clWaitForEvents(1, &ev): CL_UNKNOWN_ERROR_CODE_-9999 (-9999) at line 638

Process finished with exit code 1
My linux distribution is Arch linux, fully updated to the latest packages. The nvidia driver version is again the latest and specifically v.510.54 as you can see above in the output.

Is there any other log that I can provide you? I am running the script through pycharm with the metashape interpreter as described here https://agisoft.freshdesk.com/support/solutions/articles/31000154762-how-to-make-python-interpreter-to-use-metashape-module

------------------------------------------------------------------------------------

Also without the OpenCL workaround, initially the error usually I get is the following:

Code: [Select]

...
...
Found 1 GPUs in 0.000133 sec (CUDA: 7.3e-05 sec, OpenCL: 5.3e-05 sec)
Using device: NVIDIA GeForce RTX 2080 SUPER, 48 compute units, free memory: 6807/7980 MB, compute capability 7.5
  driver/runtime CUDA: 11060/10010
  max work group size 1024
  max work item sizes [1024, 1024, 64]
[GPU] photo 19: 8310 points
[GPU] photo 48: 8221 points
[GPU] photo 77: 8417 points
[GPU] photo 106: 7715 points
[GPU] photo 135: 6117 points
[GPU] photo 164: 8341 points
[GPU] photo 193: 9600 points
[GPU] photo 222: 8013 points
[GPU] photo 251: 9177 points
[GPU] photo 280: 9049 points
[GPU] photo 309: 7521 points
[GPU] photo 338: 9145 points
[GPU] photo 367: 8751 points
[GPU] photo 396: 8642 points
[GPU] photo 425: 9032 points
[GPU] photo 454: 8926 points
[GPU] photo 483: 9299 points
[GPU] photo 512: 8379 points
Warning: cudaStreamDestroy failed: an illegal memory access was encountered (700)
Traceback (most recent call last):
  File "/home/ttsesm/Development/metashape_project/bundler_extractor.py", line 68, in <module>
    main()
  File "/home/ttsesm/Development/metashape_project/bundler_extractor.py", line 47, in main
    chunk.matchPhotos()
Exception: Kernel failed: an illegal memory access was encountered (700) at line 143

Process finished with exit code 1
In case it helps somehow.

[ttsesm]

Hi Alexey,

any update how I could resolve the issue or apply any workaround?

Thanks.

[Alexey Pasumansky]

Hello ttsesm,

To investigate the problem further, can you please check, if the same operation works or returns the error in the version 1.6.5:
https://s3-eu-west-1.amazonaws.com/download.agisoft.com/metashape-pro_1_6_5_amd64.tar.gz

If the issue is still there, please save the related log.

In case it is possible for you, please also check, if the issue persists in 1.8.1 and 1.6.5 with the older driver version (latest available in 4xx.xx series).

[ttsesm]

Hello ttsesm,

To investigate the problem further, can you please check, if the same operation works or returns the error in the version 1.6.5:
https://s3-eu-west-1.amazonaws.com/download.agisoft.com/metashape-pro_1_6_5_amd64.tar.gz

If the issue is still there, please save the related log.

In case it is possible for you, please also check, if the issue persists in 1.8.1 and 1.6.5 with the older driver version (latest available in 4xx.xx series).

Hi Alexey,

Some update. Downgrading to an older version didn't really help, I was getting the same errors as well. Then reading around that this might be a hardware related issue, I plugged in an older nvidia card that I had available and more specifically an Nvidia GTX 1080 Ti with 12Gb memory and all of a sudden everything works smoothly without errors without anything. My current card is an RTX Nvidia 2080 super with 7Gb of memory. Thus, apparently it is related to the hardware somehow. Now I am not sure whether it is due to the memory difference or just because my 2080 super is broken (though for everything else works fine) or because in the new rtx cards there is something different in the processing of memory or something. Unfortunately, I do not have any other RTX 20xx card to test but it would be interesting to see if the other guys who have similar issues are also using an RTX card.

In any case, thanks for the support and your time.

[flogs]

Hi,

swithing to OpenCL did not help, another error occured.

Filip

[Alexey Pasumansky]

Hello Filip,

Please provide the logs related to CUDA and OpenCL runs and also specify the OS version used and NVIDIA driver version installed.