Forum

Author Topic: "all CUDA-capable devices are busy or unavailable (46) at line 33" when aligning  (Read 1105 times)

marcel.d

  • Newbie
  • *
  • Posts: 6
    • View Profile
Hello,

I have a very simple project with 20 photos which aligns easily on CPU, but fails to align when using the GPU. Since this fails, I cannot do any of the other steps. Any ideas?

Error message: "Error: cudaMemGetInfo(&free_mem_size, &total_mem_size): all CUDA-capable devices are busy or unavailable (46) at line 33"

Specs:

Agisoft Metashape Professional Version: 1.5.5 build 9097 (64 bit)
Platform: Windows
CPU: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (server)
CPU family: 6 model: 79 signature: 406F1h
RAM: 511.9 GB
OpenGL Vendor: NVIDIA Corporation
OpenGL Renderer: GeForce GTX TITAN X/PCIe/SSE2
OpenGL Version: 4.6.0 NVIDIA 461.72

Thanks in advance!

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 12742
    • View Profile
Hello Marcel,

I can suggest to make a clean driver install and check if it helps.

Also please check, if you are observing any similar issues in versions 1.6.6 and 1.7.1.
Best regards,
Alexey Pasumansky,
Agisoft LLC

marcel.d

  • Newbie
  • *
  • Posts: 6
    • View Profile
Hello Alexey,

Thank you for these suggestions and for replying in the other thread (https://www.agisoft.com/forum/index.php?topic=10394.0). Sorry for crossposting, but with several driver versions we did have the error from the other thread.

I have avoided clean installs so far, because several people are using the machine and I was not sure if it would break anything for them. If updating windows (currently running 1803, available: 2004) will not help, we will try the clean install of the Nvidia drivers (461.72 or 397.93 if that fails).

Regarding Metashape versions 1.6.6 and 1.7.1: we were unsure if the license would be affected by an update, but judging by the info here https://www.agisoft.com/downloads/installer/and here https://agisoft.freshdesk.com/support/solutions/articles/31000135256-how-to-update-metashape-pro-to-the-new-version it should be fine. We will try it too.

Kind regards,
Marcel

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 12742
    • View Profile
Hello Marcel,

If you plan to update Windows version, I recommend to deactivate Metashape license first and then re-activate it once all the changes are made and one of the latest Metashape versions is installed.
Depth maps generation algorithms do differ considerably in 1.6 and 1.7 versions, so the behavior could be different.
Best regards,
Alexey Pasumansky,
Agisoft LLC

marcel.d

  • Newbie
  • *
  • Posts: 6
    • View Profile
Hello Alexey,

I did the following:

1) Updated Windows 10 latest version:
Edition   Windows 10 Pro
Version   20H2
Installed on ‎10.‎03.‎2021
OS Build 19042.867
Windows Feature Experience Pack 120.2212.551.0

The issue persisted.

2) Installed Nvidia Driver 461.72 (clean install!)

The issue persisted

3) Installed Metashape 1.6.6 (1.5.5 was deleted in the process)

Ran the procedure twice and got 2 different results (possibly with different data sets):

First: "Error: CUDA_ERROR_UNKNOWN (999) at line 146" . Console output in output1.txt

Second: "Error: cudaMemGetInfo(&free_mem_size, &total_mem_size): all CUDA-capable devices are busy or unavailable (46) at line 40" . Console output in output2.txt

4) Installed Metashape 1.7.2 (I renamed the 1.6.6 folder to keep both versions)

Ran the procedure twice and got the same 2 results, in different order:

First: "Error: cudaMemGetInfo(&free_mem_size, &total_mem_size): all CUDA-capable devices are busy or unavailable (46) at line 40" . Console output in output3.txt

Second: "Error: CUDA_ERROR_UNKNOWN (999) at line 146" . Console output in output4.txt

The output of nvidia-smi when the last project was running:

Code: [Select]
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 461.72       Driver Version: 461.72       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX TIT... WDDM  | 00000000:03:00.0  On |                  N/A |
| 22%   47C    P8    27W / 250W |    331MiB / 12288MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1224    C+G   Insufficient Permissions        N/A      |
|    0   N/A  N/A      3524    C+G   ...nputApp\TextInputHost.exe    N/A      |
|    0   N/A  N/A      8920    C+G   ...5n1h2txyewy\SearchApp.exe    N/A      |
|    0   N/A  N/A      9612    C+G   C:\Windows\explorer.exe         N/A      |
|    0   N/A  N/A     10652    C+G   ...lPanel\SystemSettings.exe    N/A      |
|    0   N/A  N/A     11056    C+G   ...artMenuExperienceHost.exe    N/A      |
|    0   N/A  N/A     12632    C+G   ...y\ShellExperienceHost.exe    N/A      |
|    0   N/A  N/A     14696    C+G   ...b3d8bbwe\WinStore.App.exe    N/A      |
|    0   N/A  N/A     16064    C+G   ...e Pro 1.7.2\metashape.exe    N/A      |
+-----------------------------------------------------------------------------+

I don't have any more ideas what to try. Do you habe a guess what we could do?

Thanks
Marcel

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 12742
    • View Profile
Hello Marcel,

Which output do you get after creating main/gpu_enable_cuda tweak and setting its value to False (application should be re-started)?
Best regards,
Alexey Pasumansky,
Agisoft LLC

marcel.d

  • Newbie
  • *
  • Posts: 6
    • View Profile
Hello Alexey,

Thanks for the suggestion. I tried it and the results are below.

However what I tried first was restarting the computer multiple times and trying to run Metashape every time. It always crashed either with the error 999 or 46. But one time it did not crash and successfully aligned the photos! So I opened the proper project with over 1000 photos without closing Metashape in between. I ran the whole processing without issues. Then I shortly restarted Metashape (the PC ran continuously) and again it did not run even a small alignment process. Crashed with the same errors as always.

What struck me is that it did not throw the error in the beginning and then never again while the software was running.

Then we had a fresh Windows install and used the drivers 461.72 (I know that in between 461.92 are available, but did not try installing those). With this setup - still crashing.

Last thing I tried was disabling CUDA as described here: https://www.agisoft.com/forum/index.php?topic=10100.msg46133#msg46133
It worked and the output is attached. However we would prefer using CUDA, as other users are unhappy with the processing times when using OpenCL (https://www.agisoft.com/forum/index.php?topic=10100.msg46142#msg46142). Any suggestions?

Thanks,
Marcel

Alexey Pasumansky

  • Agisoft Technical Support
  • Hero Member
  • *****
  • Posts: 12742
    • View Profile
Hello Marcel,

I don't think there should be considerable performance difference between OpenCL and CUDA implementation of the algorithms in 1.7 version.

You are referring to the version 1.5.0 pre-release, almost 2.5 years have passed since then.
Best regards,
Alexey Pasumansky,
Agisoft LLC