Photoscan crash when using GPU

[cosmojiang]

I installed a Photoscan trial version (1.3.4) on my university cluster for testing its capabilities. Two issues were identified.

First, the Photoscan cannot display its GUI when I use PuTTY with X11. The program frame appeared, but no menu or button was shown, which means that I cannot use the GUI to load images and start a reconstruction.

Second, I can use the command line with custom python scripts to let Photoscan work. The Photoscan worked well in CPU alone mode, but it crashed every time in the GPU mode. As there is only one GPU card, I configured as 'Photoscan.app.gpu_mask = 1'. Below is the error message I got:

Warning: cudaStreamDestroy failed: all CUDA-capable devices are busy or unavailable (46)
Using device: Tesla K40m, 15 compute units, 11439 MB global memory, compute capability 3.5
  max work group size 1024
  max work item sizes [1024, 1024, 64]
Traceback (most recent call last):
  File "chainprocess.py", line 34, in <module>
    chunk.matchPhotos(accuracy=PhotoScan.HighAccuracy, preselection=PhotoScan.GenericPreselection)
RuntimeError: CUDA_ERROR_INVALID_DEVICE (101) at line 115

It would be highly appreciated if anyone can help.

[Alexey Pasumansky]

Hello cosmojiang,

Looks like there are some issues in proper utilization of the remote computer GPU both for visualization and processing.

Can you provide full Console pane output?

[cosmojiang]

Alexey,

Thanks for your quick response. The trial version has been expired on the cluster, so I am not able to retrieve the error of using GPU for processing. However, attached please find the error when using the GUI.

[PolarNick]

You can try to run glxgears and glxinfo to check that OpenGL correctly works over your connection configuration.

[cosmojiang]

You can try to run glxgears and glxinfo to check that OpenGL correctly works over your connection configuration.

Hi PolarNick,

Thanks for your suggestions. I tested both and they worked well through my connection to the cluster. Today, as someone else exclusively used the GPU, so I connected the cluster without GPU. The Photoscan GUI worked, but it showed one warning and one error (please see attached snapshot). Any idea on this?

Thanks in advance,
Yu

[PolarNick]

Can you please post logs like Alexey suggested (they can be accessed in GUI)? Logs contain information about OpenGL version available. And it looks like it is too low on your configuration, there is similar problem - https://github.com/Ultimaker/Cura/issues/784

Also I am not sure that Tesla supports any OpenGL.

[cosmojiang]

Can you please post logs like Alexey suggested (they can be accessed in GUI)? Logs contain information about OpenGL version available. And it looks like it is too low on your configuration, there is similar problem - https://github.com/Ultimaker/Cura/issues/784

Also I am not sure that Tesla supports any OpenGL.

Please see the latest log retrieved from the Photoscan console:

2017-11-07 09:39:16 OpenGL Vendor: Mesa Project
2017-11-07 09:39:16 OpenGL Renderer: Software Rasterizer
2017-11-07 09:39:16 OpenGL Version: 2.1 Mesa 10.1.2
2017-11-07 09:39:16 Maximum Texture Size: 16384
2017-11-07 09:39:16 Quad Buffered Stereo: not enabled
2017-11-07 09:39:16 ARB_vertex_buffer_object: supported
2017-11-07 09:39:16 ARB_texture_non_power_of_two: supported
2017-11-07 09:39:16 Using simple console. Rich console can be enabled in Preferences dialog.
2017-11-07 09:40:03 Can't load OpenCL library2017-11-07 09:39:15 Agisoft PhotoScan Professional Version: 1.3.4 build 5067 (64 bit)
2017-11-07 09:39:15 Platform: Linux

Also, I found that if I run glxgears first, the Photoscan will not recognize the GPU but will show the GUI. Otherwise, it recognizes GPU but cannot show the GUI properly. Even for glxgears, after stopping it, an error occurred: "XIO: fatal IO error 11 (Resource temporarily unavailable) on X server". Would this indicate anything?

I also attached the glxinfo screen snapshot, and it seems that the server glx version (1.2) is different from my local machine (1.4). Would this be a possible reason?

[PolarNick]

Even for glxgears, after stopping it, an error occurred: "XIO: fatal IO error 11 (Resource temporarily unavailable) on X server". Would this indicate anything?

This is looks similar to this (fix?). But this is not reproduces with PhotoScan, so I think that this doesn't indicate anything

I also attached the glxinfo screen snapshot, and it seems that the server glx version (1.2) is different from my local machine (1.4). Would this be a possible reason?

Yes, this seems to be the reason. Because at least GLX 1.3 is required. I don't know can GLX 1.3 be supported somehow with mesa. I don't know how to fix this.

[PolarNick]

About computations without GUI (via python script or network processing):

Warning: cudaStreamDestroy failed: all CUDA-capable devices are busy or unavailable (46)

Welcome to the club! This problem pops up sometimes and I encountered it too. I can only suggest you to try these things:
 - Update/downgrade drivers
 - Test some simple cuda application (something like GPU-Z or deviceQuery)