Black Screen / CUDA error 715 on line 112

[tomburtonwood]

Howdy, first post, medium long term user.

First up - great software - love it! - really useful.

I run Photoscan Standard on my MacBook and a Dell laptop without any issues, apart from super slow processing.

I have a new machine though - a PC desktop box with a decent graphics card, but i'm running into what i think is a compatibility or driver error.

specs
Win 10 for edu
Intel Xeon CPU 1.8ghz (2 processors)
32GB RAM
64 bit
NVIDIA Tesla C2075
Tesla driver ver 377.35

at first when i ran Photoscan i got the "black screen error" - i read this post on the forum, http://www.agisoft.com/forum/index.php?topic=6994.msg33739#msg33739 ,  and followed the steps suggested.

this is the read from the console:

2017-06-06 20:52:18 Agisoft PhotoScan Standard Version: 1.3.2 build 4195 (64 bit)
2017-06-06 20:52:18 Platform: Windows
2017-06-06 20:52:19 OpenGL Vendor: Microsoft Corporation
2017-06-06 20:52:19 OpenGL Renderer: GDI Generic
2017-06-06 20:52:19 OpenGL Version: 1.1.0
2017-06-06 20:52:19 Maximum Texture Size: 1024
2017-06-06 20:52:19 Quad Buffered Stereo: not enabled
2017-06-06 20:52:19 ARB_vertex_buffer_object: not supported
2017-06-06 20:52:19 ARB_texture_non_power_of_two: not supported
2017-06-06 20:52:51 OpenGL Vendor: Microsoft Corporation
2017-06-06 20:52:51 OpenGL Renderer: GDI Generic
2017-06-06 20:52:51 OpenGL Version: 1.1.0
2017-06-06 20:52:51 Maximum Texture Size: 1024
2017-06-06 20:52:51 Quad Buffered Stereo: not enabled
2017-06-06 20:52:51 ARB_vertex_buffer_object: not supported
2017-06-06 20:52:51 ARB_texture_non_power_of_two: not supported

i am not running WRD or any other kind of remote login - to my knowledge. so - the questions i suppose are -
why is Photoscan ignoring the installed GPU?
 is the Tesla compatible with Photoscan in the first place?
what should i do next?

ALSO - when i try to run a scan with the GPU selected in preferences i get - an error - "CUDA error 715 on line 112" - when i run a scan without the GPU selected in the preferences then it runs fine - but slow.

any help would be greatly appreciated i have about 50 scans to run thru and i need to get this moving.

[Alexey Pasumansky]

Hello tomburtonwood,

Unless I am mistaken, Tesla card can be used only for the computations and cannot act as OpenGL rendering device.

As for the CUDA error, can you specify the driver version and provide the full processing log from the start of the corresponding operation up to the error message (for Align Photos and Build Dense Cloud stages).

[tomburtonwood]

Hi Alexey,

Thank you for your fast response. I did not know the Tesla does not run OpenGL - that explains a lot!

See below console - i closed Photoscan, reopened the pre-aligned file (from other machine), and reran Dense Point Cloud at High. Error msg is now "Unknown at line 195" -

2017-06-07 09:23:50 Agisoft PhotoScan Standard Version: 1.3.2 build 4195 (64 bit)
2017-06-07 09:23:50 Platform: Windows
2017-06-07 09:23:51 OpenGL Vendor: Microsoft Corporation
2017-06-07 09:23:51 OpenGL Renderer: GDI Generic
2017-06-07 09:23:51 OpenGL Version: 1.1.0
2017-06-07 09:23:51 Maximum Texture Size: 1024
2017-06-07 09:23:51 Quad Buffered Stereo: not enabled
2017-06-07 09:23:51 ARB_vertex_buffer_object: not supported
2017-06-07 09:23:51 ARB_texture_non_power_of_two: not supported
2017-06-07 09:24:10 OpenGL Vendor: Microsoft Corporation
2017-06-07 09:24:10 OpenGL Renderer: GDI Generic
2017-06-07 09:24:10 OpenGL Version: 1.1.0
2017-06-07 09:24:10 Maximum Texture Size: 1024
2017-06-07 09:24:10 Quad Buffered Stereo: not enabled
2017-06-07 09:24:10 ARB_vertex_buffer_object: not supported
2017-06-07 09:24:10 ARB_texture_non_power_of_two: not supported
2017-06-07 09:24:19 LoadProject
2017-06-07 09:24:19 Loading project...
2017-06-07 09:24:20 loaded project in 0.375 sec
2017-06-07 09:24:20 Finished processing in 0.381 sec (exit code 1)
2017-06-07 09:24:30 Checking for missing images...
2017-06-07 09:24:30 checking for missing images... done in 0.043 sec
2017-06-07 09:24:30 Finished processing in 0.044 sec (exit code 1)
2017-06-07 09:24:30 BuildDenseCloud: quality = High, depth filtering = Aggressive
2017-06-07 09:24:30 Initializing...
2017-06-07 09:24:30 Using device: Tesla C2075, 14 compute units, 5316 MB global memory, Device Compute Capability 2.0
2017-06-07 09:24:30 max work group size 1024
2017-06-07 09:24:30 max work item sizes [1024, 1024, 64]
2017-06-07 09:24:30 Using CUDA device 'Tesla C2075' in concurrent. (2 times)
2017-06-07 09:24:30 sorting point cloud... done in 0.828 sec
2017-06-07 09:24:31 processing matches... done in 0.286 sec
2017-06-07 09:24:31 initializing...
2017-06-07 09:24:37 selected 507 cameras from 535 in 5.333 sec
2017-06-07 09:24:37 Loading photos...
2017-06-07 09:25:27 loaded photos in 50.038 seconds
2017-06-07 09:25:27 Reconstructing depth...
2017-06-07 09:25:27 [GPU] estimating 1403x2862x640 disparity using 1403x1431x8u tiles
2017-06-07 09:25:27 [GPU] estimating 1418x4291x352 disparity using 1418x1073x8u tiles
2017-06-07 09:25:27 Error: 2017-06-07 09:25:27 GPU processing failed, switching to CPU mode
2017-06-07 09:25:27 GPU processing failed, switching to CPU mode
2017-06-07 09:25:27 [CPU] estimating 1403x2862x640 disparity using 1403x954x8u tiles
2017-06-07 09:25:27 [CPU] estimating 1418x4291x352 disparity using 1418x1073x8u tiles
Kernel failed: unknown error at line 98Error: Kernel failed: unknown error at line 98
2017-06-07 09:25:27
2017-06-07 09:25:28
2017-06-07 09:25:28 Depth reconstruction devices performance:
2017-06-07 09:25:28 - 0%   done by Tesla C2075
2017-06-07 09:25:28 Total time: 0.87 seconds
2017-06-07 09:25:28
2017-06-07 09:25:28 Warning: all CUDA-capable devices are busy or unavailable
2017-06-07 09:25:29 Warning: all CUDA-capable devices are busy or unavailable
2017-06-07 09:25:29 Finished processing in 58.67 sec (exit code 0)
2017-06-07 09:25:29 Error: unknown error at line 195

i have a Quadro 4000 card. i'm going to install that on this machine and see what happens.

Thank you!

[Alexey Pasumansky]

Hello tomburtonwood,

Can you also check that Tesla card is set to compute mode in NVIDIA control panel? Also you can try to switch it to TCC mode using NVIDIA SMI (here's a brief instruction: http://www.agisoft.com/forum/index.php?topic=6992.msg33756#msg33756).

And please specify NVIDIA driver version installed.

[tomburtonwood]

Sorry NVIDIA Driver is 377.35.

the Quadro seems to have done the trick.

this is from the console - running Point Cloud now on High -

2017-06-07 10:01:34 Checking for missing images...
2017-06-07 10:01:34 checking for missing images... done in 0.036 sec
2017-06-07 10:01:34 Finished processing in 0.036 sec (exit code 1)
2017-06-07 10:01:34 BuildDenseCloud: quality = High, depth filtering = Aggressive
2017-06-07 10:01:34 Initializing...
2017-06-07 10:01:34 Using device: Quadro 4000, 8 compute units, 2048 MB global memory, Device Compute Capability 2.0
2017-06-07 10:01:34 max work group size 1024
2017-06-07 10:01:34 max work item sizes [1024, 1024, 64]
2017-06-07 10:01:34 Using device: Tesla C2075, 14 compute units, 5375 MB global memory, Device Compute Capability 2.0
2017-06-07 10:01:34 max work group size 1024
2017-06-07 10:01:34 max work item sizes [1024, 1024, 64]
2017-06-07 10:01:34 Using CUDA device 'Tesla C2075' in concurrent. (2 times)
2017-06-07 10:01:34 sorting point cloud... done in 0.816 sec
2017-06-07 10:01:35 processing matches... done in 0.292 sec
2017-06-07 10:01:35 initializing...
2017-06-07 10:01:40 selected 507 cameras from 535 in 5.559 sec
2017-06-07 10:01:41 Loading photos...
2017-06-07 10:02:31 loaded photos in 50.507 seconds
2017-06-07 10:02:31 Reconstructing depth...
2017-06-07 10:02:31 [GPU] estimating 1657x2184x704 disparity using 829x1092x8u tiles
2017-06-07 10:02:31 [GPU] estimating 1418x4291x352 disparity using 1418x1431x8u tiles
2017-06-07 10:02:31 [GPU] estimating 1252x2009x416 disparity using 1252x1005x8u tiles

so it seems all is good -

thank you

[tomburtonwood]

also it is running in compute mode.

est is 4 hours to process the scan. which is a massive improvement on prior attempts.

[Alexey Pasumansky]

Hello tomburtonwood,

It seems that the problem with "all CUDA-capable devices are busy or unavailable" happens from time to time on different applications that are exploiting CUDA on NVIDIA cards. We'll try to dig deeper, but the issue may be related to the driver behavior.

But if the issue will re-appear, please let us know.

[tomburtonwood]

will do.

does it make sense to limited the quadro only to graphics tasks - or to run it in tandem with the Tesla for computing?

right now it is running in tandem, - with both cards computing the scan.

[Alexey Pasumansky]

Hello tomburtonwood,

You can run any sample dataset on both cards with CPU disabled and check the percentage of task completed by every card. If the there is no considerable difference, it's better to use both cards to get higher performance and faster processing.