Good day,
I have a similar problem. Yesterday I installed a 1080Ti with the previous 970. Agi would not process using both cards:
2018-06-08 08:44:42 Agisoft PhotoScan Professional Version: 1.4.2 build 6205 (64 bit)
2018-06-08 08:44:42 Platform: Windows
2018-06-08 08:44:42 CPU: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz (desktop)
2018-06-08 08:44:42 CPU family: 6 model: 60 signature: 306C3h
2018-06-08 08:44:42 RAM: 31.9 GB
2018-06-08 08:44:42 OpenGL Vendor: NVIDIA Corporation
2018-06-08 08:44:42 OpenGL Renderer: GeForce GTX 1080 Ti/PCIe/SSE2
2018-06-08 08:44:42 OpenGL Version: 4.6.0 NVIDIA 398.11
2018-06-08 08:44:42 Maximum Texture Size: 32768
2018-06-08 08:44:42 Quad Buffered Stereo: not enabled
2018-06-08 08:44:42 ARB_vertex_buffer_object: supported
2018-06-08 08:44:42 ARB_texture_non_power_of_two: supported
2018-06-08 08:44:42 Using simple console. Rich console can be enabled in Preferences dialog.
2018-06-08 08:44:54 Loading photos...
2018-06-08 08:44:54 Finished processing in 0.016 sec (exit code 1)
2018-06-08 08:45:33 AlignPhotos: accuracy = Medium, preselection = reference, keypoint limit = 40000, tiepoint limit = 4000, apply masks = 0, filter tie points = 0, adaptive fitting = 0
2018-06-08 08:45:33 Matching photos...
2018-06-08 08:45:33 Detecting points...
2018-06-08 08:45:33 Using device: GeForce GTX 1080 Ti, 28 compute units, 11264 MB global memory, compute capability 6.1
2018-06-08 08:45:33 driver version: 9020, runtime version: 5050
2018-06-08 08:45:33 max work group size 1024
2018-06-08 08:45:33 max work item sizes [1024, 1024, 64]
2018-06-08 08:45:33 Using device: GeForce GTX 970, 13 compute units, 4096 MB global memory, compute capability 5.2
2018-06-08 08:45:33 driver version: 9020, runtime version: 5050
2018-06-08 08:45:33 max work group size 1024
2018-06-08 08:45:33 max work item sizes [1024, 1024, 64]
2018-06-08 08:45:33 [GPU] photo 1: 19627 points
2018-06-08 08:45:37 Warning: cudaStreamDestroy failed: out of memory (2)
2018-06-08 08:45:37 Finished processing in 4.33 sec (exit code 0)
2018-06-08 08:45:37 Error: CUDA_ERROR_OUT_OF_MEMORY (2) at line 123
It works when selecting each card individually, they wont work together.
Thanks in advance.