performance drop from multiple titan X's

[MetaUser555]

Hi,
I've been benchmark testing setups with multi GPUs
I have 3 Titan X's in a PC i'm going from 1 to 2 to 3 gpus with the same scene.
i'm getting diminishing returns as i add GPUs on performance - I was wondering if i have an obvious [to you] bottleneck somewhere in my system that is causing this or is it just what happens?
All drivers are up to date and running windows 7 enterprise
SLI is enabled but i believe it's bypasses this as we're using open CL

edit:
The scene is identical in each case
It's the dense cloud process from a saved pre-processed align photos scene
the machine is restarted before each test

1 GPU:
OpenCL  active CPU cores 15/16
Device 1 performance: 202.403 million samples/sec (CPU)
Device 2 performance: 1142.96 million samples/sec (GeForce GTX TITAN X)

2 GPUs
OpenCL  active CPU cores 14/16
Device 1 performance: 174.528 million samples/sec (CPU)
Device 2 performance: 718.264 million samples/sec (GeForce GTX TITAN X)
Device 3 performance: 754.264 million samples/sec (GeForce GTX TITAN X)

3 GPUs
OpenCL  active CPU cores 13/16
Device 1 performance: 197.528 million samples/sec (CPU)
Device 2 performance: 469.848 million samples/sec (GeForce GTX TITAN X)
Device 3 performance: 491.990 million samples/sec (GeForce GTX TITAN X)
Device 3 performance: 493.464 million samples/sec (GeForce GTX TITAN X)

Processor: Intel(R) Core(TM) i7-5960X CPU @ 3.00GHz (16 CPUs), ~3.0GHz
Memory: 32768MB RAM

any helpful advice or comments much appreciated!
thanks

[dtmcnamara]

Disable SLI and try again.

[Alexey Pasumansky]

Hello bolegna,

I can suggest to perform tests with all CPU cores disabled in OpenCL preferences window.

[MetaUser555]

Thanks dtm - I will try that, do you think it's worth taking the physical SLI cable out too?

Thanks Alexey - I'm trying your suggestions right now.

[MetaUser555]

Hi,
Here are some results of testing with three GPUs:

13/16 CPU + SLI
finished depth reconstruction in 2002.72 seconds
Device 1 performance: 172.682 million samples/sec (CPU)
Device 2 performance: 395.05 million samples/sec (GeForce GTX TITAN X)
Device 3 performance: 388.065 million samples/sec (GeForce GTX TITAN X)
Device 4 performance: 421.179 million samples/sec (GeForce GTX TITAN X)
Total performance: 1376.97 million samples/sec

zero CPU + SLI
finished depth reconstruction in 1900.26 seconds
Device 1 performance: 443.138 million samples/sec (GeForce GTX TITAN X)
Device 2 performance: 478.98 million samples/sec (GeForce GTX TITAN X)
Device 3 performance: 479.976 million samples/sec (GeForce GTX TITAN X)
Total performance: 1402.09 million samples/sec

13/16 CPU + SLI disabled
finished depth reconstruction in 1717.77 seconds
Device 1 performance: 147.353 million samples/sec (CPU)
Device 2 performance: 633.477 million samples/sec (GeForce GTX TITAN X)
Device 3 performance: 629.034 million samples/sec (GeForce GTX TITAN X)
Device 4 performance: 666.721 million samples/sec (GeForce GTX TITAN X)
Total performance: 2076.58 million samples/sec

zero CPU + SLI disabled
finished depth reconstruction in 1330.81 seconds
Device 1 performance: 899.79 million samples/sec (GeForce GTX TITAN X)
Device 2 performance: 914.139 million samples/sec (GeForce GTX TITAN X)
Device 3 performance: 917.935 million samples/sec (GeForce GTX TITAN X)
Total performance: 2731.86 million samples/sec

i'll repeat the tests with 2 GPUs and post results
thanks for your help!

[MetaUser555]

15/16 CPU
finished depth reconstruction in 2671.97 seconds
Device 1 performance: 202.403 million samples/sec (CPU)
Device 2 performance: 1142.96 million samples/sec (GeForce GTX TITAN X)
Total performance: 1345.36 million samples/sec

zero CPU
finished depth reconstruction in 2814.99 seconds
Device 1 performance: 1246.93 million samples/sec (GeForce GTX TITAN X)
Total performance: 1246.93 million samples/sec

zero CPU SLI
finished depth reconstruction in 1899 seconds
Device 1 performance: 881.139 million samples/sec (GeForce GTX TITAN X)
Device 2 performance: 853.223 million samples/sec (GeForce GTX TITAN X)
Total performance: 1734.36 million samples/sec

zero CPU no SLI
finished depth reconstruction in 1666.46 seconds
Device 1 performance: 1127.63 million samples/sec (GeForce GTX TITAN X)
Device 2 performance: 1137.76 million samples/sec (GeForce GTX TITAN X)
Total performance: 2265.39 million samples/sec

14/16 CPU SLI
finished depth reconstruction in 2043.22 seconds
Device 1 performance: 174.528 million samples/sec (CPU)
Device 2 performance: 718.264 million samples/sec (GeForce GTX TITAN X)
Device 3 performance: 754.264 million samples/sec (GeForce GTX TITAN X)
Total performance: 1647.06 million samples/sec

14/16 CPU no SLI
finished depth reconstruction in 1928.05 seconds
Device 1 performance: 172.549 million samples/sec (CPU)
Device 2 performance: 909.391 million samples/sec (GeForce GTX TITAN X)
Device 3 performance: 921.898 million samples/sec (GeForce GTX TITAN X)
Total performance: 2003.84 million samples/sec

here are some more results
so the original question was, is this an expected performance drop from using multiple GPU?
I wouldn't expect to double the performance by adding a gpu - 1+1 is rarely 2 but it seems we're still getting a drop in performance when we add a gpu, roughly:
1 x 1246 million samples/sec
2 x 1150 million samples/sec
3 x 914 million samples/sec
We would obviously not use SLI or CPUs for multiple GPU with our titan X's thanks for the guidance.
We're trying to figure out an optimal setup for our needs.
thanks again for your help most useful!

[igor73]

I reported benchmark results with 2 Titan X a few weeks ago.  Can´t find it right now but search and you will find it.  I think i had got something like 1500milion  sampls per second per card or might have even been 1700. I use the the EIVGA super clocked cards so they are a bit over clocked from scratch.   The 2nd card is a tad slower becuse of temp on my system.  The cards sit very near togheteher and one of the cards does not reach the same turbo boast clock speed as the other.  Around -100mhz.  Not a big deal and i will not do anything about it as it would not be worth the hassle to change MB and case for me.
 
Is the scene you process small?   I noticed i got better performance on larger scenes as the cards need a bit of time before they reach full performance and full turbo boost. The more cards you enable the faster the scene will process and it might be so quick you dont see the full performance of your system.    Also use the Nvidia Gforce experience - LED Visualizer to monitor your cards clock speed and temperature.  It may be that you need more ventilation to keep them cold enough if they are stacked near each other.   Run a large scene with  at least 100 preferably 200 or more high res images and report back pls.

[MetaUser555]

thanks for the reply - I've done the testing i needed to do with the cards now.
FYI the scene was 100 masked 5184x3456 Tiffs run at high on dense cloud process.
Our cards are reference cards with no overclocking.
I'd be interested in seeing all your results!
cheers