Forum

Author Topic: multi-GPU cooling query  (Read 7945 times)

Dave Martin

  • Full Member
  • ***
  • Posts: 170
    • View Profile
multi-GPU cooling query
« on: October 12, 2015, 12:26:45 AM »
Have tried searching the forum but can't find anything relating to a multi-GPU / motherboard query.

Currently looking to assemble a system with a pair of GeForce GTX-980 GPUs to go in a dual-socket Xeon system with probably just one processor initially.

Initially looked at a Dell 7810/7910 workstation, but was concerned that position of the two slots suitable for the GPU cards meant that the fan on one card is only a couple of millimetres away from the PCB of the next GPU so there was negligible air circulation space between the two GPUs.  Also a little concerned as to whether the two suitable slots were tied one per processor? - so if only one processor installed initially, can it use both those GPUs?

Alternatively, looking at building a system using the Asus "Z10PE-D8 WS" motherboard (or maybe the D16 version) - they have four suitable Gen-3 16x link slots - but it appears these slots may be in two pairs, two per processor, and the slots for each processor are adjacent so I couldn't have a single processor / two GPU cards with a big air gap between them (like I could on a single-processor board like the Asus P9X79_WS).

Does anyone have experience with long PhotoScan runs with two GTX-980 cards right up alongside each other? any vendors implementation of the GTX-980 more reliable in such situations?

Alternatively, does anyone have a recommendation for a workstation / mother board that's dual socket, with slots that allow air gap between GPUs, which would allow me to run 2 x GTX-980 both with one Xeon now and add a second Xeon later if required?

Thanks / Dave
« Last Edit: October 21, 2015, 04:19:05 PM by Dave Martin »

Wishgranter

  • Hero Member
  • *****
  • Posts: 1202
    • View Profile
    • Museum of Historic Buildings
Re: multi-Xeon motherboard / multi-GPU : slots & cooling query
« Reply #1 on: October 12, 2015, 01:31:31 AM »
look for  water cooling, its mostly 1-slot solution.. preparing my workstation for complete water cooling setup because of this issues...
----------------
www.mhb.sk

Dave Martin

  • Full Member
  • ***
  • Posts: 170
    • View Profile
Re: multi-GPU cooling query
« Reply #2 on: October 21, 2015, 04:25:25 PM »
Thanks for watercooling suggestion but we're looking at aircooled if at all possible.

If we use a Dell Precision workstation, don't believe there is any way to install two water-cooled or hybrid GTX 980-Ti cards; and even if we scratch build a system, since it will be transported in car from time-to-time, keen to avoid any liquid solution.

Does anyone actually run a pair (or more) of GTX 980-Ti yet?   If so, which cards do you use?

If we end up scratch-building our own system, motherboard choice looks like Asus Z10PE-D16 or Supermicro X10DAI - anyone have experience of either of these motherboards?  (plan would be initially just one decent Xeon + 128GB RAM, then add second processor if it was useful in the future).

Thanks/Dave

Wishgranter

  • Hero Member
  • *****
  • Posts: 1202
    • View Profile
    • Museum of Historic Buildings
Re: multi-GPU cooling query
« Reply #3 on: October 21, 2015, 04:47:29 PM »
http://www.techpowerup.com/216890/gigabyte-launches-the-geforce-gtx-980-waterforce-graphics-card.html

for the watercoling..

If you plan it carefully can get in many  case ( mean think on expansion later = approx 300 EUR )

----------------
www.mhb.sk

Dave Martin

  • Full Member
  • ***
  • Posts: 170
    • View Profile
Re: multi-GPU cooling query
« Reply #4 on: October 21, 2015, 06:09:44 PM »
Thanks for the link, but that Gigabyte card is only a 980 not a 980-Ti

EVGA do a GTX 980-Ti hybrid with ready-connected small radiator (and Asus have a water-cooling-ready Poseidon GTX 980-Ti), but as I said, if we drop these cards in a Dell precision workstation, there's no room to install any water cooling - it will have to be solely air-cooled.

Hence why we're still trying to find if anyone is actually using multiple air-cooled GTX 980-Ti cards...

Thanks / Dave
« Last Edit: October 22, 2015, 06:44:37 PM by Dave Martin »

Magnus

  • Jr. Member
  • **
  • Posts: 83
    • View Profile
Re: multi-GPU cooling query
« Reply #5 on: October 22, 2015, 12:06:59 AM »
Hello Dave!

I found this link which might be helpful https://www.pugetsystems.com/labs/articles/Agisoft-PhotoScan-GPU-Acceleration-710/
Not only do they have interesting benches (also for CPU scaling) but they sell systems as you can see at the bottom of the article, one which is a Dual-CPU Quad-GPU (so it should be no problem).

Best, Magnus.

Dave Martin

  • Full Member
  • ***
  • Posts: 170
    • View Profile
Re: multi-GPU cooling query
« Reply #6 on: October 22, 2015, 01:41:47 PM »
Thanks very Magnus.

I'm aware of that page, and in fact Puget's pages helped me to decide on a pair of the GTX 980-Ti cards for the system I'm currently assembling; and if you look in the discussion at the bottom of that page you'll see I've been asking the same questions there as well!

William at Puget has confirmed they have run multiple cards, but hasn't been able to tell me which implementation of GTX 980-Ti cards they have actually had in successful use - but he has highlighted that extra fans would probably be needed.

One of my biggest questions related to running a pair of the GTX 980-Ti cards in adjacent slots in a Dell Precision workstation, where there's no room for watercooling (or extra fans).

The feedback I'm getting from a variety of sites (PhotoScan related and even tried asking in the gamers' forums) is that watercooling is best, but you can run a pair of the better reference cards (although no-one will say which) - provided you can arrange a good airflow in addition to the onboard fans, and this probably means adding extra fan(s) on the side of the PC so they blow down onto the top of the cards - which I can't do on the Dell Precision.

I'm still keen try to set this up in a Dell Precision workstation (a) because of (trans)portability - the Precision 7810/7910 is a very manageable size; and (b) because there are good value deals to be had from time-to-time.

If I have to scratch build, it'll be much bigger & probably cost more.

Hence why I'm trying to find if anyone is actually running multiple GTX 980-Ti (or Titan) cards........

Dave

GPC

  • Jr. Member
  • **
  • Posts: 83
    • View Profile
    • Geopro
Re: multi-GPU cooling query
« Reply #7 on: December 09, 2015, 04:23:28 AM »
Did you get the multi GTX 980 Ti system yet? Looking at this option now myself. Have a dual xeon Dell 7910, the design with extra PCIe on the top of the system looks like a good fit.
When things get weird, the weird turn pro.

flyingburrito

  • Newbie
  • *
  • Posts: 2
    • View Profile
Re: multi-GPU cooling query
« Reply #8 on: December 09, 2015, 05:55:42 PM »
Hi, folks.

For what it's worth, I've been running multiple GPUs and Photoscan Pro on a Dell Precision 7910. 

Inside the case I have a Titan X (12 GB RAM is needed for some large point clouds) and a Quadro K2200 (I typically do not use it for photo-crunching, but there are occasionally, other 3D applications).  Additionally, I hae a Netstor NA250 GPU enclosure (through one of the PCIe Gen3 slots) that runs three GTX 980Tis.  The enclosure has its own power supply and cooling.  Also, each GPU on the motherboard is linked to a separate CPU.  The system has approximately 200GB RAM.

When I run a big job (ultra-high with a few thousand photos), the GPU temps rise up to approximately 70-75C during depth map filtering -- that's not too hot, but if I am concerned, I can easily remove the side panel of each enclosure.  My GPU cards are not significantly overclocked -- I don't see the point in pushing that as GPUs are only used in on aspect of the processing.

In summary, multiple GPUs can work, and the 980Tis are very good.  The main reason I dropped extra money for the Titan X was the GPU RAM for displaying very large point clouds (a few billion points including colors) -- not for photo-processing.

Paco


Dave Martin

  • Full Member
  • ***
  • Posts: 170
    • View Profile
Re: multi-GPU cooling query
« Reply #9 on: December 09, 2015, 11:49:49 PM »
hi CS,

In the end we discounted the Dell Precision 7910 for our specific situation. Our config was a 7910 dual socket but with only one Xeon installed initially. In the 7910, as with most other dual-socket motherboards, if you only have one processor the two GPUs would be adjacent; with a dual processor you could have free slots.

Having spoken to someone who has had heat problems with two adjacent GPUs in a Dell workstation, so we went with a workstation with specific GPU cooling which actually has sufficient cooling for passive GPUs.

So, the new workstation we're using is based on a SuperMicro 7048GR-TR 'super GPU workstation', with one E5-2667v3 Xeon and 128GB RAM (4 x Samsung 32GB). The two GPUs are EVGA GTX 980 Ti SC (EVGA part no. 06G-P4-4992-KR) with no overclocking yet. These GPUs have linear airflow, and although we bought the workstation additional GPU cooling kit (p/n MCP-320-74701-0N-KIT), the workstation's standard cooling is so good we haven't had to install the extra cooling kit yet.

Dave
« Last Edit: December 10, 2015, 11:47:07 AM by Dave Martin »