GPU recommendations

[Woodles]

I've recently built a new PC almost exclusively for crunching but to date, the only GPU is the integrated graphics processor in the CPU.

I'm running Ubuntu and from what I read, NVIDIA cards are more reliable/easier to set up than ATI cards (unless anyone else knows different?) Also apparently Linux doesn't do SLI or Crossfire so that's not a consideration. (Does it matter for Boinc?)

The motherboard has three PCIe slots and while I'm not planning on filling them all just yet, I may do so in the future so any card would need to be able to be partnered with others. For the same reason, very wide cards should be avoided, one PCIe slot will take a triple wide card, one a double wide card and the other only a single width card. Obviously, I'd like some space left for other cards!

The PCIe slots are 1 * 16, 1 * 8 and 1 * 4. As most of the Boinc crunching is done on the cards and not much data transferred between the cards or a card and the CPU, I assume the bus widths aren't that critical and multiple cards will work?

Also am I right in thinking that the main 'features' so far as Boinc crunching is concerned is cores/shaders followed by clock speed? And memory size/bandwidth aren't all that important?

It will be sharing a case with an i7-4970K CPU, 16G of RAM, a Gigabyte GA-Z97X motherboard, a nearly empty 500G hard disc and a 750W PSU should any of that be important. Reasonably well ventilated but I can add another three, 120mm fans if need be (don't really want to start messing with water cooling just at the moment) I could also shuffle PSUs around and swap it for a 1200W if necessary but the other factors are pretty fixed (no more usable bits left in the 'recycle' bin )

I'm looking at the Geforce 970/780/780Ti/980/980Ti cards at the moment but I'm open to other suggestions/criticisms.

Mark

[UBT - Timbo]

I've recently built a new PC almost exclusively for crunching but to date, the only GPU is the integrated graphics processor in the CPU.

Hi Mark,

I have some "user experience" to chuck your way !!

I'm running Ubuntu and from what I read, NVIDIA cards are more reliable/easier to set up than ATI cards (unless anyone else knows different?) Also apparently Linux doesn't do SLI or Crossfire so that's not a consideration. (Does it matter for Boinc?)

I think that these days, both AMD and NVidia are both well supported on Linux...though, for choice I went with NVidia, as I found on my Windows PC's that NVidia's "taskbar" app was a little more useable.

SLI/Crossfire are only relevant if you want to house two video cards in the one PC...this is great for gamers as they can run at higher speed and resolution, using 2 VGA cards to run on one monitor. For BOINC GPU tasks, I was using 2x GTX580 cards in SLI mode, which meant I could run 2 Collatz and/or 2 MilkyWay WU's per card...so 4 GPU tasks concurrently. But the cards got very hot (80+ deg C) as did the case, so for safety and bank account reasons, I dropped down to 1x GTX580 and that works fine now.

The motherboard has three PCIe slots and while I'm not planning on filling them all just yet, I may do so in the future so any card would need to be able to be partnered with others. For the same reason, very wide cards should be avoided, one PCIe slot will take a triple wide card, one a double wide card and the other only a single width card. Obviously, I'd like some space left for other cards!

The PCIe slots are 1 * 16, 1 * 8 and 1 * 4. As most of the Boinc crunching is done on the cards and not much data transferred between the cards or a card and the CPU, I assume the bus widths aren't that critical and multiple cards will work?

Most cards these days will need the 1 * 16 slot, as this ensures max data transfer from the CPU on the mobo to the GPU on the card and back again. Some cards need 2 * 16 slots, so be careful in case your chosen card needs this and the brand/version of mobo you have, doesn't support it.

Just like ISA, EISA and now with PCI slots, they will all "work together" - but be aware that the Northbridge and Southbridge heatsinks might need extra cooling, if you add in too many cards...

Also am I right in thinking that the main 'features' so far as Boinc crunching is concerned is cores/shaders followed by clock speed? And memory size/bandwidth aren't all that important?

Cores/shaders are indeed most important, though there comes a cost issue on the better cards, simply down to the extra cost of making such a card. And again, memory is fairly important, as on some higher spec cards, you can run multiple GPU WU's at once...So look for something with perhaps 2Gb on board as a minimum.

It will be sharing a case with an i7-4970K CPU, 16G of RAM, a Gigabyte GA-Z97X motherboard, a nearly empty 500G hard disc and a 750W PSU should any of that be important. Reasonably well ventilated but I can add another three, 120mm fans if need be (don't really want to start messing with water cooling just at the moment) I could also shuffle PSUs around and swap it for a 1200W if necessary but the other factors are pretty fixed (no more usable bits left in the 'recycle' bin )

I'm looking at the Geforce 970/780/780Ti/980/980Ti cards at the moment but I'm open to other suggestions/criticisms.

Mark

Given the option, I'd upgrade the PSU from the 750W type....simply due to the amount of juice the higher spec cards in your wish list will consume. Most of the higher spec cards will use perhaps 2x 6 or even 2x 8 pin PCIe connectors....and that will strain some PSU's. If the 750W PSU is a "good one", then try it and see. OTOH, if the 1200W is "spare" then I'd use it, simply because it'll be running well within it's limits, whereas the 750W might struggle, esp if you add other cards in later on.

You will find plenty of GTX770's and 780's on fleabay - simply because a lot of gamers are upgrading to the newer higher spec models. But they are having to pay a price premium for their upgrades, and hence they need to move their older cards onto new buyers. And for BOINC use, the difference between a 7xx and 9xx isn't that great, as both are pretty good.

Good ventilation will be essential, as although the VGA cards have their own fans, with the CPU and other chipsets generating heat, plus what comes from the GPU, the PC internals will be cooking pretty quickly, esp if it's a compact case with the sides attached. So, a little bit of planning on the airflow would be helpful, esp in terms of getting cool air "into" the box. You might also want to think about air filters too, (to prevent cat hairs and dust building up inside....you've no doubt seen inside a large ASIC and see how much dust can be "drawn in", which can affect overall performance).

GTX7xx spec:
https://en.wikipedia.org/wiki/GeForce_7 ... .29_series

GTX9xx spec:
https://en.wikipedia.org/wiki/GeForce_9 ... .29_series

Hope this helps

regards
Tim

[Woodles]

Hi Mark,

I have some "user experience" to chuck your way !!

I think that these days, both AMD and NVidia are both well supported on Linux...though, for choice I went with NVidia, as I found on my Windows PC's that NVidia's "taskbar" app was a little more useable.

Hi Tim,

Thanks for the feedback, I've ordered one card (supposed to be delivered today!) but I'll keep your comments in mind for any future ones.

I've ended up with a NVIDIA Geforce GTX 970 as my first card. I saw a few posts on Ubuntu forums mentioning problems with the AMD drivers and none mentioning the NVIDIA drivers so, since I have exactly two weeks experience with Ubuntu and even less with GPUs, I took the easy way out

SLI/Crossfire are only relevant if you want to house two video cards in the one PC...this is great for gamers as they can run at higher speed and resolution, using 2 VGA cards to run on one monitor. For BOINC GPU tasks, I was using 2x GTX580 cards in SLI mode, which meant I could run 2 Collatz and/or 2 MilkyWay WU's per card...so 4 GPU tasks concurrently. But the cards got very hot (80+ deg C) as did the case, so for safety and bank account reasons, I dropped down to 1x GTX580 and that works fine now.

I don't plan on playing many games on it and those that I do would not really gain much from super fast graphics (more the strategy type gamer myself) so I'll probably not bother with SLI. I believe you can run multiple WUs on one card anyway if the capacity is there and the project supports it. I'm much happier fiddling with config and ini files.

Most cards these days will need the 1 * 16 slot, as this ensures max data transfer from the CPU on the mobo to the GPU on the card and back again. Some cards need 2 * 16 slots, so be careful in case your chosen card needs this and the brand/version of mobo you have, doesn't support it.

Just like ISA, EISA and now with PCI slots, they will all "work together" - but be aware that the Northbridge and Southbridge heatsinks might need extra cooling, if you add in too many cards...

My understanding was that the PCIe/PCI/ISA bus was only used for communication with the CPU (and other cards if no SLI/Crossfire was available)? In which case, lower width buses will add a bit extra to the overhead of a WU but not affect the actual running. I'll just have to aim at long running, small data load WUs

To start with though, there's only the one card, it's a single slot device and it will be in a x16 slot although I'll experiment running it at x8 and x4 just to get an idea of the impact it has.

Cores/shaders are indeed most important, though there comes a cost issue on the better cards, simply down to the extra cost of making such a card. And again, memory is fairly important, as on some higher spec cards, you can run multiple GPU WU's at once...So look for something with perhaps 2Gb on board as a minimum.

That's what I thought as some sites had them listed as 'cores', more cores and running faster = better crunching. I decided on a 4GB card so memory should be sufficient.

Given the option, I'd upgrade the PSU from the 750W type....simply due to the amount of juice the higher spec cards in your wish list will consume. Most of the higher spec cards will use perhaps 2x 6 or even 2x 8 pin PCIe connectors....and that will strain some PSU's. If the 750W PSU is a "good one", then try it and see. OTOH, if the 1200W is "spare" then I'd use it, simply because it'll be running well within it's limits, whereas the 750W might struggle, esp if you add other cards in later on.

It would require quite a bit of shuffling around to get boards and PSUs in the right configurations so I'm sticking with the 750W (it's a Corsair) for now to save upsetting anything else. If I decide on adding another card(s) then I'll take a couple of days and do the job properly.

You will find plenty of GTX770's and 780's on fleabay - simply because a lot of gamers are upgrading to the newer higher spec models. But they are having to pay a price premium for their upgrades, and hence they need to move their older cards onto new buyers. And for BOINC use, the difference between a 7xx and 9xx isn't that great, as both are pretty good.

I'm a little cautious about buying secondhand electronic components as I never know how they've been treated in the past. An extra 10 degrees of operating temperature halves the expected lifetime. Especially likely to have happened with gamers trying to wring the last ounce of performance out of their card before being forced to upgrade. Buying new, the '970 was half the price of the '780.

Good ventilation will be essential, as although the VGA cards have their own fans, with the CPU and other chipsets generating heat, plus what comes from the GPU, the PC internals will be cooking pretty quickly, esp if it's a compact case with the sides attached. So, a little bit of planning on the airflow would be helpful, esp in terms of getting cool air "into" the box. You might also want to think about air filters too, (to prevent cat hairs and dust building up inside....you've no doubt seen inside a large ASIC and see how much dust can be "drawn in", which can affect overall performance).

Hope this helps

regards
Tim

It's a full size case, two fans on the front, two on the back with the future options of one on the side and two on the top. The '970 vents it's hot air out of the back of the case through the mounting plate so it shouldn't add to the heat inside the case and one of the front fans is directed directly at the expansion slots so it should be getting a good supply of 'outside' air. No air filters at the moment but I'll check after a month or so and see what it's like (the cats aren't allowed in the same room as it!)

Thanks for the detailed response,

Regards, Mark

[UBT - Timbo]

big snip.
.
.
It's a full size case, two fans on the front, two on the back with the future options of one on the side and two on the top. The '970 vents it's hot air out of the back of the case through the mounting plate so it shouldn't add to the heat inside the case and one of the front fans is directed directly at the expansion slots so it should be getting a good supply of 'outside' air. No air filters at the moment but I'll check after a month or so and see what it's like (the cats aren't allowed in the same room as it!)

Thanks for the detailed response,

Regards, Mark

Hi Mark,

No problem - I think you've covered most of the points I've mentioned, so well done for working your way through it A 4Gb GTX970 will certainly raise your crunching capability (though not as much as an ASIC)....looks like Father Christmas arrived early with you this year

One other point, re: cooling
Don't forget that the entire VGA card will be radiating heat to the inside of the PC case from both sides. And even though there are vents on the end of the "rear" of the card, they aren't very large and hence it's easier for the heat to stay inside the PC, warming it up, than to be squeezed out through a few small slots. Plus the fans on the VGA cards are pushing air from the bottom of the case (if used vertically, as normal) "upwards" onto the heatsinks, whereas, in design terms, they should be blowing air across the heatsinks towards the rear of the case...

So, be prepared for a much warmer PC case And be careful of using "cheap" power extension/convertor cables - I did this once and found that the insulation on a convertor cable (going from Molex to 6 pin PCIe) couldn't handle the current draw and the insulation melted Thankfully, the side was off and I was monitoring something else, when I saw some molten plastic coming from the short cable.

regards
Tim

[Woodles]

Hi Mark,

No problem - I think you've covered most of the points I've mentioned, so well done for working your way through it A 4Gb GTX970 will certainly raise your crunching capability (though not as much as an ASIC)....looks like Father Christmas arrived early with you this year

One other point, re: cooling
Don't forget that the entire VGA card will be radiating heat to the inside of the PC case from both sides. And even though there are vents on the end of the "rear" of the card, they aren't very large and hence it's easier for the heat to stay inside the PC, warming it up, than to be squeezed out through a few small slots. Plus the fans on the VGA cards are pushing air from the bottom of the case (if used vertically, as normal) "upwards" onto the heatsinks, whereas, in design terms, they should be blowing air across the heatsinks towards the rear of the case...

So, be prepared for a much warmer PC case And be careful of using "cheap" power extension/convertor cables - I did this once and found that the insulation on a convertor cable (going from Molex to 6 pin PCIe) couldn't handle the current draw and the insulation melted Thankfully, the side was off and I was monitoring something else, when I saw some molten plastic coming from the short cable.

regards
Tim

Hi Tim,

Well I thought that with the Primegrid crunch coming up very soon and the 2016 monthly projects being both GPU and CPU based, I'd better start looking at GPUs (Father Christmas is much more reliable when you give people websites and product numbers instead of 'a jumper' )

Fair point, I hadn't considered that, I shall keep an eye on internal case temperatures as well. Fortunately, the CPU is running with all cores 24/7 at the moment and only reaches 68 degrees, the spec says fine until 105 degrees so there's a bit of headroom.

The fan does indeed push air upwards into the card but it's enclosed with a full length heatsink so the air has to travel across the heatsink before exiting. Still worth keeping an eye on though, thanks for the warning.

i had a similar problem with extension cables before. I made a cable up from a 6 way PCIe connector to a length of terminal block to plug four RBox 32s into. Foolishly, I thought that a single piece of mains cable would be alright for the current ... it wasn't

Mark

[UBT - Timbo]

Hi Tim,

Well I thought that with the Primegrid crunch coming up very soon and the 2016 monthly projects being both GPU and CPU based, I'd better start looking at GPUs (Father Christmas is much more reliable when you give people websites and product numbers instead of 'a jumper' )

Fair point, I hadn't considered that, I shall keep an eye on internal case temperatures as well. Fortunately, the CPU is running with all cores 24/7 at the moment and only reaches 68 degrees, the spec says fine until 105 degrees so there's a bit of headroom.

The fan does indeed push air upwards into the card but it's enclosed with a full length heatsink so the air has to travel across the heatsink before exiting. Still worth keeping an eye on though, thanks for the warning.

i had a similar problem with extension cables before. I made a cable up from a 6 way PCIe connector to a length of terminal block to plug four RBox 32s into. Foolishly, I thought that a single piece of mains cable would be alright for the current ... it wasn't

Mark

Hi Mark,

No probs.

Most of the GTX970s I've seen are like this:



with most having say two or three fans.

I guess the one you are getting is like this:



which is very similar to my GTX580 (and 680's and 780's too.)

The only issue I found is that a single fan can be noisier, simply because it has to turn faster, even though (in theory) it is blowing the (warm) air down the length of the enclosure and towards the rear of the case. I did find also, that the PCB side of the 580 got very, very warm.....so I added a case fan to blow air over the PCB side, which helped.

Hope you have some fun with PrimeGrid

regards
Tim

[UBT - Timbo]

Hi Mark,

Just another thought - you might be able to run multiple WU's on a single 970.

See this thread on MilkyWay:

http://milkyway.cs.rpi.edu/milkyway/for ... 3118#56955

for the general idea.

On PrimeGrid forum, there's this thread:

http://www.primegrid.com/forum_thread.p ... true#90391

though I'm sure you can find others

Make sure you get the GPU-Z application - it's free and before you do multiple WU's you can see how stressed the GPU is, running a single.

regards
Tim

[Woodles]

Hi Mark,

No probs.

Most of the GTX970s I've seen are like this:

snip

with most having say two or three fans.

I guess the one you are getting is like this:

snip

which is very similar to my GTX580 (and 680's and 780's too.)

The only issue I found is that a single fan can be noisier, simply because it has to turn faster, even though (in theory) it is blowing the (warm) air down the length of the enclosure and towards the rear of the case. I did find also, that the PCB side of the 580 got very, very warm.....so I added a case fan to blow air over the PCB side, which helped.

Hope you have some fun with PrimeGrid

regards
Tim

Hi Tim,

The one I'm getting looks like this

The ones with multiple fans are open cased and yes, the hot air is exhausted straight into the case.

There were a couple of comments about the fan being audible when run flat out but I think it'll be a while before I get anywhere near trying that sort of performance. The way the case is set up, there's a fan at the front blowing outside air onto the fan end from about 3 inches away and another case fan sucking air out from just above it. There's also the PSU fan blowing up at it from roughly the same distance ... although I'm not sure that'll help

Thanks, that's the idea.

It looks like I may indeed manage more than one WU at a time ... once I figure out to do it!

Regards,

Mark

[UBT - Timbo]

Hi Tim,

The one I'm getting looks like this (snip)

The ones with multiple fans are open cased and yes, the hot air is exhausted straight into the case.

There were a couple of comments about the fan being audible when run flat out but I think it'll be a while before I get anywhere near trying that sort of performance. The way the case is set up, there's a fan at the front blowing outside air onto the fan end from about 3 inches away and another case fan sucking air out from just above it. There's also the PSU fan blowing up at it from roughly the same distance ... although I'm not sure that'll help

Thanks, that's the idea.

It looks like I may indeed manage more than one WU at a time ... once I figure out to do it!

Regards,

Mark

Hi Mark,

That looks like a nice card...always liked the Gigabyte designs (Even though I've also got 2x MSI Twin Frozr cards as well...only one currently being used as the other PC is waiting to be built, once I get a new mobo).

I just downloaded some PG WU's as I wanted to see how well my GTX5xx series cards might fare on them.

Sadly, using GPU-Z, I found that both of the "in use" cards I have run at near "full pelt" on just one WU....so, even though the PG WU's don't use much memory, they do use a lot of "processing power" and hence I can only run one PG WU at a time.

What was also a little worrying is that the GPU temp (which was less than 70 deg C on a single MilkyWay WU), was into the 80's on a PG task - and that's with the fan running at max too. So, be careful at first, and maybe just run some single WUs to begin with and monitor what GPU-Z tells you about how hard the card is being "pushed".

regards
Tim

[Woodles]

Hi Mark,

That looks like a nice card...always liked the Gigabyte designs (Even though I've also got 2x MSI Twin Frozr cards as well...only one currently being used as the other PC is waiting to be built, once I get a new mobo).

I just downloaded some PG WU's as I wanted to see how well my GTX5xx series cards might fare on them.

Sadly, using GPU-Z, I found that both of the "in use" cards I have run at near "full pelt" on just one WU....so, even though the PG WU's don't use much memory, they do use a lot of "processing power" and hence I can only run one PG WU at a time.

What was also a little worrying is that the GPU temp (which was less than 70 deg C on a single MilkyWay WU), was into the 80's on a PG task - and that's with the fan running at max too. So, be careful at first, and maybe just run some single WUs to begin with and monitor what GPU-Z tells you about how hard the card is being "pushed".

regards
Tim

Hi Tim,

Finally got Ubuntu to recognise the card and update the drivers and running a few experiments on the PG tasks.

Sadly, GPU-Z hasn't been ported to Linux as far as I could find out but the Nvidia control panel reports usage and thermal info.

Same as you found, one WU takes almost the whole card (90% - 92%) however the temperatures aren't as bad as yours. Ambient was 29 degrees, GPU idle was 35 degrees and when running, 76 degrees with 44% fan.

Encouragingly, the PCIe bandwidth used was less than 2% so there shouldn't be a hit from running it in a x8 or even a x4 slot and as you said, plenty of memory available, each WU only takes about 250 Meg.

Regards,

Mark

[UBT - Timbo]

Hi Tim,

Finally got Ubuntu to recognise the card and update the drivers and running a few experiments on the PG tasks.

Sadly, GPU-Z hasn't been ported to Linux as far as I could find out but the Nvidia control panel reports usage and thermal info.

Same as you found, one WU takes almost the whole card (90% - 92%) however the temperatures aren't as bad as yours. Ambient was 29 degrees, GPU idle was 35 degrees and when running, 76 degrees with 44% fan.

Encouragingly, the PCIe bandwidth used was less than 2% so there shouldn't be a hit from running it in a x8 or even a x4 slot and as you said, plenty of memory available, each WU only takes about 250 Meg.

Regards,

Mark

Hi Mark

Excellent news. Sorry about the mis-info on GPU-Z - I was stuck in "Windows" mode and hadn't remembered you were going to run Linux.

I've never been able to figure how PCIe slots work...I thought that if the card had a long connector, you *had* to use a long slot (ie x16)...sounds like you can use a shorter slot, as long as you don't saturate the card with data. I'm not quite sure how that would work if one is using the card for display purposes as well as GPU crunching

I've only got the one PG WU left now - it's a Genefer 3.09 WU with a run time of...wait for it...120 hours !! It's slowly being processed and GPU is being underclocked (to keep heat dissipation low) and temp is currently just 52 deg C.

Looks like you're getting well over 3,000 credits per hour at the moment....so, maybe 80,000 per day That's a good amount and will stand you in good stead over the coming weeks and months

regards
Tim

[Woodles]

Hi Mark

Excellent news. Sorry about the mis-info on GPU-Z - I was stuck in "Windows" mode and hadn't remembered you were going to run Linux.

Hi Tim,

No problem, I'm slowly getting used to not having the usual Notepad, calculator, Explorer type programmes around and seem to be running a text based terminal more often than not, it takes me back to teh days of DOS

I've never been able to figure how PCIe slots work...I thought that if the card had a long connector, you *had* to use a long slot (ie x16)...sounds like you can use a shorter slot, as long as you don't saturate the card with data. I'm not quite sure how that would work if one is using the card for display purposes as well as GPU crunching

Same as PCI slots, they auto adjust to what's available. You can plug a full length graphics card into a PCIe x1 slot if there's no barriers at the ends of the connector and it'll still work. The slot width only affects how quickly data can get to and from the card, not what it does with it once it has it. So loading different images into a game would be penalised but moving them around the screen wouldn't.

I've only got the one PG WU left now - it's a Genefer 3.09 WU with a run time of...wait for it...120 hours !! It's slowly being processed and GPU is being underclocked (to keep heat dissipation low) and temp is currently just 52 deg C.

Looks like you're getting well over 3,000 credits per hour at the moment....so, maybe 80,000 per day That's a good amount and will stand you in good stead over the coming weeks and months

regards
Tim

No chance of me attempting anything that long lasting yet, I'm still looking for instant results to judge how things are going and the best combinations.

(3,371 credits every 11 3/4 minutes at the moment once they validate )

Regards,

Mark

[UBT - Timbo]

No chance of me attempting anything that long lasting yet, I'm still looking for instant results to judge how things are going and the best combinations.

(3,371 credits every 11 3/4 minutes at the moment once they validate )

Regards,

Mark

That's even better - I was going by the UBTstats page on PrimeGrid and that showed about 3,300+ for one hour (at 9pm tonight) and then another 3,300+ for the second hour (10pm). And now another 3,300+ for the last hour (11pm).

http://ubtstats.co.uk/ubt/ubt_graph.php ... me=Woodles

regards
Tim

[Woodles]

Hopefully when there's a steady stream of validated results I'll have a better idea of how it's really doing. (It still gets more credits in an hour than my old, single core laptop does in a day!)