CEB Form Factor? (4/7 GPUs)

Having problems installing that new stick of memory? Found some great software or having issues with something? Or maybe want to chat about your PlayStation, X-Box, Nintendo, Sega, even your old Spectrum 48k....! Or maybe something you want to sell or acquire (computing related of course!). Let us know here...
Post Reply
chriscambridge
Active UBT Contributor 1+ yr
Posts: 2178
Joined: Mon Aug 08, 2016 1:56 pm
Location: UK

CEB Form Factor? (4/7 GPUs)

Post by chriscambridge »

https://www.asus.com/Motherboards/X99-E ... fications/

I have found a motherboard (LGA 2001-V3) which has 7 x16 sized GPU slots, and more importantly can run the number GPUs at the following speeds:

(1) x16
(2) x16,x16
(3) x16,x16,x16
(4) x16,x16.x16,x16
(7) x16,x8,x8,x8,x8,x8,x8

However the form factor is listed as CEB, which is one I have never heard of.

CEB Form Factor
12 inch x 10.5 inch ( 30.5 cm x 26.7 cm )

Anyone have any experience of this? Would any of the ATX holes line up?

Image
Last edited by chriscambridge on Tue Apr 04, 2017 8:25 am, edited 2 times in total.
chriscambridge
Active UBT Contributor 1+ yr
Posts: 2178
Joined: Mon Aug 08, 2016 1:56 pm
Location: UK

Re: CEB Form Factor? (The 4/7 GPU Rack)

Post by chriscambridge »

The SSI CEB specification was derived from the EEB and ATX specifications. SSI CEB motherboards have the same IO connector area and many of the same motherboard mounting holes as ATX motherboards, although SSI CEB motherboards are larger in size than ATX motherboards and have different processor mounting holes.

https://en.wikipedia.org/wiki/SSI_CEB

Which I guess is saying that an ATX case would have some of the correct holes, but would be just too small, where as an E-ATX case should fit fine, and have many of the correct holes.

Its good that the four x16 speed slots (grey) are double width so you could actually fit 4 high GPUs into these, with enough fans within the case, @ Mark :)

Although personally I like the idea of a 4 or 7 GPU Rack..
Woodles
UBT Contributor
Posts: 11757
Joined: Thu Dec 20, 2007 12:00 am
Location: Cambridgeshire

Re: CEB Form Factor? (4/7 GPUs)

Post by Woodles »

Hi Chris,

Nice board but expensive though!

I've had a look at the specifications for the CEB mounting holes and the ATX mounting holes and the only one missing is 'B'
http://www.formfactors.org/developer/specs/atx2_2.pdf so no reason why it shouldn't fit on ATX mountings..

I was a little confused about the PCIe distribution as the 2011 chips only provide 40 lanes of PCIe and the X99 chipset another 8? Reading the manual though, only 32 lanes come from the CPU for all the PCIe slots and they're multiplexed so you may get x16/x16/x16/x16 connectivity but you're only getting x8/x8/x8/x8 flat out speed. Shouldn't matter in most cases as the PCIe bus is very rarely maxed out, especially at 3.0 speeds.

Some thought required,

Mark
Last edited by Woodles on Tue Apr 04, 2017 10:38 am, edited 1 time in total.
chriscambridge
Active UBT Contributor 1+ yr
Posts: 2178
Joined: Mon Aug 08, 2016 1:56 pm
Location: UK

Re: CEB Form Factor? (4/7 GPUs)

Post by chriscambridge »

Hi Mark,

There is a couple on Ebay for £250-£300, around 50% cheaper than retail.

Thanks for the heads up, on both points.

Perhaps I should read the manual instead of reading the misleading 'specifications' on their website!

"Slot size, width, link, speed, gen,.."

I had no idea the X99 chipset also provided some PCIE lanes.

In truth, I am still unsure how the PCIE lanes fit in regards to the Front side bus (FSB) which is obviously the main highway from the CPU to other elements.

OK well the search continues then.. I am looking for FULL x16 slot width, as well as x16 speed/link, @ Gen 3,..

I am thinking perhaps a dual Xeon board, E-ATX, should be what I need, as then you would have 80 + chipset PCIE lanes available. I guess max would be five GPUs.

Obviously I would not be using any other PCI/PCIE devices.
Woodles
UBT Contributor
Posts: 11757
Joined: Thu Dec 20, 2007 12:00 am
Location: Cambridgeshire

Re: CEB Form Factor? (4/7 GPUs)

Post by Woodles »

Hi Chris,

Most chipsets produce PCIe lanes, the X99 produces 8 @ 2.0 speeds but the motherboard in question dedicates these to the USB/SATA/Network cards etc

The front side bus used to be the main highway to the rest of the peripherals but these days basically just connects the CPU to the memory. It's been superseded by a DMI bus to the northbridge since the FSB can't keep up.

The PCIe lanes from the CPU usually go directly to the slots, some switching is involved in this and other cases to make it appear that there are more lanes than physically present. The other PCIe lanes come from the chipset and may or may not be multiplexed similarly or be dedicated to a particular slot (as with the dual xeon motherboards previously)

You have a choice of:
An i7-6700 with a Z270 based motherboard
40 PCIe lanes at 3.0
~£450 new
Best price, reasonable CPU performance, new components

An i7-6900K with an X99 based motherboard
40 PCIe lanes at 3.0 plus 8 PCIe lanes @ 2.0
~£1250 new
Most expensive, brilliant CPU performance, new CPU, established motherboard

Dual E5-2630s on a C602 based motherboard
80 PCIe lanes @ 3.0 plus 8 PCIe lanes @ 2.0
~£500 but I doubt you'll get them new
Reasonable price, loads of PCIe lanes, fair CPU performance (helped by the sheer number of cores), retired components.

These are all best case, different motherboards will carve up the PCIe lanes for onboard peripherals or simply not use them if there's not enough board space for another connector.

As an aside, I didn't find more than a 1-3% difference in BOINC GPU crunching capabilities in going from x16 to x8 bus width and x4 was only about a 7-8% drop. Of course, you may not be using it for Boinc (low bandwidth)

HTH.

Mark
Last edited by Woodles on Tue Apr 04, 2017 3:09 pm, edited 1 time in total.
chriscambridge
Active UBT Contributor 1+ yr
Posts: 2178
Joined: Mon Aug 08, 2016 1:56 pm
Location: UK

Re: CEB Form Factor? (4/7 GPUs)

Post by chriscambridge »

The front side bus used to be the main highway to the rest of the peripherals but these days basically just connects the CPU to the memory. It's been superseded by a DMI bus to the northbridge these days since the FSB can't keep up.
That shows how long ago I did my degree in computer science; nearly 20 years ago now. AGP, FSB, its all different now :? DMI, Northbridge; I am going to have to do some Googling.

If you remember rightly, I already have an X99 mobo with a V3; but a single CPU like we know only offers the dedicated 40 lanes, so no true 5 full speed GPU slots on this.

I think the best would be dual Xeons, as at least that will give 40+40 lanes, which 5*16 obviously equates to the same number. I prefer the V3's as they use substantially less TDP than the V1's, plus you get the DDR4 ram, and AVX2.

I have found some Supermicro mobos that seem to offer what I need, but not as E-ATX, but their silly proprietary form factor size; perhaps the way forward is to get one of these mobos, and order a sheet of plastic (as used on the existing GPU Rack), and just drill the holes in the correct place. Although if I remember rightly you have use a special drill attachment to put the screw threads in. I am no expert in this though.

The main reason for wanting full speed etc, is to future proof the rack. As we know every N months/years new GPUs come out which pump more data through the PCIE bus, plus no doubt you will get some future projects using more CPU cores, or perhaps SLI etc, and hence keeping the full dedicated speeds/sizes if possible I think would be best.

I do not mind the 970's running at x8 speed but really I want anything above to be at x16, full width, gen3. Although all of this logic may be undone if PCIE 4/5 gets released anytime soon.
Woodles
UBT Contributor
Posts: 11757
Joined: Thu Dec 20, 2007 12:00 am
Location: Cambridgeshire

Re: CEB Form Factor? (4/7 GPUs)

Post by Woodles »

The only way you're going to get full x16 connectivity for 5 GPUs is with dual (or more) Xeons. Even then you have to get the correct motherboard or they might not be connected up (as the Intel 2600s weren't)

The V3s use less power and use faster RAM but I was just looking at it from a GPU point of view. You'd need to calculate the difference between expensive initial costs with lower running costs and cheaper initial cost and higher running costs.

If you're going to put it into a GPU rack instead of a case then the form factor doesn't matter at all, you just make a platform of the correct size and shape. Screw threads are normally put in by hand with a die after the holes have been drilled but it's easier just to drill plain holes and use a threaded spacer that you can then bolt to the rack and the board like one of these http://uk.farnell.com/productimages/sta ... 683596.jpg

Let me know how you get on as I'm back in the market for a GPU rack :D

Mark
chriscambridge
Active UBT Contributor 1+ yr
Posts: 2178
Joined: Mon Aug 08, 2016 1:56 pm
Location: UK

Re: CEB Form Factor? (4/7 GPUs)

Post by chriscambridge »

If after a passed away, you were to dig up my casket, there you would find next to me my dearly beloved gpu rack!

So I have just taken a look at the rack, as its empty at the moment waiting for the X99, and it actually uses the method you mention. They have drilled holes in the board, which stand-offs fit in, but special ones that also screw in place from beneath to hold them in place.

So that makes it very simple to setup a new form factor; drill new holes of the right size and mobo position -and then just move the screwed stand-off's! Its just a case of making sure the mobo fits with the PSU and HD in place. That also means you could actually make the rack bigger by just using a new bit of plastic and drilling new holes for SSI-EEB, or Supermicro mobos etc.
Last edited by chriscambridge on Wed Apr 05, 2017 9:56 am, edited 1 time in total.
chriscambridge
Active UBT Contributor 1+ yr
Posts: 2178
Joined: Mon Aug 08, 2016 1:56 pm
Location: UK

Re: CEB Form Factor? (4/7 GPUs)

Post by chriscambridge »

In terms of a 5 GPU board with full speeds, then I think I may have found the solution:

The Supermicro X10DRG-Q (Intel C612) Workstation Motherboard (LGA2011-V3)

[4 PCI-E 3.0 x16 (double-width) slots + 2 PCI-E 3.0 x8 (1 in x16) slots]

What do you think Mark? Do you see any problems with getting full speeds from this mobo?

This review on this mobo from ServeTheHome states that the full 80+ PCIE lanes are available to the slots

"Four dual slot PCIe x16 slots that do not have components underneath to obscure. Solid cooling layout. 80 PCIe 3.0 and 4 PCIe 2.0 lanes exposed to users via slots which is best in class"..

https://www.servethehome.com/supermicro ... therboard/

https://www.supermicro.com/products/mot ... 0DRG-Q.cfm (product info)

https://www.supermicro.com/manuals/moth ... L-1677.pdf (pdf manual)

Image

Woodles wrote:Let me know how you get on as I'm back in the market for a GPU rack :D
Well seeing as you GPU collection is coming on some what, and you do not mind x8 speed.. How about an 10 GPU box/rack?

The Supermicro X10DRX (Intel C612) Server Motherboard (LGA2011-V3)

[10x PCI-E 3.0 x8 slots]

https://www.supermicro.com/products/mot ... X10DRX.cfm (product info)

https://www.supermicro.com/manuals/moth ... L-1725.pdf (pdf manual)

Image

The only thing that makes me wonder about these boards is that they seem to use the rectangle 2011-V3 socket rather than the square version - and I am not sure if the normal heat sinks eg EVO-212 etc, can work with these?

They are also proprietary form factor so you would have to either use a Supermicro case, or a customised GPU rack.
The X10DRG-Q is sized: 15.2″ x 13.2″.
Woodles
UBT Contributor
Posts: 11757
Joined: Thu Dec 20, 2007 12:00 am
Location: Cambridgeshire

Re: CEB Form Factor? (4/7 GPUs)

Post by Woodles »

chriscambridge wrote:In terms of a 5 GPU board with full speeds, then I think I may have found the solution:

The Supermicro X10DRG-Q (Intel C612) Workstation Motherboard (LGA2011-V3)

[4 PCI-E 3.0 x16 (double-width) slots + 2 PCI-E 3.0 x8 (1 in x16) slots]

What do you think Mark? Do you see any problems with getting full speeds from this mobo?
Hi Chris,

As far as I can see that would be the perfect solution for you. Plenty of room for four double width, x16 GPUs and a couple of slots left over for other peripherals.

According to the manual, all the x16 slots are direct connections to the PCI ports on the CPUs so they are true PCIe 3.0 at x16 width providing full speed all the time.

The only possible problem would be if you used a CPU that didn't have 40 PCIe lanes (I haven't checked the list of CPUs supported to see if that's even possible as I expect you'd avoid that anyway)
chriscambridge wrote:Well seeing as your GPU collection is coming on some what, and you do not mind x8 speed.. How about a 10 GPU box/rack?

The Supermicro X10DRX (Intel C612) Server Motherboard (LGA2011-V3)

[10x PCI-E 3.0 x8 slots]
That's no good, slots 9 and 10 aren't open ended so you can't plug a x16 card in there without a x8 to x16 extender cable or modifying the socket, you'd be limited to only having eight x16 cards :P

I'm looking at the Asus P9X79-E WS and Asrock X79 Extreme11 at the moment (as well as the Asus X99-E-10G WS you found previously) to see if they're any good. They also multiplex the PCIe lanes to get seven x8/x16 slots but I'm not too bothered about that. I'll add the Supermicro Z10DRX to the list.
chriscambridge wrote:The only thing that makes me wonder about these boards is that they seem to use the rectangle 2011-V3 socket rather than the square version - and I am not sure if the normal heat sinks eg EVO-212 etc, can work with these?
The EVO 212 is specified as being able to connect to the 2011-3 socket as well as the normal 2011 socket so it shouldn't be a problem.

Regards,

Mark
chriscambridge
Active UBT Contributor 1+ yr
Posts: 2178
Joined: Mon Aug 08, 2016 1:56 pm
Location: UK

Re: CEB Form Factor? (4/7 GPUs)

Post by chriscambridge »

Thanks for the advice; that did not take long to find a solution for the Killer GPU Rack..

BTW I replied to your message.
That's no good, slots 9 and 10 aren't open ended so you can't plug a x16 card in there without a x8 to x16 extender cable or modifying the socket, you'd be limited to only having eight x16 cards
Supermicro are unbelievable; it is like they just do not care about the 10 GPU Wizards operating in this current sphere. How inconsiderate!
I'm looking at the Asus P9X79-E WS and Asrock X79 Extreme11
Well something else to keep in mind Mark is I may be open to selling the X99 mobo I have; its practically brand new. If I remember rightly it has 5/6 GPU slots. That is if you do not mind x16/x8 speeds and PCIE switching/mulitplexing..

The RAM problem turned out to be a case of Xeons cannot handle that fast RAM, so if you were going to run an i7 then I could probably put the XMP Ram into the deal too; If you remember rightly you can O/C CPUs on this board just by clicking a button and it works out all the factors involved, which is why I went with the Corsair Vengeance XMP ram.

If I am not mistaken the Deluxe is the best X99 Asus mobo of the range. It has some pretty serious kit that comes with it..

What I would do is get another E5-2683 V3 and just take the existing one out of the X99, hence why I could sell the mobo and RAM; the new mobo would need RAM at lower speeds (due to the Xeons), probably ECC would be best.

If not, then I will just keep the board, and then at some stage just get a 2011-V3 I7 K version, and O/C the CPU and run it as a super fast box, like the i7 4790s I have; fast compared to the Xeons that is.
Woodles
UBT Contributor
Posts: 11757
Joined: Thu Dec 20, 2007 12:00 am
Location: Cambridgeshire

Re: CEB Form Factor? (4/7 GPUs)

Post by Woodles »

Hi Chris,

Well if you're happy with only four full speed GPUs ... :P

As did I, thanks.

I know, I'm surprised they're still in business the way the only cater to the lower end of the market! :shock:

Thanks, I shall keep that in mind, do you know the model number of the X99 motherboard you have? (I could go and find it from the forum but it's easier to ask :) ) and I'll take a detailed look at the specification. Makes sense to recycle kit that you don't have an immediate need for, by the time you get around to wanting it again, Supermicro might actually be making a 'proper' 10 GPU motherboard :D

Regards,

Mark
chriscambridge
Active UBT Contributor 1+ yr
Posts: 2178
Joined: Mon Aug 08, 2016 1:56 pm
Location: UK

Re: CEB Form Factor? (4/7 GPUs)

Post by chriscambridge »

Err model number? :? Where would I find that?

I just know its the Asus X99 Duluxe (not Deluxe II which has less PCIE slots or speed), or Deluxe-U3 (I think thats the name, U something).

https://www.asus.com/uk/Motherboards/X9 ... fications/ (specifications)
https://www.asus.com/uk/Motherboards/X99DELUXE/ (product info)

Image

Image
chriscambridge
Active UBT Contributor 1+ yr
Posts: 2178
Joined: Mon Aug 08, 2016 1:56 pm
Location: UK

Re: CEB Form Factor? (4/7 GPUs)

Post by chriscambridge »

Just to add it has a serious amount of fan headers - and a plug in device to add in even more, which clearly if you were running loads of GPUs would be *very handy to run 7 fans on the case, plus the fan on the CPU.

Looking at the BIOS above, it looks like it has 8 fan headers, with full graphic control, either manual or automatic.

It's also ATX form factor. You might want to have a look at this cheap but good ATX case I got the other day which comes with 3 fans as standard (front + two top). The only con is no HD space only SSD; But I just screwed the HD into a SSD hole and that has keep it place okay.

£35 Ebay UK.

http://www.ebay.co.uk/itm/371860635022? ... EBIDX%3AIT

Image
Woodles
UBT Contributor
Posts: 11757
Joined: Thu Dec 20, 2007 12:00 am
Location: Cambridgeshire

Re: CEB Form Factor? (4/7 GPUs)

Post by Woodles »

chriscambridge wrote:Err model number? :? Where would I find that?

I just know its the Asus X99 Duluxe (not Deluxe II which has less PCIE slots or speed), or Deluxe-U3 (I think thats the name, U something).

https://www.asus.com/uk/Motherboards/X9 ... fications/ (specifications)
https://www.asus.com/uk/Motherboards/X99DELUXE/ (product info)
Hi Chris,

That's what i was looking for so I could look up it's specifications and user manual ... but i see you've already listed them, thanks.

re the fan headers, if it can do five PCIe 3.0 slots at x8 (which it appears to do) then it's going in the GPU rack and fans aren't an issue. Same goes for the case, thanks anyway.

Mark
chriscambridge
Active UBT Contributor 1+ yr
Posts: 2178
Joined: Mon Aug 08, 2016 1:56 pm
Location: UK

Re: CEB Form Factor? (4/7 GPUs)

Post by chriscambridge »

I agree on the GPU Rack front; I only used the ATX box for the i7-4790, which was pretty good as the mobo (gaming 3) glows red, just like the case fans.

From the specification:

40-Lane CPU-
5 x PCIe 3.0/2.0 x16 (x16, x16/x16, x16/x16/x8, x8/x8/x16/x8, x8/x8/x8/x8/x8 mode) *1
1 x PCIe 2.0 x4 (max at x4 mode) #1

#1: The PCIe x16_5 shares bandwidth with M.2 x4. Triple PCIe 3.0/2.0 configuration is default set at x8/x8/x8. Adjust PCIEX16_5 Slot Bandwidth in BIOS.

28-Lane CPU-
3 x PCIe 3.0/2.0 x16 (x16, x16/x8, x8/x8/x8)
2 x PCIe 2.0 x16 (x1 mode)
1 x PCIe 2.0 x4 (max at x4 mode) #2

#2: The PCIex4_1, USB3_E12 and SATAEXPRESS_E1 connectors share the same bandwidth. The SATAEXPRESS_E1 will be disabled when there is a device installed on PCIEX4_1 slot. Set this option to X2 Mode or X4 Mode when the installed PCIe device is higher than X4 interface.
Post Reply