Hi

I was wondering if I could get the assistance of those more knowledgeable than me (ie most of you) in cabling up a new system:

My hardware - a 12 bay Supermicro SC826BE2C with SAS3 dual-expander backplane, an X9-DRF EATX motherboard and 2 x Xeon E5 2650l V2 CPUs ('controller').

A second Supermicro CSE826 JBOD 12-bay chassis with BPN-SAS2-826EL1 backplane ('shelf').

My questions are:

a) What HBA will be best (I was looking at the LSI 9211-8i, but am completely open to suggestions)

b) What cabling will I need? I was thinking HBA to backplane of 'controller' and then downstream SAS connection from 'controller' to 'shelf', or am I making things way too straightforward? 

I'd appreciate your help, as although I am likely to learn from my mistakes, I'd rather get it right first time than waste money needlessly!

I am referencing the EL1 manual, which can be found here. If I've left any pertinent information out, let me know and I will supply it. Thanks again, and Merry Christmas everyone!

Edited December 22, 20187 yr by DavidInDE
Included explanation of which chassis was the controller and which the shelf

I would use a 9207-8i instead, since it's PCIe 3.0 and if you use all bays with fast disks you can use the extra bandwidth, this would get you around 185MB/s per disk with all slots occupied, as for cabling, one cable from the HBA to each backplane, no need for cascading,

Thanks for your advice johnnie.black....would I be better off going with something like this, as it is also PCIe 3.0, but has the four internal connections (for the controller), plus four external (which would go to the shelf). Am I thinking correctly?  

8 minutes ago, DavidInDE said:

would I be better off going with something like this, as it is also PCIe 3.0, but has the four internal connections (for the controller), plus four external (which would go to the shelf). Am I thinking correctly?  

Yep, that's even better, so you don't need an internal to external adapter.

Quick question...since the backplane of the controller is SAS3, and the backplane of the disk shelf is SAS2, will I get higher IO sppeds from those drives connected to the controller backplane? or will the bottleneck be at the PCIe slot? Thanks for the benefit of your knowledge!

22 minutes ago, DavidInDE said:

since the backplane of the controller is SAS3, and the backplane of the disk shelf is SAS2, will I get higher IO sppeds from those drives connected to the controller backplane?

Only if you get a SAS3 HBA like the 9300-8i (or 4i4e) and your devices are SAS3, but for 12 disks SAS2 is good, though if you can get a 9300 for not much more money it's worth getting.

 

A 4 lane SAS1 link has a maximum theoretical bandwidth of 1200MB/s, SAS2 doubles that to 2400, double again for SAS3 to 4800MB/s, there's always some overhead, I tested with SAS1 and SAS2 expanders and real world max speeds are 1100 and 2200MB/s respectively, never tested SAS3 but assume overhead would be similar, so around 4400MB/s usable.

 

So, if you use a SAS2 HBA you'll have 2200MB/s usable for each backplane, the previously mentioned 185MB/s per disk if all 12 bays are in use, 4400MB/s for both backplanes total, which is above the x8 PCIe 2.0 link, hence why I suggested a PCIe 3.0 HBA.

 

If you get a SAS3 HBA you'll have around 4400 + 2200 max bandwidth, which is likely a little above what a x8 PCIe 3.0 can handle, but don't forget only if you were to use SAS3 devices, and very fast ones at that, like SAS3 SSDs, if using SAS2 or SATA3 disks link will still be 2400MB/s, so probably not worth getting one unless you can find a cheap 9300 and/or you're using SAS3 SSDs of very fast SAS3 disks, like 12 or 14TB 7200rpm disks.

6 hours ago, johnnie.black said:

but don't forget only if you were to use SAS3 devices

Let me correct myself, since I never had one I sometimes forget, LSI SAS3 HBAs and expanders have a feature, called DataBolt, that is supposedly able to use full SAS3 bandwidth with SAS2/SATA3 devices, never tested so don't know how well it works, but good to keep in mind.

I am impressed - I actually learned more from those two posts than several hours of web searching (with an unhealthy fixation on Supermicro's web site) yielded...

Btw, let me also post this, I was looking at some LSI pdfs because I didn't remember the name of the DataBolt feature, and found these real world bandwidth values that line up well with my experience, especially the SAS bandwidth, PCIe might be a little optimistic, at least I was never able to get more than 3000MB/s from a x8 PCIe 2.0 slot:

 

 

 

Edited December 23, 20187 yr by johnnie.black

Thank you. You've been really helpful; it's always good to know you've learned something new.

I am the first to admit my ignorance on this topic, so...

 

Can one hook up a disk shelf (as we discussed) and use both the original storage server AND the shelf to store 12 SATA disks on each, or is this locking the user in to the use of SAS drives?

Edited December 26, 20187 yr by DavidInDE
I can't type worth two cents

You can use SAS or SATA on any of the shelfs, you can even have mixed SAS and SATA on the same shelf, only ting to be aware is speed will be limited if using slower linking devices, like SAS1/SATA2 devices in any of them.

Thanks Johnnie and Happy New Year!