SAS controllers and their usage

[brian89gp]

Just a curiosity I have had based on observation of the preference by many on here when using the common 8 port SAS controllers to have 1 drive per port (I understand that on a cost basis, but not a performance basis), or if using an expander to have the cache/parity/whatever not on the expander and directly on one of the eight ports.  My question is, is there a noticable difference?

 

For reference, worked on a SAN that uses LSI 9201-16e SAS cards (16x 6gb, PCIe 2.0 8x)  More ports but same PCIe bus speed/bandwidth as a typical IBM M1015.  It uses full loops for the SAS connections, so the 4 ports on the card will do two loops.  Each loop is up to 96 drives for up to 192 drives per card (up to 15k enterprise drives even).  And interestingly enough, it is almost impossible even with that many drives to push the limits of that card, and that is doing the traditional RAID in software/CPU where every read/write will hit multiple drives at once not the one plus parity in unRAID.

 

Maybe I am missing something obvious.  I saw the cards, remember the reading here, and thought hmmm.

[garycase]

First, let's do a bit of math ...

 

A PCIe v2 x8 slot has 4GB/s of bandwidth.  Independent of the topology of a card that's plugged into it (i.e. whether it spreads its bandwidth via extenders or not), that's what's available.  That IS a lot of bandwidth, but with modern 1TB/platter drives capable of transfer rates up to almost 200MB/s, you could max it out at around 20 drives.

 

There's a BIG difference in the use you outlined, however ==> it's NOT USING the PCIe interface EXCEPT for RAID <-> PC transfers ... i.e. the actual data FROM the RAID array.  All RAID calculations and array reads/writes are done by the CONTROLLER (i.e. the LSI card) ... NOT by the PC.  UnRAID is a software RAID, so all of those reads/writes have to be done by the PC, and the data passed to/from the disks via the PCIe bus interface.

 

On the card itself, you have a bunch of 6Gb/s SATA ports ... and each of those ports can utilize the full bandwidth without regard to any usage on the PCIe bus, since the data flow is all "local".  On disks that can do 200MB/s of transfer (only the fastest drives, and then only on the outer cylinders), that's enough bandwidth to handle ~ 4 disks at a time at full bandwidth.      With 16 ports, that's 16x4 = 64 drives that could be in an array and run at full speed for reads/writes to the array.  I'd say that arrays that large (64 drives) or larger are indeed "pushing the limits" of the card ... although in most cases the array could be notably larger, since the average drive is certainly not hitting 200MB/s of transfer speed.

 

Note that the RAID <-> PC transfer in those case will still be "limited" by the 4GB/s bandwidth of the x8 PCIe bus, but that's a VERY good transfer speed !!  I can't image it being a "bottleneck" in many cases !!

 

[brian89gp]

It is a software RAID SAN, the LSI controllers are just SAS HBA's.

 

Perhaps its just the method that unRAID works, eg the parity check uses a lot of bandwidth across all drives at the same time, not something that is typical in normal RAID systems except during a rebuild?

[garycase]

It is a software RAID SAN, the LSI controllers are just SAS HBA's.

 

Perhaps its just the method that unRAID works, eg the parity check uses a lot of bandwidth across all drives at the same time, not something that is typical in normal RAID systems except during a rebuild?

 

Ahh, didn't realize it wasn't a hardware-based RAID.  A parity check shouldn't use any more bandwidth than a RAID-5 scrub operation ... both require reading all of the drives, and they have the same number of drives.    In any event, I have to agree with you that it doesn't seem like it should be a bottleneck ==> with UnRAID's 24 drive max this should have plenty of bandwidth for all drives to operate at max transfer rates.

[c3]

It is a software RAID SAN, the LSI controllers are just SAS HBA's.

 

Perhaps its just the method that unRAID works, eg the parity check uses a lot of bandwidth across all drives at the same time, not something that is typical in normal RAID systems except during a rebuild?

 

For many of the same reasons as unRAID, most large scale SAN/NAS are software RAID. The typical recommended limit for a loop is about ~100 drives, allowing for customers to accept below perfect performance for lower $/TB configurations. Notable exception is loops supporting SSDs, when the balance shifts to pure IO/sec and hardware costs skyrocket.

[c3]

Ahh, didn't realize it wasn't a hardware-based RAID.  A parity check shouldn't use any more bandwidth than a RAID-5 scrub operation ... both require reading all of the drives, and they have the same number of drives.    In any event, I have to agree with you that it doesn't seem like it should be a bottleneck ==> with UnRAID's 24 drive max this should have plenty of bandwidth for all drives to operate at max transfer rates.

 

You would not want data scrubs to run at maximum speed to minimize impacting operational workload.

 

There are test results of using a SAS expander for all 24 drives in the Atlas thread. He switched from 3x SAS to 1x SAS.