HBA/SAS/SATA Controller and SAS Expanders


Recommended Posts

Firstly I would like to thank all the info provided by johnnie.black, any reference to speeds are taken from his real world tests post here.

 

I'm still trying to wrap my head around the different configurations on how to hook up 24+ drives. I've also noticed a lot of question around using expanders. I've made some terrible diagrams in paint to help visually.

 

I've been hung up on the different ways to connect the controllers to the sas exapnders, and what max speed the resulting drives will have available to them.

 

I'm going to try to have the questions and hopefully the answers universal so it's helpful to all. These examples are how to connect 24 drives. Of course there is also on board ports/controllers, some people may want to add a couple SSDs for cache and increasing the needed amount of ports. Everyone will have their own optimal configuration. Hopefully we can have some discussion about different configurations in this thread.

 

 

Some info on the diagrams bellow:

  • Each black line is a cable.

  • 8 Port controller is actually a 2 port controller with 4 channels/lanes per port for a total of 8 channels/lanes per card. This can be a Dell Perc H310 or IBM M1015 for example. There are others but these seem to be the most popular and been used with the expanders before so they are compatible.

 

 

Connection from the expander or controller (if not using expander) to the hard drives will use a SFF8087 to SATA (forward breakout) cable, therefore allowing 4 drives per expander/controller port.

20111227154239292_large.jpeg?v=1459325897

 

Connection from controller to expander uses SFF8087 to SFF8087 cable

Controller to backplane with built in expander would also use the same SFF8087 to SFF8087 cable

8087-to-8087-300.gif

 

 

 

2 Controllers and 2 Intel RAID SAS Expander RES2SV240 (Shown in Dual Link)

090U52cl.jpg

 

1 Controller and 1 HP 6Gb (3Gb SATA) SAS Expander (Dual Link)

XwWQ3jBl.jpg

 

3 Controllers, Direct Connect

r823L1il.jpg

 

 

1 Controller 2 Intel RAID SAS Expander RES2SV240

Expected but not tested speeds from jonnie.black

using a PCIe 2.0 HBA the bottleneck is the PCIe bus, max speed ~110/125MB/s

using a PCIe 3.0 HBA the bottleneck are the SAS2 links, 2200 * 2 / 24 = 185MB/s

The M1015 and H310 are both PCIe 2.0 x8 cards

2zgB2V6l.jpg

 

2 Controllers 1 HP 6Gb (3Gb SATA) SAS Expander - Not Possible

This configuration is not possible, you can't connect the same expander to more than one controller. - jonnie.black

k1f129Jl.jpg

 

 

Some more info

 

General Expander info.

General Expander info 2

HP vs Intel.

Intel RES2SV240 Wiki

Some info on the HP 6Gb (3Gb SATA) SAS Expander. It does require a PCI-E x4 port for power only. I do suspect that you could use one of following type of adapters but I haven't seen anyone confirm that it does work.

Type 1

Type 2

Type 3

Type 4

 

 

Now I know some cases have expanders built in, I don't have one (yet) so I don't know a lot about them yet, as far as dual link, single link, speed etc.

 

I would like to add some info/discussion on disk speed and what is considered too much of a bottle neck for current drives, future drives and how much the speed decrease will effect unraid usage.

 

With most people using gigabit lan which has a theoretical max of 125 MB/s and a real world max close to that. Any disk speed over 125MB/s won't make any difference writing to the array. Only benefit of speeds over 125MB/s is decreased time for parity checks.

 

 

Will add more info as I find it and hopefully some of the smarter people start to chime in.

  • Like 2
  • Upvote 1
Link to comment

Configurations listed in order of speed

 

3 Controllers, Direct Connect                                     - 320MB/s         - 3 PCIe x8 slots

2 Controllers and 2 Intel Expanders (dual link)             - 205MB/s         - 2 PCIe x8 slots

2 Controllers and 2 HP Expanders   (dual link)             - 182MB/s          - 2 PCIe x8 slots

 

2 Controllers and 1 Expander, HP or Intel

   - 16 drives connected through expander (dual link)    - 140MB/s          - 2 PCIe x8 slots

   - 8 drives via breakout cable                                    - 320MB/s

 

1 Controller and 2 Intel Expanders                               - 110/125MB/s    - 1 PCIe x8 slots

 

1 Controllers and 1 Intel Expander

   - 20 drives connected through expander (single link) - 110MB/s          - 1 PCIe x8 slots

   - 4 drives via breakout cable                                     - 320MB/s

 

1 Controller and 1 HP Expander (dual link)                     - 95MB/s           - 1 PCIe x8 slot and 1 PCIe x4 slots for power but could possibly be powered externally/from adapter

 

 

Possible bottlenecks:

Expander

PCIe bus

DMI bus

 

 

Maximum Theoretical Speed vs "Real World"

 

PCIe x8 2.0

4000MB/s maximum theoretical speed, johnnie.black tests found that the max "real world" speed is between 2600 and 3000MB/s.

 

4port SAS2 link

2400MB/s (4x 600MB/s) max theoretical bandwidth real world speed is ~2200MB/s

 

 

The IBM M1015 is PCIe 2.0

The Dell H310  is PCIe 2.0

Intel RAID SAS Expander RES2SV240 (4000MB/s max theoretical speed, 2600 and 3000MB/s real world)

HP 3Gb SATA (6Gb SAS) Expander (2400MB/s max theoretical speed, 2200MB/s real world)(running SATA, didn't include SAS speed due to little chance anyone here will use SAS)

 

 

3 Controllers, Direct Connect            - 320MB/s        - No PCIe or expander bottleneck possible DMI bus bottleneck

2 Controllers and 2 Intel Expanders - 205MB/s        - PCIe 2.0 bus is the bottleneck (12 drives on one PCIe slot)

1 Controller  and 2 Intel Expanders  - 110/125MB/s - PCIe 2.0 bus is the bottleneck (24 drives on one PCIe slot)

1 Controller  and 1 HP Expander      - 95MB/s          - Expander is the bottleneck

 

 

So a single card with 2 4port SAS2 links would be able to provide connection of 24 drives at 185MB/s with the use of an expander if no bottlenecks are present. Each of the of the ports of the HBA will need to connect to 12 drives via an expander. The HBA would need a total 4400MB/s of bandwidth to run at max speeds. But the PCIe 2.0 x8 bus max speed is between:

2600MB/s /24 = 108MB/s per disk

3000MB/s /24 = 125MB/s per disk

 

So the max speed per disk on a single PCIe x8 2.0 slot is 108-125MB/s

 

 

 

On 8/5/2016 at 4:18 AM, johnnie.black said:

A 4port SAS2 link has 2400MB/s (4x 600MB/s) max theoretical bandwidth, in my tests I've found real world speed is ~2200MB/s, so:

 

2200 / 12 = 185MB/s

 

Quote

You main bottleneck is DMI, which in your board is first gen, up to 1000MB/s, all 6 onboard ports + how many drives are in the bottom pcie x16(x4) controller have to share it, still should give you more than 15MB/s, even with 14 disks going through it it should be roughly 55Mb/s, you may be needing some tuning.

 

Speed won't improve by replacing the controllers, the most you can do with that board is to connect only 6 to the bottom controller plus the 6 onboard, with some tuning rebuilds should start at ~65MB/s.

 

The SASLP with no other bottlenecks is not the fastest card but it's capable of 80MB/s fully loaded.

 

 

Edited by bnevets27
  • Like 1
Link to comment

Make and model of all suitable LSI2008 chipset HBA's

 

LSI 9211-8i

IBM M1015

Dell Perc H310

Dell Perc H200

 

Please respond with any other variations that are possible.

 

Any more suitable expanders would also be helpful.

 

 

DMI bus bottleneck

 

This will be very different for each machine. No one size fits all here. But best to refrence your motherboard manual and johnnie.blacks post here.

Link to comment

1 Controller 2 Intel RAID SAS Expander RES2SV240

2zgB2V6l.jpg

 

Never tested as I only have 1 Intel expander but I would expect these speeds:

 

using a PCIe 2.0 HBA the bottleneck is the PCIe bus, max speed ~110/125MB/s

 

using a PCIe 3.0 HBA the bottleneck are the SAS2 links, 2200 * 2 / 24 = 185MB/s

 

 

 

 

2 Controllers 1 HP 6Gb (3Gb SATA) SAS Expander

k1f129Jl.jpg

 

This configuration is not possible, you can't connect the same expander to more than one controller.

Link to comment
  • 2 years later...

Hello,

 

I am trying to figure out how to get the most out of my 14 HDD drives, I do however have a 24HDD suitable case. However, how would i connect 24 drives to get the most out of it. When i use regular HDDs (Wd reds) is the HBA or the Expander the limiting factor?

Would this be possible ? And what would the speeds be? Untitled.jpg.985f8e1fd634affd85595acc993d569c.jpg

 

Is there a good 16port controller out there so I could just skip the expander?

Edited by makarai
Link to comment
31 minutes ago, makarai said:

to go for the 12gb adapters tho, mostlikely i'll only use hdds?

It does for 16 ports if you don't want to risk a bottleneck, not because it's 6gbps but because it's PCIe 2.0, though for WD Reds up to 6TB it shouldn't be much of one, but it can be for faster disks.

Link to comment
  • 7 months later...
On 8/13/2016 at 2:58 AM, johnnie.black said:

Never tested as I only have 1 Intel expander but I would expect these speeds:

 

using a PCIe 2.0 HBA the bottleneck is the PCIe bus, max speed ~110/125MB/s

 

I am using a PCIe 3.0 HBA on a PCIe 2.0 server, connected to a SAS1 expander (hp dl180 g6).

I can speed test at ~205-210 MB/s on a single drive, but transferring disk to to disk its limited to 85-90 MB/s.

 

Does this make sense? 

Link to comment
8 hours ago, johnnie.black said:

Not enough info, is this array disk to array disk? With parity? Turbo write enable?

Array disk to array disk

8TB Red to 8TB Red

No parity

Turbo write enabled

 

HP H220 HBA (SAS2308) -> SINGLE SFF-8087 Cable -> HP 12 Drive Backplane with integrated expander.

 

 

 

Interestingly - While using unBALANCE to move files from drive to drive - at ~81 MB/s (lots of smaller files) I am able to speed test one of the other drives in the array at 200+ MB/s with only slight impact on performance. Slight speed hit to the drive being tested, but the 2 drives that are doing a transfer aren't really affected in performance.

 

Edit: My model shows this in the specs

12HDD Models

 

HP Smart Array P212/256MB Controller (RAID 0/1/1+0/5/5+0)
NOTE: Available upgrades: P410 with FBWC, 256MB with BBWC, 512MB with FBWC, Battery kit upgrade (for the 256MB cache), and Smart Array Advanced Pack (SAAP).
NOTE: Support transfer rate up to 3Gb/s SAS or 3Gb/s SATA

 

I am using an HP H220 HBA capable of 6Gb/s sas or 3Gb/s SATA - but It appears the 12 drive backplane will only negotiate 3gb/s SAS.

 

The P212 supports 6Gb/s SAS so I assume its the expander/backplane that does not.

 

Edit2: hba connected at single link not dual 

Edited by squirrelslikenuts
Link to comment
On 1/12/2020 at 11:54 AM, squirrelslikenuts said:

... While using unBALANCE to move files from drive to drive - at ~81 MB/s (lots of smaller files) ...

Bad way to test for bandwidth limits/bottlenecks.

=====

Fud's First Law: "If you push something hard enough, it will fall over."

 

Link to comment
  • 8 months later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.