Jump to content
johnnie.black

Sata/SAS controllers tested (real world max throughput during parity check)

29 posts in this topic Last Reply

Recommended Posts

I had the opportunity to test the “real word” bandwidth of some commonly used controllers in the community, so I’m posting my results in the hopes that it may help some users choose a controller and others understand what may be limiting their parity check/sync speed.

 

Note that these tests are only relevant for those operations, normal read/writes to the array are usually limited by hard disk or network speed.

 

Next to each controller is its maximum theoretical throughput and my results depending on the number of disks connected, result is observed parity check speed using a fast SSD only array with Unraid V6.1.2 (SASLP and SAS2LP tested with V6.1.4 due to performance gains compared with earlier releases)

 

Values in green are the measured controller power consumption with all ports in use.

 

2 Port Controllers

 

SIL 3132 PCIe gen1 x1 (250MB/s)

1 x 125MB/s

2 x 80MB/s

 

Asmedia ASM1061 PCIe gen2 x1 (500MB/s) - e.g., SYBA SY-PEX40039 and other similar cards

1 x 375MB/s

2 x 206MB/s

 

JMicron JMB582 PCIe gen3 x1 (985MB/s) - e.g., SYBA SI-PEX40148 and other similar cards

1 x 570MB/s

2 x 450MB/s

 

 

4/5 Port Controllers

 

SIL 3114 PCI (133MB/s)

1 x 105MB/s

2 x 63.5MB/s

3 x 42.5MB/s

4 x 32MB/s

 

Adaptec AAR-1430SA PCIe gen1 x4 (1000MB/s)

4 x 210MB/s

 

Marvell 9215 PCIe gen2 x1 (500MB/s) - 2w - e.g., SYBA SI-PEX40064 and other similar cards (possible issues with virtualization)

2 x 200MB/s

3 x 140MB/s

4 x 100MB/s

 

Marvell 9230 PCIe gen2 x2 (1000MB/s) - 2w - e.g., SYBA SI-PEX40057 and other similar cards (possible issues with virtualization)

2 x 375MB/s

3 x 255MB/s

4 x 204MB/s

 

IBM H1110 PCIe gen2 x4 (2000MB/s) - LSI 2004 chipset, results should be the same as for an LSI 9211-4i and other similar controllers

2 x 570MB/s

3 x 500MB/s

4 x 375MB/s

 

JMicron JMB585 PCIe gen3 x2 (1970MB/s) - e.g., SYBA SI-PEX40139 and other similar cards

2 x 570MB/s

3 x 565MB/s

4 x 440MB/s

5 x 350MB/s

 

8 Port Controllers

 

Supermicro AOC-SAT2-MV8 PCI-X (1067MB/s)

4 x 220MB/s (167MB/s*)

5 x 177.5MB/s (135MB/s*)

6 x 147.5MB/s (115MB/s*)

7 x 127MB/s (97MB/s*)

8 x 112MB/s (84MB/s*)

 

*on PCI-X 100Mhz slot (800MB/S)

 

Supermicro AOC-SASLP-MV8 PCIe gen1 x4 (1000MB/s) - 6w

4 x 140MB/s

5 x 117MB/s

6 x 105MB/s

7 x 90MB/s

8 x 80MB/s

 

Supermicro AOC-SAS2LP-MV8 PCIe gen2 x8 (4000MB/s) - 6w

4 x 340MB/s

6 x 345MB/s

8 x 320MB/s (205MB/s*, 200MB/s**)

 

*on PCIe gen2 x4 (2000MB/s)

**on PCIe gen1 x8 (2000MB/s)

 

Dell H310 PCIe gen2 x8 (4000MB/s) - 6w – LSI 2008 chipset, results should be the same as for an LSI 9211-8i and other similar controllers

4 x 455MB/s

6 x 377.5MB/s

8 x 320MB/s (190MB/s*, 185MB/s**)

 

*on PCIe gen2 x4 (2000MB/s)

**on PCIe gen1 x8 (2000MB/s)

 

LSI 9207-8i PCIe gen3 x8 (4800MB/s) - 9w - LSI 2308 chipset

8 x 525MB/s+ (*)

 

LSI 9300-8i PCIe gen3 x8 (4800MB/s with the SATA3 devices used for this test) - LSI 3008 chipset

8 x 525MB/s+ (*)

 

* used SSDs maximum read speed

 

SAS Expanders

 

HP 6Gb (3Gb SATA) SAS Expander - 11w

 

Single Link on Dell H310 (1200MB/s*)

8 x 137.5MB/s

12 x 92.5MB/s

16 x 70MB/s

20 x 55MB/s

24 x 47.5MB/s

 

Dual Link on Dell H310 (2400MB/s*)

12 x 182.5MB/s

16 x 140MB/s

20 x 110MB/s

24 x 95MB/s

 

* Half 6GB bandwidth because it only links @ 3Gb with SATA disks

 

Intel® RAID SAS2 Expander RES2SV240 - 10w

 

Single Link on Dell H310 (2400MB/s)

8 x 275MB/s

12 x 185MB/s

16 x 140MB/s (112MB/s*)

20 x 110MB/s (92MB/s*)

 

Dual Link on Dell H310 (4000MB/s)

12 x 205MB/s

16 x 155MB/s (185MB/s**)

 

Dual Link on LSI 9207-8i (4800MB/s)

16 x 275MB/s

 

LSI SAS3 expander (included on a Supermicro BPN-SAS3-826EL1 backplane)

 

Single Link on LSI 9300-8i (tested with SATA3 devices, max usable bandwidth would be 2200MB/s, but with LSI's Databolt technology we can get almost SAS3 speeds)

8 x 475MB/s

12 x 340MB/s

 

Dual Link on LSI 9300-8i (tested with SATA3 devices, max usable bandwidth would be 4400MB/s, but with LSI's Databolt technology we can get almost SAS3 speeds, limit here is going to be the PCIe 3.0 slot, around 6000MB/s usable)

10 x 510MB/s

12 x 460MB/s

 

 

* Avoid using slower linking speed disks with expanders, as it will bring total speed down, in this example 4 of the SSDs were SATA2, instead of all SATA3.

** Two different boards have consistent different results, will need to test a third one to see what's normal, 155MB/s is the max on a Supermicro X9SCM-F, 185MB/s on Asrock B150M-Pro4S.

 

Sata 2 vs Sata 3

I see many times on the forum users asking if changing to Sata 3 controllers or disks would improve their speed, Sata 2 has enough bandwidth (between 265 and 275MB/s according to my tests) for the fastest disks currently on the market, if buying a new board or controller you should buy sata 3 for the future, but except for SSD use there’s no gain in changing your Sata 2 setup to Sata 3.

 

Single vs. Dual Channel RAM

In arrays with many disks, and especially with low “horsepower” CPUs, memory bandwidth can also have a big effect on parity check speed, obviously this will only make a difference if you’re not hitting a controller bottleneck, two examples with 24 drive arrays:

 

Asus A88X-M PLUS with AMD A4-6300 dual core @ 3.7Ghz

Single Channel – 99.1MB/s

Dual Channel - 132.9MB/s

 

Supermicro X9SCL-F with Intel G1620 dual core @ 2.7Ghz

Single Channel – 131.8MB/s

Dual Channel – 184.0MB/s

 

DMI

There is another bus that can be a bottleneck for Intel based boards, much more so than Sata 2, the DMI that connects the south bridge or PCH to the CPU.

Socket 775, 1156 and 1366 use DMI 1.0, socket 1155, 1150 and 2011 use DMI 2.0, socket 1151 uses DMI 3.0

 

DMI 1.0 (1000MB/s)

4 x 180MB/s

5 x 140MB/s

6 x 120MB/s

8 x 100MB/s

10 x 85MB/s

 

DMI 2.0 (2000MB/s)

4 x 270MB/s (Sata2 limit)

6 x 240MB/s

8 x 195MB/s

9 x 170MB/s

10 x 145MB/s

12 x 115MB/s

14 x 110MB/s

 

DMI 3.0 (3940MB/s)

6 x 330MB/s (Onboard SATA only*)

10 X 297.5MB/s

12 x 250MB/s

16 X 185MB/s

 

*Despite being DMI 3.0, Skylake, Kaby Lake and Coffee Lake chipsets have a max combined bandwidth of approximately 2GB/s for the onboard SATA ports.

 

DMI 1.0 can be a bottleneck using only the onboard Sata ports, DMI 2.0 can limit users with all onboard ports used plus an additional controller onboard or on a PCIe slot that shares the DMI bus, in most home market boards only the graphics slot connects directly to CPU, all other slots go through the DMI (more top of the line boards, usually with SLI support, have at least 2 slots), server boards usually have 2 or 3 slots connected directly to the CPU, you should always use these slots first.

 

You can see below the diagram for my X9SCL-F test server board, for the DMI 2.0 tests I used the 6 onboard ports plus one Adaptec 1430SA on PCIe slot 4.

 

UMI (2000MB/s) - Used on most AMD APUs, equivalent to intel DMI 2.0

6 x 203MB/s

7 x 173MB/s

8 x 152MB/s

 

Ryzen link - PCIe 3.0 x4 (3940MB/s)

6 x 467MB/s (Onboard SATA only)

 

 

 

I think there are no big surprises and most results make sense and are in line with what I expected, exception maybe for the SASLP that should have the same bandwidth of the Adaptec 1430SA and is clearly slower, can limit a parity check with only 4 disks.

 

I expect some variations in the results from other users due to different hardware and/or tunnable settings, but would be surprised if there are big differences, reply here if you can get a significant better speed with a specific controller.

 

 

 

 

How to check and improve your parity check speed

 

System Stats from Dynamix V6 Plugins is usually an easy way to find out if a parity check is bus limited, after the check finishes look at the storage graph, on an unlimited system it should start at a higher speed and gradually slow down as it goes to the disks slower inner tracks, on a limited system the graph will be flat at the beginning or totally flat for a worst-case scenario. See screenshots below for examples (arrays with mixed disk sizes will have speed jumps at the end of each one, but principle is the same).

 

If you are not bus limited but still find your speed low, there’s a couple things worth trying:

 

Diskspeed - your parity check speed can’t be faster than your slowest disk, a big advantage of Unraid is the possibility to mix different size disks, but this can lead to have an assortment of disk models and sizes, use this to find your slowest disks and when it’s time to upgrade replace these first.

 

Tunables Tester - on some systems can increase the average speed 10 to 20Mb/s or more, on others makes little or no difference.

 

 

That’s all I can think of, all suggestions welcome.

x9scl-f.png.fc4e744b1f64de82f2c8e7c417b03e87.png

speed_unlimited.png.40b2fa9c53917183be317747dc492192.png

speed_limited.png.5e1bb1bb2533cd98746723a1ef915c52.png

  • Like 1
  • Thanks 2

Share this post


Link to post
Supermicro AOC-SAS2LP-MV8 PCIe gen2 8x (4000MB/s)

There’s currently an issue with Unraid V6, will update if/when fixed.

 

I'm running this card on v6.0.1 without any issues at all?  Was just about to upgrade to v6.1.3.  Given your statement, should I wait to upgrade?

Share this post


Link to post

I'm running this card on v6.0.1 without any issues at all?  Was just about to upgrade to v6.1.3.  Given your statement, should I wait to upgrade?

 

There was an issue with any V6 and parity check performance, although it didn’t affect all users, it was fixed in 6.1.4, will post test results soon.

Share this post


Link to post

 

2 Port Controllers

 

SIL 3132 PCIe gen1 1x (250MB/s)

2 x 80MB/s

Thanks Johnnie, that explains my pcheck limitation on my boxes - as I have 2xLSI2008, 6 onboard plus this PCIe card.

It seems, there is no better alternative for PCIex1/2-port available, right? (Don't have a 3rd slot for 3rd LSI card available either...)

Share this post


Link to post

Thanks Johnnie, that explains my pcheck limitation on my boxes - as I have 2xLSI2008, 6 onboard plus this PCIe card.

It seems, there is no better alternative for PCIex1/2-port available, right? (Don't have a 3rd slot for 3rd LSI card available either...)

 

For 2 ports I would recommend Asmedia 1601 based controllers, much better performance and they are cheap, like this one:

 

http://www.newegg.com/Product/Product.aspx?Item=N82E16816124045

 

The one I used for testing was onboard but performance should be very similar.

 

Share this post


Link to post

Thanks for the hint - indeed, that is a cheap and obviously better solution than the sil3132 based cards.

Share this post


Link to post

I know this post is old.  Resurrecting a dead thread.  But this thread has me rethinking the whole server I am currently building.

 

Question I have:  Speed of the Expanders is directly related to the number of drives actually hooked up currently and running, and not just ports correct?  So if I had all 20 ports connected to empty SATA slots, but only 8 SATA drives hooked up, I would achieve maximum  throughput for 8 drives if the drives supported it?

Share this post


Link to post

 

@johnnie.black this is a tremendous resource and was interested in your thoughts on what would be the expected speeds to determine the optimal configuration for a for 30+ drives in the following configuration:

 

SuperMicro SuperStorage 6047R-E1R36L configured as:

  • BPN-SAS2-846EL1 (front 24 bays) with three 8087 connectors
  • BPN-SAS2-826EL1 (rear 12 bays) with two 8087 connectors
  • LSI 2308 SAS2 PCIe 3.0 host adapter with two 8087 connectors (motherboard integrated)
  • 6x SATA ports on the motherboard, running at 6GB SATA3 (DMI 2.0)
  • 2x NVMe slots via Supermicro AOC-SLG3-2M2 add-in card
  • The drives are all WD Red/White 10TB & 8TB models

Unraid setup as of 4/27/2019:

  • Data disks x28
    • x24 connected to BPN-SAS2-846EL1 dual linked to LSI 2308
    • x4 connected to motherboard SATA ports
  • Parity disk
    • x1 disk connected to motherboard SATA port
  • cache disk
    • x1 NVMe drive connected to AOC-SLG-2M2
  • Not in use/connected
    • BPN-SAS2-826EL1 (rear 12 bays)
    • x1 SATA port
    • x1 NVMe port

 

Is this the most optimal setup to spread out the utilization and not encounter bottlenecks? Any opportunities for improvement?

 

If I was to daisy chain a single link from the third 8087 connector on the BPN-SAS2-846EL1 to the downstream BPN-SAS2-826EL1 how much of a negative impact on performance would that have? The 826-EL1 would be empty, no disks. Would that reduce PHYs available or add significant additional SAS overhead?

 

 

 

 

Share this post


Link to post
9 hours ago, Lev said:

Is this the most optimal setup to spread out the utilization and not encounter bottlenecks? Any opportunities for improvement?

Looks optimal to me, max bandwidth is enough (or close enough) for the the current number of disks, and their max speed.

 

9 hours ago, Lev said:

If I was to daisy chain a single link from the third 8087 connector on the BPN-SAS2-846EL1 to the downstream BPN-SAS2-826EL1 how much of a negative impact on performance would that have? The 826-EL1 would be empty, no disks.

No impact without disks, with disks it would share the bandwidth of the other link, so if in the future you use that backplane for array disks it would be better to connect it to a different HBA.

Share this post


Link to post

 

@johnnie.black Great info here.

 

I have a server I built around the ASRock E3C224D4I-14S motherboard that comes with dual (2x) integrated LSI 2308 mini SAS connectors that are connected to (8x) drives. I want to use this server as a DAS box.

 

I have 2x R710 I just acquired, what would be better/faster to connect a R710 to the ASRock server as a DAS?

 

LSI 9207-8e inside of the R710 -> SFF-8088 to SFF-8088 -> SFF-8088 to SFF-8087 PCI bracket adapter -> SFF-8087 to SATA 4x

or

LSI 9207-8e inside of the R710 -> SFF-8088 to SFF-8088 -> RES2SV240 -> (2x) SFF-8088 to (2x) LSI 2308 -> SFF-8087 to SATA 4x

 

 

Looking to see what would give me the most speed and/or power consumption. Assuming I use all SSD's.

Share this post


Link to post

Sorry, not sure I understand what you're trying to do, do you mean use the previous Asrock based server without a mother board? You can't connect one HBA on the Dell to the Asrock HBA.

 

 

Share this post


Link to post
5 hours ago, johnnie.black said:

Sorry, not sure I understand what you're trying to do, do you mean use the previous Asrock based server without a mother board? You can't connect one HBA on the Dell to the Asrock HBA.

 

 

Oh, seems like I don't know what I'm doing. I was trying to use my existing ASRock server as a storage expansion (DAS i think) to my SFF Dell R710. How would I connect them? Looking to get the highest speed possible. Thank you for your help.

Share this post


Link to post
1 minute ago, Omar said:

Oh, seems like I don't know what I'm doing. I was trying to use my existing ASRock server as a storage expansion (DAS i think) to my SFF Dell R710.

For that you'd need to remove (or don't use) the Asrock board, you could connect the HBA on the Dell directly to the disks, but keep in mind max total cable length from HBA to disks is 1 meter (assuming SATA disks), or you could use an expander and with that HBA, HBA to expander cable can be 10 meters long (though you should use one a small as possible), then for expander to disks 1m should be plenty, similar to this:

 

1915971291_2017-08-2413_30.07-Cpia.thumb.jpg.6f7035e589e2a3312a4d94b261a48a27.jpg

Share this post


Link to post
17 minutes ago, johnnie.black said:

For that you'd need to remove (or don't use) the Asrock board, you could connect the HBA on the Dell directly to the disks, but keep in mind max total cable length from HBA to disks is 1 meter (assuming SATA disks), or you could use an expander and with that HBA, HBA to expander cable can be 10 meters long (though you should use one a small as possible), then for expander to disks 1m should be plenty, similar to this:

 

1915971291_2017-08-2413_30.07-Cpia.thumb.jpg.6f7035e589e2a3312a4d94b261a48a27.jpg

Would adding an expander card slow down the HBA or is there no loss? 

From your speed test it seems like the LSI 9207-8e and RES2SV240 would be a good combo.

Would you recommend something better? I want to get the most speed possible when using (8x) SSD drives. 

Share this post


Link to post
7 minutes ago, Omar said:

Would adding an expander card slow down the HBA or is there no loss? 

The expander by itself doesn't create a bottleneck, but it does allow you to connect more devices instead of the 8 max you can use without one, with more devices same HBA bandwidth will be shared by all, so that could introduce a bottleneck.

 

8 minutes ago, Omar said:

From your speed test it seems like the LSI 9207-8e and RES2SV240 would be a good combo.

It is, and enough bandwidth for 8 SSDs.

 

 

 

Share this post


Link to post
11 minutes ago, johnnie.black said:

It is, and enough bandwidth for 8 SSDs.

Wouldn't the max throughput be (slightly) limited by the PCIe Gen2 of the R710? [ie, to ~400 MB/s each for 8xSata3-SSDs]

 

Share this post


Link to post
1 minute ago, UhClem said:

Wouldn't the max throughput be (slightly) limited by the PCIe Gen2 of the R710?

If the R710 is PCIe 2.0 then yes, might as well get a 9211-8i or similar.

Share this post


Link to post

Awesome thank you, time to upgrade my newly acquired R710? 

 

How do you calculate max throughput, bandwidth, theoretical speed?

Share this post


Link to post

I'm loving this topic. I'm running the equivalent of the Norco 4224 and am looking at freeing up some pci-e slots on my X10-SRI-F motherboard

 

I currently have 24 disks (old spinners) all connected to 3 x Dell H200 HBA's (Cross flashed to IT Mode) which gives me maximum throughput. But it takes up so many pci-e slots! I currently have the following on my board but was hoping to be able to get rid of 2 x Dell H200's with a solid 24 ports expander...

 

1 x PCI-e Gen3 (x16) - Dell H200 #1

1 x PCI-e Gen3 (x8) - Dell H200 #2

1 x PCI-e Gen3 (x8) - Dell H200 #3

1 x PCI-e Gen3 (x4) - Intel X540-T2 

1 x PCI-e Gen2 (x4)

1 x PCI-e Gen2 (x1)

 

I've been looking today at replacing 2 of the HBA's with an expander, but the HP SAS Expander (limited to 3gb/s for SATA) Looks to be a bit of a slouch and the Intel RES2SV240 can only support 20 drives! 

 

Would either of these configs be possible utilizing 2 x RES2SV24 as they seem quite cheap right now and i can power them from my 2 x pcie 2.0 slots without sacrificing any pci-e 3.0 slots?

 

Dell H200 HBA Card 1 --> Link 1 --> 12 ports on Intel SAS expander RES2SV240 card (1) 

                                 --> Link 2 --> 12 ports on Intel SAS expander RES2SV240 card (2)

 

OR

 

Dell H200 HBA Card 1 --> Link 1+2--> 12 ports on Intel SAS expander RES2SV240 card 1) 

Dell H200 HBA Card 2 --> Link 1+2 --> 12 ports on Intel SAS expander RES2SV240 card (2) 

 

??

 

What would you guys say is the top end read/write speed of the new 10/12/14TB disks? I'm thinking if i can offer up around 185mb/s for each of my 24 spinner disks that's likely to be pretty optimal....

 

??

Edited by thestraycat

Share this post


Link to post
15 minutes ago, thestraycat said:

Would either of these configs be possible utilizing 2 x RES2SV24 as they seem quite cheap right now and i can power them from my 2 x pcie 2.0 slots without sacrificing any pci-e 3.0 slots?

Both are possible, there's also a 36 port Intel SAS2 expander (RESCV360JBD) but it's usually more expensive.

 

17 minutes ago, thestraycat said:

What would you guys say is the top end read/write speed of the new 10/12/14TB disks?

Top speed in the outer sectors range from around 225MB/s for a 10TB 5400rpm drive to around 275MB/s for 14TB 7200rpm one.

 

Share this post


Link to post
How do you calculate max throughput, bandwidth, theoretical speed?

Mostly HBA bus width and link speed, for SAS expanders number of wide ports in use and devices used link speed.

 

 

Share this post


Link to post
23 hours ago, johnnie.black said:

If the R710 is PCIe 2.0 then yes, might as well get a 9211-8i or similar.

Maybe ... maybe not. Consider that the 9211-8i (and its LSI SAS2008-based brethren) appear to have an ultimate bottleneck of their internal CPU/memory/firmware which precludes them achieving the maximum (real-world/measurable) PCIe Gen2 x8 throughput of 3200 MB/s. Your tests, on the H310, show 2560 (8x320) MB/s.

 

Whereas the 9207-8 (and its SAS2308 brethren) do have the "muscle-power" to achieve, and exceed, full  Gen2x8 throughput (3200); you measured, on a Gen3x8, 4200 (8x525) MB/s--and that is limited, not by the 2308, but by maxing out the Sata3 (real-world) speed of ~525 [I suspect that SAS-12Gb SSDs would do better, no?***] So, for 9211 vs 9207, using Gen2, it comes down to a cost/benefit decision (plus, an appropriate degree of future-proof factor; re-use following a mobo upgrade).

 

*** [Edit] Actually, NO -- it looks like 12G SAS connectivity was not offered until the 93xx series.

 

Edited by UhClem
Addendum ***

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.