Sata/SAS controllers tested (real world max throughput during parity check)


Recommended Posts

I had the opportunity to test the “real word” bandwidth of some commonly used controllers in the community, so I’m posting my results in the hopes that it may help some users choose a controller and others understand what may be limiting their parity check/sync speed.

 

Note that these tests are only relevant for those operations, normal read/writes to the array are usually limited by hard disk or network speed.

 

Next to each controller is its maximum theoretical throughput and my results depending on the number of disks connected, result is observed parity/read check speed using a fast SSD only array with Unraid V6

 

Values in green are the measured controller power consumption with all ports in use.

 

2 Port Controllers

 

SIL 3132 PCIe gen1 x1 (250MB/s)

1 x 125MB/s

2 x 80MB/s

 

Asmedia ASM1061 PCIe gen2 x1 (500MB/s) - e.g., SYBA SY-PEX40039 and other similar cards

1 x 375MB/s

2 x 206MB/s

 

JMicron JMB582 PCIe gen3 x1 (985MB/s) - e.g., SYBA SI-PEX40148 and other similar cards

1 x 570MB/s

2 x 450MB/s

 

 

4 Port Controllers

 

SIL 3114 PCI (133MB/s)

1 x 105MB/s

2 x 63.5MB/s

3 x 42.5MB/s

4 x 32MB/s

 

Adaptec AAR-1430SA PCIe gen1 x4 (1000MB/s)

4 x 210MB/s

 

Marvell 9215 PCIe gen2 x1 (500MB/s) - 2w - e.g., SYBA SI-PEX40064 and other similar cards (possible issues with virtualization)

2 x 200MB/s

3 x 140MB/s

4 x 100MB/s

 

Marvell 9230 PCIe gen2 x2 (1000MB/s) - 2w - e.g., SYBA SI-PEX40057 and other similar cards (possible issues with virtualization)

2 x 375MB/s

3 x 255MB/s

4 x 204MB/s

 

IBM H1110 PCIe gen2 x4 (2000MB/s) - LSI 2004 chipset, results should be the same as for an LSI 9211-4i and other similar controllers

2 x 570MB/s

3 x 500MB/s

4 x 375MB/s

 

Asmedia ASM1064 PCIe gen3 x1 (985MB/s) - e.g., SYBA SI-PEX40156 and other similar cards

2 x 450MB/s

3 x 300MB/s

4 x 225MB/s

 

Asmedia ASM1164 PCIe gen3 x2 (1970MB/s) - NOTE - not actually tested, performance inferred from the ASM1166 with up to 4 devices

2 x 565MB/s

3 x 565MB/s

4 x 445MB/s

 

 

5 and 6 Port Controllers

 

JMicron JMB585 PCIe gen3 x2 (1970MB/s) - 2w - e.g., SYBA SI-PEX40139 and other similar cards

2 x 570MB/s

3 x 565MB/s

4 x 440MB/s

5 x 350MB/s

 

Asmedia ASM1166 PCIe gen3 x2 (1970MB/s) - 2w

2 x 565MB/s

3 x 565MB/s

4 x 445MB/s

5 x 355MB/s

6 x 300MB/s

 

 

8 Port Controllers

 

Supermicro AOC-SAT2-MV8 PCI-X (1067MB/s)

4 x 220MB/s (167MB/s*)

5 x 177.5MB/s (135MB/s*)

6 x 147.5MB/s (115MB/s*)

7 x 127MB/s (97MB/s*)

8 x 112MB/s (84MB/s*)

 

* PCI-X 100Mhz slot (800MB/S)

 

Supermicro AOC-SASLP-MV8 PCIe gen1 x4 (1000MB/s) - 6w

4 x 140MB/s

5 x 117MB/s

6 x 105MB/s

7 x 90MB/s

8 x 80MB/s

 

Supermicro AOC-SAS2LP-MV8 PCIe gen2 x8 (4000MB/s) - 6w

4 x 340MB/s

6 x 345MB/s

8 x 320MB/s (205MB/s*, 200MB/s**)

 

* PCIe gen2 x4 (2000MB/s)

** PCIe gen1 x8 (2000MB/s)

 

LSI 9211-8i PCIe gen2 x8 (4000MB/s) - 6w – LSI 2008 chipset

4 x 565MB/s

6 x 465MB/s

8 x 330MB/s (190MB/s*, 185MB/s**)

 

* PCIe gen2 x4 (2000MB/s)

** PCIe gen1 x8 (2000MB/s)

 

LSI 9207-8i PCIe gen3 x8 (4800MB/s) - 9w - LSI 2308 chipset

8 x 565MB/s

 

LSI 9300-8i PCIe gen3 x8 (4800MB/s with the SATA3 devices used for this test) - LSI 3008 chipset

8 x 565MB/s (425MB/s*, 380MB/s**)

 

* PCIe gen3 x4 (3940MB/s)

** PCIe gen2 x8 (4000MB/s)

 

 

 

SAS Expanders

 

HP 6Gb (3Gb SATA) SAS Expander - 11w

 

Single Link with LSI 9211-8i (1200MB/s*)

8 x 137.5MB/s

12 x 92.5MB/s

16 x 70MB/s

20 x 55MB/s

24 x 47.5MB/s

 

Dual Link with LSI 9211-8i (2400MB/s*)

12 x 182.5MB/s

16 x 140MB/s

20 x 110MB/s

24 x 95MB/s

 

* Half 6GB bandwidth because it only links @ 3Gb with SATA disks

 

Intel® SAS2 Expander RES2SV240 - 10w

 

Single Link with LSI 9211-8i (2400MB/s)

8 x 275MB/s

12 x 185MB/s

16 x 140MB/s (112MB/s*)

20 x 110MB/s (92MB/s*)

 

* Avoid using slower linking speed disks with expanders, as it will bring total speed down, in this example 4 of the SSDs were SATA2, instead of all SATA3.

 

Dual Link with LSI 9211-8i (4000MB/s)

12 x 235MB/s

16 x 185MB/s

 

Dual Link with LSI 9207-8i (4800MB/s)

16 x 275MB/s

 

LSI SAS3 expander (included on a Supermicro BPN-SAS3-826EL1 backplane)

 

Single Link with LSI 9300-8i (tested with SATA3 devices, max usable bandwidth would be 2200MB/s, but with LSI's Databolt technology we can get almost SAS3 speeds)

8 x 500MB/s

12 x 340MB/s

 

Dual Link with LSI 9300-8i (*)

10 x 510MB/s

12 x 460MB/s

 

* tested with SATA3 devices, max usable bandwidth would be 4400MB/s, but with LSI's Databolt technology we can closer to SAS3 speeds, with SAS3 devices limit here would be the PCIe link, which should be around 6600-7000MB/s usable.

 

HP 12G SAS3 EXPANDER (761879-001)

 

Single Link with LSI 9300-8i (2400MB/s*)

8 x 270MB/s

12 x 180MB/s

16 x 135MB/s

20 x 110MB/s

24 x 90MB/s

 

Dual Link with LSI 9300-8i (4800MB/s*)

10 x 420MB/s

12 x 360MB/s

16 x 270MB/s

20 x 220MB/s

24 x 180MB/s

 

* tested with SATA3 devices, no Databolt or equivalent technology, at least not with an LSI HBA, with SAS3 devices limit here would be the around 4400MB/s with single link, and the PCIe slot with dual link, which should be around 6600-7000MB/s usable.

 

Intel® SAS3 Expander RES3TV360

 

Single Link with LSI 9308-8i (*)

8 x 490MB/s

12 x 330MB/s

16 x 245MB/s

20 x 170MB/s

24 x 130MB/s

28 x 105MB/s

 

Dual Link with LSI 9308-8i (*)

12 x 505MB/s

16 x 380MB/s

20 x 300MB/s

24 x 230MB/s

28 x 195MB/s

 

* tested with SATA3 devices, PMC expander chip includes similar functionality to LSI's Databolt, with SAS3 devices limit here would be the around 4400MB/s with single link, and the PCIe slot with dual link, which should be around 6600-7000MB/s usable.

Note: these results were after updating the expander firmware to latest available at this time (B057), it was noticeably slower with the older firmware that came with it.

 

 

Sata 2 vs Sata 3

I see many times on the forum users asking if changing to Sata 3 controllers or disks would improve their speed, Sata 2 has enough bandwidth (between 265 and 275MB/s according to my tests) for the fastest disks currently on the market, if buying a new board or controller you should buy sata 3 for the future, but except for SSD use there’s no gain in changing your Sata 2 setup to Sata 3.

 

Single vs. Dual Channel RAM

In arrays with many disks, and especially with low “horsepower” CPUs, memory bandwidth can also have a big effect on parity check speed, obviously this will only make a difference if you’re not hitting a controller bottleneck, two examples with 24 drive arrays:

 

Asus A88X-M PLUS with AMD A4-6300 dual core @ 3.7Ghz

Single Channel – 99.1MB/s

Dual Channel - 132.9MB/s

 

Supermicro X9SCL-F with Intel G1620 dual core @ 2.7Ghz

Single Channel – 131.8MB/s

Dual Channel – 184.0MB/s

 

DMI

There is another bus that can be a bottleneck for Intel based boards, much more so than Sata 2, the DMI that connects the south bridge or PCH to the CPU.

Socket 775, 1156 and 1366 use DMI 1.0, socket 1155, 1150 and 2011 use DMI 2.0, socket 1151 uses DMI 3.0

 

DMI 1.0 (1000MB/s)

4 x 180MB/s

5 x 140MB/s

6 x 120MB/s

8 x 100MB/s

10 x 85MB/s

 

DMI 2.0 (2000MB/s)

4 x 270MB/s (Sata2 limit)

6 x 240MB/s

8 x 195MB/s

9 x 170MB/s

10 x 145MB/s

12 x 115MB/s

14 x 110MB/s

 

DMI 3.0 (3940MB/s)

6 x 330MB/s (Onboard SATA only*)

10 X 297.5MB/s

12 x 250MB/s

16 X 185MB/s

 

*Despite being DMI 3.0** , Skylake, Kaby Lake, Coffee Lake, Comet Lake and Alder Lake chipsets have a max combined bandwidth of approximately 2GB/s for the onboard SATA ports.

 

**Except low end H110 and H310 chipsets which are only DMI 2.0, Z690 is DMI 4.0 and not yet tested by me, but except same result as the other Alder Lake chipsets.

 

DMI 1.0 can be a bottleneck using only the onboard Sata ports, DMI 2.0 can limit users with all onboard ports used plus an additional controller onboard or on a PCIe slot that shares the DMI bus, in most home market boards only the graphics slot connects directly to CPU, all other slots go through the DMI (more top of the line boards, usually with SLI support, have at least 2 slots), server boards usually have 2 or 3 slots connected directly to the CPU, you should always use these slots first.

 

You can see below the diagram for my X9SCL-F test server board, for the DMI 2.0 tests I used the 6 onboard ports plus one Adaptec 1430SA on PCIe slot 4.

 

UMI (2000MB/s) - Used on most AMD APUs, equivalent to intel DMI 2.0

6 x 203MB/s

7 x 173MB/s

8 x 152MB/s

 

Ryzen link - PCIe 3.0 x4 (3940MB/s)

6 x 467MB/s (Onboard SATA only)

 

 

 

I think there are no big surprises and most results make sense and are in line with what I expected, exception maybe for the SASLP that should have the same bandwidth of the Adaptec 1430SA and is clearly slower, can limit a parity check with only 4 disks.

 

I expect some variations in the results from other users due to different hardware and/or tunnable settings, but would be surprised if there are big differences, reply here if you can get a significant better speed with a specific controller.

 

 

 

 

How to check and improve your parity check speed

 

System Stats from Dynamix V6 Plugins is usually an easy way to find out if a parity check is bus limited, after the check finishes look at the storage graph, on an unlimited system it should start at a higher speed and gradually slow down as it goes to the disks slower inner tracks, on a limited system the graph will be flat at the beginning or totally flat for a worst-case scenario. See screenshots below for examples (arrays with mixed disk sizes will have speed jumps at the end of each one, but principle is the same).

 

If you are not bus limited but still find your speed low, there’s a couple things worth trying:

 

Diskspeed - your parity check speed can’t be faster than your slowest disk, a big advantage of Unraid is the possibility to mix different size disks, but this can lead to have an assortment of disk models and sizes, use this to find your slowest disks and when it’s time to upgrade replace these first.

 

Tunables Tester - on some systems can increase the average speed 10 to 20Mb/s or more, on others makes little or no difference.

 

 

That’s all I can think of, all suggestions welcome.

x9scl-f.png.fc4e744b1f64de82f2c8e7c417b03e87.png

speed_unlimited.png.40b2fa9c53917183be317747dc492192.png

speed_limited.png.5e1bb1bb2533cd98746723a1ef915c52.png

  • Like 17
  • Thanks 10
  • Upvote 10
Link to comment
  • 2 months later...

I'm running this card on v6.0.1 without any issues at all?  Was just about to upgrade to v6.1.3.  Given your statement, should I wait to upgrade?

 

There was an issue with any V6 and parity check performance, although it didn’t affect all users, it was fixed in 6.1.4, will post test results soon.

Link to comment

 

2 Port Controllers

 

SIL 3132 PCIe gen1 1x (250MB/s)

2 x 80MB/s

Thanks Johnnie, that explains my pcheck limitation on my boxes - as I have 2xLSI2008, 6 onboard plus this PCIe card.

It seems, there is no better alternative for PCIex1/2-port available, right? (Don't have a 3rd slot for 3rd LSI card available either...)

Link to comment

Thanks Johnnie, that explains my pcheck limitation on my boxes - as I have 2xLSI2008, 6 onboard plus this PCIe card.

It seems, there is no better alternative for PCIex1/2-port available, right? (Don't have a 3rd slot for 3rd LSI card available either...)

 

For 2 ports I would recommend Asmedia 1601 based controllers, much better performance and they are cheap, like this one:

 

http://www.newegg.com/Product/Product.aspx?Item=N82E16816124045

 

The one I used for testing was onboard but performance should be very similar.

 

Link to comment
  • 1 year later...

I know this post is old.  Resurrecting a dead thread.  But this thread has me rethinking the whole server I am currently building.

 

Question I have:  Speed of the Expanders is directly related to the number of drives actually hooked up currently and running, and not just ports correct?  So if I had all 20 ports connected to empty SATA slots, but only 8 SATA drives hooked up, I would achieve maximum  throughput for 8 drives if the drives supported it?

Link to comment
  • 2 months later...

 

@johnnie.black this is a tremendous resource and was interested in your thoughts on what would be the expected speeds to determine the optimal configuration for a for 30+ drives in the following configuration:

 

SuperMicro SuperStorage 6047R-E1R36L configured as:

  • BPN-SAS2-846EL1 (front 24 bays) with three 8087 connectors
  • BPN-SAS2-826EL1 (rear 12 bays) with two 8087 connectors
  • LSI 2308 SAS2 PCIe 3.0 host adapter with two 8087 connectors (motherboard integrated)
  • 6x SATA ports on the motherboard, running at 6GB SATA3 (DMI 2.0)
  • 2x NVMe slots via Supermicro AOC-SLG3-2M2 add-in card
  • The drives are all WD Red/White 10TB & 8TB models

Unraid setup as of 4/27/2019:

  • Data disks x28
    • x24 connected to BPN-SAS2-846EL1 dual linked to LSI 2308
    • x4 connected to motherboard SATA ports
  • Parity disk
    • x1 disk connected to motherboard SATA port
  • cache disk
    • x1 NVMe drive connected to AOC-SLG-2M2
  • Not in use/connected
    • BPN-SAS2-826EL1 (rear 12 bays)
    • x1 SATA port
    • x1 NVMe port

 

Is this the most optimal setup to spread out the utilization and not encounter bottlenecks? Any opportunities for improvement?

 

If I was to daisy chain a single link from the third 8087 connector on the BPN-SAS2-846EL1 to the downstream BPN-SAS2-826EL1 how much of a negative impact on performance would that have? The 826-EL1 would be empty, no disks. Would that reduce PHYs available or add significant additional SAS overhead?

 

 

 

 

Link to comment
9 hours ago, Lev said:

Is this the most optimal setup to spread out the utilization and not encounter bottlenecks? Any opportunities for improvement?

Looks optimal to me, max bandwidth is enough (or close enough) for the the current number of disks, and their max speed.

 

9 hours ago, Lev said:

If I was to daisy chain a single link from the third 8087 connector on the BPN-SAS2-846EL1 to the downstream BPN-SAS2-826EL1 how much of a negative impact on performance would that have? The 826-EL1 would be empty, no disks.

No impact without disks, with disks it would share the bandwidth of the other link, so if in the future you use that backplane for array disks it would be better to connect it to a different HBA.

  • Like 1
Link to comment

 

@johnnie.black Great info here.

 

I have a server I built around the ASRock E3C224D4I-14S motherboard that comes with dual (2x) integrated LSI 2308 mini SAS connectors that are connected to (8x) drives. I want to use this server as a DAS box.

 

I have 2x R710 I just acquired, what would be better/faster to connect a R710 to the ASRock server as a DAS?

 

LSI 9207-8e inside of the R710 -> SFF-8088 to SFF-8088 -> SFF-8088 to SFF-8087 PCI bracket adapter -> SFF-8087 to SATA 4x

or

LSI 9207-8e inside of the R710 -> SFF-8088 to SFF-8088 -> RES2SV240 -> (2x) SFF-8088 to (2x) LSI 2308 -> SFF-8087 to SATA 4x

 

 

Looking to see what would give me the most speed and/or power consumption. Assuming I use all SSD's.

Link to comment
5 hours ago, johnnie.black said:

Sorry, not sure I understand what you're trying to do, do you mean use the previous Asrock based server without a mother board? You can't connect one HBA on the Dell to the Asrock HBA.

 

 

Oh, seems like I don't know what I'm doing. I was trying to use my existing ASRock server as a storage expansion (DAS i think) to my SFF Dell R710. How would I connect them? Looking to get the highest speed possible. Thank you for your help.

Link to comment
1 minute ago, Omar said:

Oh, seems like I don't know what I'm doing. I was trying to use my existing ASRock server as a storage expansion (DAS i think) to my SFF Dell R710.

For that you'd need to remove (or don't use) the Asrock board, you could connect the HBA on the Dell directly to the disks, but keep in mind max total cable length from HBA to disks is 1 meter (assuming SATA disks), or you could use an expander and with that HBA, HBA to expander cable can be 10 meters long (though you should use one a small as possible), then for expander to disks 1m should be plenty, similar to this:

 

1915971291_2017-08-2413_30.07-Cpia.thumb.jpg.6f7035e589e2a3312a4d94b261a48a27.jpg

  • Like 1
Link to comment
17 minutes ago, johnnie.black said:

For that you'd need to remove (or don't use) the Asrock board, you could connect the HBA on the Dell directly to the disks, but keep in mind max total cable length from HBA to disks is 1 meter (assuming SATA disks), or you could use an expander and with that HBA, HBA to expander cable can be 10 meters long (though you should use one a small as possible), then for expander to disks 1m should be plenty, similar to this:

 

1915971291_2017-08-2413_30.07-Cpia.thumb.jpg.6f7035e589e2a3312a4d94b261a48a27.jpg

Would adding an expander card slow down the HBA or is there no loss? 

From your speed test it seems like the LSI 9207-8e and RES2SV240 would be a good combo.

Would you recommend something better? I want to get the most speed possible when using (8x) SSD drives. 

Link to comment
7 minutes ago, Omar said:

Would adding an expander card slow down the HBA or is there no loss? 

The expander by itself doesn't create a bottleneck, but it does allow you to connect more devices instead of the 8 max you can use without one, with more devices same HBA bandwidth will be shared by all, so that could introduce a bottleneck.

 

8 minutes ago, Omar said:

From your speed test it seems like the LSI 9207-8e and RES2SV240 would be a good combo.

It is, and enough bandwidth for 8 SSDs.

 

 

 

Link to comment

I'm loving this topic. I'm running the equivalent of the Norco 4224 and am looking at freeing up some pci-e slots on my X10-SRI-F motherboard

 

I currently have 24 disks (old spinners) all connected to 3 x Dell H200 HBA's (Cross flashed to IT Mode) which gives me maximum throughput. But it takes up so many pci-e slots! I currently have the following on my board but was hoping to be able to get rid of 2 x Dell H200's with a solid 24 ports expander...

 

1 x PCI-e Gen3 (x16) - Dell H200 #1

1 x PCI-e Gen3 (x8) - Dell H200 #2

1 x PCI-e Gen3 (x8) - Dell H200 #3

1 x PCI-e Gen3 (x4) - Intel X540-T2 

1 x PCI-e Gen2 (x4)

1 x PCI-e Gen2 (x1)

 

I've been looking today at replacing 2 of the HBA's with an expander, but the HP SAS Expander (limited to 3gb/s for SATA) Looks to be a bit of a slouch and the Intel RES2SV240 can only support 20 drives! 

 

Would either of these configs be possible utilizing 2 x RES2SV24 as they seem quite cheap right now and i can power them from my 2 x pcie 2.0 slots without sacrificing any pci-e 3.0 slots?

 

Dell H200 HBA Card 1 --> Link 1 --> 12 ports on Intel SAS expander RES2SV240 card (1) 

                                 --> Link 2 --> 12 ports on Intel SAS expander RES2SV240 card (2)

 

OR

 

Dell H200 HBA Card 1 --> Link 1+2--> 12 ports on Intel SAS expander RES2SV240 card 1) 

Dell H200 HBA Card 2 --> Link 1+2 --> 12 ports on Intel SAS expander RES2SV240 card (2) 

 

??

 

What would you guys say is the top end read/write speed of the new 10/12/14TB disks? I'm thinking if i can offer up around 185mb/s for each of my 24 spinner disks that's likely to be pretty optimal....

 

??

Edited by thestraycat
Link to comment
15 minutes ago, thestraycat said:

Would either of these configs be possible utilizing 2 x RES2SV24 as they seem quite cheap right now and i can power them from my 2 x pcie 2.0 slots without sacrificing any pci-e 3.0 slots?

Both are possible, there's also a 36 port Intel SAS2 expander (RESCV360JBD) but it's usually more expensive.

 

17 minutes ago, thestraycat said:

What would you guys say is the top end read/write speed of the new 10/12/14TB disks?

Top speed in the outer sectors range from around 225MB/s for a 10TB 5400rpm drive to around 275MB/s for 14TB 7200rpm one.

 

Link to comment
23 hours ago, johnnie.black said:

If the R710 is PCIe 2.0 then yes, might as well get a 9211-8i or similar.

Maybe ... maybe not. Consider that the 9211-8i (and its LSI SAS2008-based brethren) appear to have an ultimate bottleneck of their internal CPU/memory/firmware which precludes them achieving the maximum (real-world/measurable) PCIe Gen2 x8 throughput of 3200 MB/s. Your tests, on the H310, show 2560 (8x320) MB/s.

 

Whereas the 9207-8 (and its SAS2308 brethren) do have the "muscle-power" to achieve, and exceed, full  Gen2x8 throughput (3200); you measured, on a Gen3x8, 4200 (8x525) MB/s--and that is limited, not by the 2308, but by maxing out the Sata3 (real-world) speed of ~525 [I suspect that SAS-12Gb SSDs would do better, no?***] So, for 9211 vs 9207, using Gen2, it comes down to a cost/benefit decision (plus, an appropriate degree of future-proof factor; re-use following a mobo upgrade).

 

*** [Edit] Actually, NO -- it looks like 12G SAS connectivity was not offered until the 93xx series.

 

Edited by UhClem
Addendum ***
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.