Jump to content
johnnie.black

Sata/SAS controllers tested (real world max throughput during parity check)

12 posts in this topic Last Reply

Recommended Posts

I had the opportunity to test the “real word” bandwidth of some commonly used controllers in the community, so I’m posting my results in the hopes that it may help some users choose a controller and others understand what may be limiting their parity check/sync speed.

 

Note that these tests are only relevant for those operations, normal read/writes to the array are usually limited by hard disk or network speed.

 

Next to each controller is its maximum theoretical throughput and my results depending on the number of disks connected, result is observed parity check speed using a fast SSD only array with Unraid V6.1.2 (SASLP and SAS2LP tested with V6.1.4 due to performance gains compared with earlier releases)

 

Values in green are the measured controller power consumption with all ports in use.

 

2 Port Controllers

 

SIL 3132 PCIe gen1 x1 (250MB/s)

1 x 125MB/s

2 x 80MB/s

 

Asmedia ASM1061 PCIe gen2 x1 (500MB/s) - e.g., SYBA SY-PEX40039 and other similar cards

1 x 375MB/s

2 x 206MB/s

 

4 Port Controllers

 

SIL 3114 PCI (133MB/s)

1 x 105MB/s

2 x 63.5MB/s

3 x 42.5MB/s

4 x 32MB/s

 

Adaptec AAR-1430SA PCIe gen1 x4 (1000MB/s)

4 x 210MB/s

 

Marvell 9215 PCIe gen2 x1 (500MB/s) - 2w - e.g., SYBA SI-PEX40064 and other similar cards (possible issues with virtualization)

2 x 200MB/s

3 x 140MB/s

4 x 100MB/s

 

Marvell 9230 PCIe gen2 x2 (1000MB/s) - 2w - e.g., SYBA SI-PEX40057 and other similar cards (possible issues with virtualization)

2 x 375MB/s

3 x 255MB/s

4 x 204MB/s

 

8 Port Controllers

 

Supermicro AOC-SAT2-MV8 PCI-X (1067MB/s)

4 x 220MB/s (167MB/s*)

5 x 177.5MB/s (135MB/s*)

6 x 147.5MB/s (115MB/s*)

7 x 127MB/s (97MB/s*)

8 x 112MB/s (84MB/s*)

 

*on PCI-X 100Mhz slot (800MB/S)

 

Supermicro AOC-SASLP-MV8 PCIe gen1 x4 (1000MB/s) - 6w

4 x 140MB/s

5 x 117MB/s

6 x 105MB/s

7 x 90MB/s

8 x 80MB/s

 

Supermicro AOC-SAS2LP-MV8 PCIe gen2 x8 (4000MB/s) - 6w

4 x 340MB/s

6 x 345MB/s

8 x 320MB/s (205MB/s*, 200MB/s**)

 

*on PCIe gen2 x4 (2000MB/s)

**on PCIe gen1 x8 (2000MB/s)

 

Dell H310 PCIe gen2 x8 (4000MB/s) - 6w – LSI 2008 chipset, results should be the same as IBM M1015 and other similar cards

4 x 455MB/s

6 x 377.5MB/s

8 x 320MB/s (190MB/s*, 185MB/s**)

 

*on PCIe gen2 x4 (2000MB/s)

**on PCIe gen1 x8 (2000MB/s)

 

LSI 9207-8i PCIe gen3 x8 (4800MB/s) - 9w - LSI 2308 chipset

8 x 525MB/s+ (*)

 

LSI 9300-8i PCIe gen3 x8 (4800MB/s with the SATA3 devices used for this test) - LSI 3008 chipset

8 x 525MB/s+ (*)

 

* used SSDs maximum read speed

 

SAS Expanders

 

HP 6Gb (3Gb SATA) SAS Expander - 11w

 

Single Link on Dell H310 (1200MB/s*)

8 x 137.5MB/s

12 x 92.5MB/s

16 x 70MB/s

20 x 55MB/s

24 x 47.5MB/s

 

Dual Link on Dell H310 (2400MB/s*)

12 x 182.5MB/s

16 x 140MB/s

20 x 110MB/s

24 x 95MB/s

 

* Half 6GB bandwidth because it only links @ 3Gb with SATA disks

 

Intel® RAID SAS2 Expander RES2SV240 - 10w

 

Single Link on Dell H310 (2400MB/s)

8 x 275MB/s

12 x 185MB/s

16 x 140MB/s (112MB/s*)

20 x 110MB/s (92MB/s*)

 

Dual Link on Dell H310 (4000MB/s)

12 x 205MB/s

16 x 155MB/s (185MB/s**)

 

Dual Link on LSI 9207-8i (4800MB/s)

16 x 275MB/s

 

LSI SAS3 expander (included on a Supermicro BPN-SAS3-826EL1 backplane)

 

Single Link on LSI 9300-8i (tested with SATA3 devices, max usable bandwidth would be 2200MB/s, but with LSI's Databolt technology we can get almost SAS3 speeds)

8 x 475MB/s

12 x 340MB/s

 

Dual Link on LSI 9300-8i (tested with SATA3 devices, max usable bandwidth would be 4400MB/s, but with LSI's Databolt technology we can get almost SAS3 speeds, limit here is going to be the PCIe 3.0 slot, around 6000MB/s usable)

10 x 510MB/s

12 x 460MB/s

 

 

* Avoid using slower linking speed disks with expanders, as it will bring total speed down, in this example 4 of the SSDs were SATA2, instead of all SATA3.

** Two different boards have consistent different results, will need to test a third one to see what's normal, 155MB/s is the max on a Supermicro X9SCM-F, 185MB/s on Asrock B150M-Pro4S.

 

Sata 2 vs Sata 3

I see many times on the forum users asking if changing to Sata 3 controllers or disks would improve their speed, Sata 2 has enough bandwidth (between 265 and 275MB/s according to my tests) for the fastest disks currently on the market, if buying a new board or controller you should buy sata 3 for the future, but except for SSD use there’s no gain in changing your Sata 2 setup to Sata 3.

 

Single vs. Dual Channel RAM

In arrays with many disks, and especially with low “horsepower” CPUs, memory bandwidth can also have a big effect on parity check speed, obviously this will only make a difference if you’re not hitting a controller bottleneck, two examples with 24 drive arrays:

 

Asus A88X-M PLUS with AMD A4-6300 dual core @ 3.7Ghz

Single Channel – 99.1MB/s

Dual Channel - 132.9MB/s

 

Supermicro X9SCL-F with Intel G1620 dual core @ 2.7Ghz

Single Channel – 131.8MB/s

Dual Channel – 184.0MB/s

 

DMI

There is another bus that can be a bottleneck for Intel based boards, much more so than Sata 2, the DMI that connects the south bridge or PCH to the CPU.

Socket 775, 1156 and 1366 use DMI 1.0, socket 1155, 1150 and 2011 use DMI 2.0, socket 1151 uses DMI 3.0

 

DMI 1.0 (1000MB/s)

4 x 180MB/s

5 x 140MB/s

6 x 120MB/s

8 x 100MB/s

10 x 85MB/s

 

DMI 2.0 (2000MB/s)

4 x 270MB/s (Sata2 limit)

6 x 240MB/s

8 x 195MB/s

9 x 170MB/s

10 x 145MB/s

12 x 115MB/s

14 x 110MB/s

 

DMI 3.0 (3940MB/s)

6 x 330MB/s (Onboard SATA only*)

10 X 297.5MB/s

12 x 250MB/s

16 X 185MB/s

 

*Despite being DMI 3.0, Skylake, Kaby Lake and Coffee Lake chipsets have a max combined bandwidth of approximately 2GB/s for the onboard SATA ports.

 

DMI 1.0 can be a bottleneck using only the onboard Sata ports, DMI 2.0 can limit users with all onboard ports used plus an additional controller onboard or on a PCIe slot that shares the DMI bus, in most home market boards only the graphics slot connects directly to CPU, all other slots go through the DMI (more top of the line boards, usually with SLI support, have at least 2 slots), server boards usually have 2 or 3 slots connected directly to the CPU, you should always use these slots first.

 

You can see below the diagram for my X9SCL-F test server board, for the DMI 2.0 tests I used the 6 onboard ports plus one Adaptec 1430SA on PCIe slot 4.

 

UMI (2000MB/s) - Used on most AMD APUs, equivalent to intel DMI 2.0

6 x 203MB/s

7 x 173MB/s

8 x 152MB/s

 

Ryzen link - PCIe 3.0 x4 (3940MB/s)

6 x 467MB/s (Onboard SATA only)

 

 

 

I think there are no big surprises and most results make sense and are in line with what I expected, exception maybe for the SASLP that should have the same bandwidth of the Adaptec 1430SA and is clearly slower, can limit a parity check with only 4 disks.

 

I expect some variations in the results from other users due to different hardware and/or tunnable settings, but would be surprised if there are big differences, reply here if you can get a significant better speed with a specific controller.

 

 

 

 

How to check and improve your parity check speed

 

System Stats from Dynamix V6 Plugins is usually an easy way to find out if a parity check is bus limited, after the check finishes look at the storage graph, on an unlimited system it should start at a higher speed and gradually slow down as it goes to the disks slower inner tracks, on a limited system the graph will be flat at the beginning or totally flat for a worst-case scenario. See screenshots below for examples (arrays with mixed disk sizes will have speed jumps at the end of each one, but principle is the same).

 

If you are not bus limited but still find your speed low, there’s a couple things worth trying:

 

Diskspeed - your parity check speed can’t be faster than your slowest disk, a big advantage of Unraid is the possibility to mix different size disks, but this can lead to have an assortment of disk models and sizes, use this to find your slowest disks and when it’s time to upgrade replace these first.

 

Tunables Tester - on some systems can increase the average speed 10 to 20Mb/s or more, on others makes little or no difference.

 

 

That’s all I can think of, all suggestions welcome.

x9scl-f.png.fc4e744b1f64de82f2c8e7c417b03e87.png

speed_unlimited.png.40b2fa9c53917183be317747dc492192.png

speed_limited.png.5e1bb1bb2533cd98746723a1ef915c52.png

Edited by johnnie.black

Share this post


Link to post
Supermicro AOC-SAS2LP-MV8 PCIe gen2 8x (4000MB/s)

There’s currently an issue with Unraid V6, will update if/when fixed.

 

I'm running this card on v6.0.1 without any issues at all?  Was just about to upgrade to v6.1.3.  Given your statement, should I wait to upgrade?

Share this post


Link to post

I'm running this card on v6.0.1 without any issues at all?  Was just about to upgrade to v6.1.3.  Given your statement, should I wait to upgrade?

 

There was an issue with any V6 and parity check performance, although it didn’t affect all users, it was fixed in 6.1.4, will post test results soon.

Share this post


Link to post

 

2 Port Controllers

 

SIL 3132 PCIe gen1 1x (250MB/s)

2 x 80MB/s

Thanks Johnnie, that explains my pcheck limitation on my boxes - as I have 2xLSI2008, 6 onboard plus this PCIe card.

It seems, there is no better alternative for PCIex1/2-port available, right? (Don't have a 3rd slot for 3rd LSI card available either...)

Share this post


Link to post

Thanks Johnnie, that explains my pcheck limitation on my boxes - as I have 2xLSI2008, 6 onboard plus this PCIe card.

It seems, there is no better alternative for PCIex1/2-port available, right? (Don't have a 3rd slot for 3rd LSI card available either...)

 

For 2 ports I would recommend Asmedia 1601 based controllers, much better performance and they are cheap, like this one:

 

http://www.newegg.com/Product/Product.aspx?Item=N82E16816124045

 

The one I used for testing was onboard but performance should be very similar.

 

Share this post


Link to post

Thanks for the hint - indeed, that is a cheap and obviously better solution than the sil3132 based cards.

Share this post


Link to post

I know this post is old.  Resurrecting a dead thread.  But this thread has me rethinking the whole server I am currently building.

 

Question I have:  Speed of the Expanders is directly related to the number of drives actually hooked up currently and running, and not just ports correct?  So if I had all 20 ports connected to empty SATA slots, but only 8 SATA drives hooked up, I would achieve maximum  throughput for 8 drives if the drives supported it?

Share this post


Link to post

 

@johnnie.black this is a tremendous resource and was interested in your thoughts on what would be the expected speeds to determine the optimal configuration for a for 30+ drives in the following configuration:

 

SuperMicro SuperStorage 6047R-E1R36L configured as:

  • BPN-SAS2-846EL1 (front 24 bays) with three 8087 connectors
  • BPN-SAS2-826EL1 (rear 12 bays) with two 8087 connectors
  • LSI 2308 SAS2 PCIe 3.0 host adapter with two 8087 connectors (motherboard integrated)
  • 6x SATA ports on the motherboard, running at 6GB SATA3 (DMI 2.0)
  • 2x NVMe slots via Supermicro AOC-SLG3-2M2 add-in card
  • The drives are all WD Red/White 10TB & 8TB models

Unraid setup as of 4/27/2019:

  • Data disks x28
    • x24 connected to BPN-SAS2-846EL1 dual linked to LSI 2308
    • x4 connected to motherboard SATA ports
  • Parity disk
    • x1 disk connected to motherboard SATA port
  • cache disk
    • x1 NVMe drive connected to AOC-SLG-2M2
  • Not in use/connected
    • BPN-SAS2-826EL1 (rear 12 bays)
    • x1 SATA port
    • x1 NVMe port

 

Is this the most optimal setup to spread out the utilization and not encounter bottlenecks? Any opportunities for improvement?

 

If I was to daisy chain a single link from the third 8087 connector on the BPN-SAS2-846EL1 to the downstream BPN-SAS2-826EL1 how much of a negative impact on performance would that have? The 826-EL1 would be empty, no disks. Would that reduce PHYs available or add significant additional SAS overhead?

 

 

 

 

Share this post


Link to post
9 hours ago, Lev said:

Is this the most optimal setup to spread out the utilization and not encounter bottlenecks? Any opportunities for improvement?

Looks optimal to me, max bandwidth is enough (or close enough) for the the current number of disks, and their max speed.

 

9 hours ago, Lev said:

If I was to daisy chain a single link from the third 8087 connector on the BPN-SAS2-846EL1 to the downstream BPN-SAS2-826EL1 how much of a negative impact on performance would that have? The 826-EL1 would be empty, no disks.

No impact without disks, with disks it would share the bandwidth of the other link, so if in the future you use that backplane for array disks it would be better to connect it to a different HBA.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.