Sata/SAS controllers tested (real world max throughput during parity check)

[JonathanM]

https://www.broadcom.com/support/ethernet-network-adapters/counterfeit-statement

[rosswaters]

On 3/9/2020 at 3:02 AM, JorgeB said:

JorgeB
The dual controllers in each enclosure are for redundancy/multipath and since UnRaid doesn't support SAS multipath you basically have two connection options:

 

1) single cable from HBA to first controller on the 1st enclosure then daisy chain to 2nd enclosure

2) and best option for performance, one cable from each HBA port to each enclosure

For "Option 2" What 3 SAS connectors or what one per Backplane do I use. I am having a hard time finding documents on correct performance configuration?

For my current setup it is daisy chained and I have twice the drive count with half not existing. I am also getting really bad performance across all drives. I love that it has 36 HDD bays but I with I could just run 9 SSF-8087 cables to them.

Thanks for any info you can give,
Ross

 

[JorgeB]

15 hours ago, rosswaters said:

For "Option 2" What 3 SAS connectors or what one per Backplane do I use. I am having a hard time finding documents on correct performance configuration?

 

What HBA and enclosure?

[rosswaters]

On 10/18/2021 at 8:33 AM, JorgeB said:

 

What HBA and enclosure?

I have the Supermicro SC847 Chassis
The Backplane is the SAS2-846EL2
From what I can tell you can use 2 SSF-8088 on the front backplane but that only works as failover. It has Primary 1, 2, & 3 and Secondary 1, 2, & 3.

Personally I would think all the Primary Ports are for the HBA support with Secondary 1, 2, & 3 for Cascading. The documents are lacking on what each primary SFF-8088 port does and how to use them. For instance nowhere does it talk about Primary 3 and Secondary 3 and how to use them yet they're there on the PCB.

Can I connect the 3 Primary Ports up to 3 SFF-8088's and have 300MBs per second on SAS drives? I am using only SAS drives in this unit so there are no SATA's involved.

It's a 36 bay server so it has a 24 port backplane on the front and a 12 port on the rear. The Rear has 4 SFF-8088's 2 primary and two secondary's.

I currently am using two LSI 9211-8i's HBAs in IT mode but I'm moving to two Areca 1882I SAS controller. I am willing to add a 3rd HBA or Areca controller if I can run 5 SFF-8088's.

 

I am not daisy chaining to another enclosure and I don't need redundancy on the SFF-8088s, I am just looking for better than 100MBs per drive.

Thanks for any help.
Ross

[JorgeB]

3 hours ago, rosswaters said:

The Backplane is the SAS2-846EL2
From what I can tell you can use 2 SSF-8088 on the front backplane but that only works as failover. It has Primary 1, 2, & 3 and Secondary 1, 2, & 3.

It's a dual expander model, primary ports go to one expander, secondary ports to the other, with Unraid you only need to use the primary.

 

3 hours ago, rosswaters said:

Can I connect the 3 Primary Ports up to 3 SFF-8088's and have 300MBs per second on SAS drives?

No, for best performance you can connect one HBA using dual link to the front backplane, Supermicro recommends using primary ports J1 and J2, but it should work with any two ports on the same expander, also the 9211 with dual link will bottleneck due to being PCIe 2.0, a PCIe 3.0 HBA like the 9207-8i or 9300-8i would be better, assuming the board/CPU supports PCIe 3.0.

 

Not sure the seconds backplane supports dual link, you can test or post the model for that one to see if I can find any info, though with half the drives it will have the same approximate bandwidth with a single link as the front one with dual, and Unraid max array size is 30 devices, so you'll never need to use the 36 devices simultaneously.

 

 

 

 

 

 

 

 

[Masterwishx]

Finally i received the 9207-8i the original card from ebay and was very suprised , however was hoping that

its original and by all signes i know it is original card ,also by link posted befor . also looks like new ...

 

Edited October 20, 2021 by Masterwishx

[rosswaters]

6 hours ago, JorgeB said:

It's a dual expander model, primary ports go to one expander, secondary ports to the other, with Unraid you only need to use the primary.

 

No, for best performance you can connect one HBA using dual link to the front backplane, Supermicro recommends using primary ports J1 and J2, but it should work with any two ports on the same expander, also the 9211 with dual link will bottleneck due to being PCIe 2.0, a PCIe 3.0 HBA like the 9207-8i or 9300-8i would be better, assuming the board/CPU supports PCIe 3.0.

 

Not sure the seconds backplane supports dual link, you can test or post the model for that one to see if I can find any info, though with half the drives it will have the same approximate bandwidth with a single link as the front one with dual, and Unraid max array size is 30 devices, so you'll never need to use the 36 devices simultaneously.

 

 

 

 

 

 

 

 

Thank you JorgeB!

The MB is PCI 3.0 and I do have a hand full of 9207's on hand so I'll swap them out. I'm okay with 200MBs a second, 100 is just horrible. The documents says the 12 port backplane is a SAS-826EL2. It has 4 SFF-8087s, Primary 1 & 2 then Secondary 1 & 2.  They are all SFF-8087's I have a couple 8088's on my desk and I think I looked at them and then typed out the they were SFF-8088's without thinking about it. All internal ports are 8087s. ( I'm guessing you know that though  )

Thanks for the info. I will write this info down in my manual.

Ross

[JorgeB]

36 minutes ago, rosswaters said:

SAS-826EL2

This is a SAS1 backplane, half bandwidth of SAS2, will also likely have issues with drives > 2.2TB, fill up the front backplane first, then connect both cables from one HBA to the back baclplane to check if it supports dual link, you can confirm with the output of:

 

cat /sys/class/sas_host/host#/device/port-#\:0/sas_port/port-#\:0/num_phys

 

Replace all the #s with the correct host number, if you don't known post the diags, if the output is 4 it means single link, 8 means dual link.

[rosswaters]

On 10/20/2021 at 8:55 AM, JorgeB said:

This is a SAS1 backplane, half bandwidth of SAS2, will also likely have issues with drives > 2.2TB, fill up the front backplane first, then connect both cables from one HBA to the back backplane to check if it supports dual link, you can confirm with the output of:

 

cat /sys/class/sas_host/host#/device/port-#\:0/sas_port/port-#\:0/num_phys

 

Replace all the #s with the correct host number, if you don't known post the diags, if the output is 4 it means single link, 8 means dual link.

Thanks JorgeB for all your help. All outputs were 8 and I am now getting 200MBs per second on all the SAS drives. The backplane on the rear of the server is working at 200 as well so it must be a SAS2. I bought the server used in February and just didn't have the time then to make sure it was configured correctly. When I first got it the server was set for failover, it was configured for a camera surveillance server. While I was able to make UnRAID work with in that configuration it was having issues and was really slow. Now it is much better with speeds I was hoping to achieve.

Thanks again for your help.
Ross

[JorgeB]

7 hours ago, rosswaters said:

The backplane on the rear of the server is working at 200 as well so it must be a SAS2.

 

The model you mentioned is SAS1 (SAS2 would be SAS2-XXX), but SAS1 with dual link can still do 2200MB/s, so still decent performance even with 12 devices, close to 200MB/s per drive.

[Magnets]

Does anyone have experience of ASM1164? I noticed these cards from DIEWU on aliexpress (alibaba link) (aliexpress link)

ASM1164 with 2xSATA and 2xM.2

 

This website quotes lower power usage of ASM1166 vs JMB585

http://www.nosense.jp/jmb585-5port-sata/

 

JMB585 +3.6W idle/no HDD

ASM1166 +1.4W idle/no HDD

 

I assume ASM1164 will be similar numbers. The JMB585 cards (3x SATA + 2x M.2) appear to have increased price recently and fewer sellers offering them. I think we can blame chia miners 😕

 

This website quotes +2.8W idle for ASM1166

https://watchmysys.com/blog/2021/03/zydas-asm1166-sata-controller/

Edited December 2, 2021 by Magnets
added another power measurement

[JorgeB]

12 hours ago, Magnets said:

Does anyone have experience of ASM1164?

Yes, it's a good option, it's listed above, also on the recommended controllers list.

[mgutt]

On 9/21/2015 at 4:47 PM, JorgeB said:

5 and 6 Port Controllers

 

JMicron JMB585 PCIe gen3 x2 (1970MB/s) - e.g., SYBA SI-PEX40139 and other similar cards

2 x 570MB/s

3 x 565MB/s

4 x 440MB/s

5 x 350MB/s

 

Asmedia ASM1166 PCIe gen3 x2 (1970MB/s)

2 x 565MB/s

3 x 565MB/s

4 x 445MB/s

5 x 355MB/s

6 x 300MB/s

 

Maybe you want to add the power consumption. It's 2W for both controllers (cards):

https://www.hardwareluxx.de/community/threads/die-sparsamsten-systeme-30w-idle.1007101/page-68#post-28446459

 

@Magnets Maybe interesting for you, too. Both have the same power consumption (in idle of course). Most people forget, that they loose C8/C10 c-states of their cpu package, when they install a SATA card, which adds additional 2W, but this has nothing to do with the card itself. If you install multiple cards, each adds 2W.

 

Because of that the best power efficiency is reached without installing such cards at all f.e. by buying a motherboard like the C246N-WU2 with 8 onboard sata ports.

 

[JorgeB]

14 hours ago, mgutt said:

Maybe you want to add the power consumption. It's 2W for both controllers (cards):

Thanks, I will.

[cpu]

@JorgeByou may check my post -> 

and consider updating power draw for different cards. 

[flaggart]

On 9/21/2015 at 3:47 PM, JorgeB said:

Intel® SAS2 Expander RES2SV240 - 10w

 

Single Link with LSI 9211-8i (2400MB/s)

8 x 275MB/s

12 x 185MB/s

16 x 140MB/s (112MB/s*)

20 x 110MB/s (92MB/s*)

 

* Avoid using slower linking speed disks with expanders, as it will bring total speed down, in this example 4 of the SSDs were SATA2, instead of all SATA3.

 

Dual Link with LSI 9211-8i (4000MB/s)

12 x 235MB/s

16 x 185MB/s

 

Dual Link with LSI 9207-8i (4800MB/s)

16 x 275MB/s

 

Hello

 

This information is really useful and has inspired be to revamp my setup.  I wonder if someone could explain why the single port link speed between HBA and expander would be 2400MB/s rather than 3000MB/s (6gbit * 4 lanes = 24gbit or 3000MB/s).

 

Intended setup:

• 1x 9207-8e in a PCI-e 3.0 8x slot (max 7.877 GB/s)
• Single links in to 2x RES2SV240
• 16 disks per expander, all support SATA3
• Would this not provide 3000MB/s to each expander, and therefore up to 187.5MB/sec per disk - rather than 2400MB/s and 150MB/s?

 

Thanks

[JorgeB]

21 minutes ago, flaggart said:

 I wonder if someone could explain why the single port link speed between HBA and expander would be 2400MB/s rather than 3000MB/s (6gbit * 4 lanes = 24gbit or 3000MB/s).

Because it uses 8b/10b encoding, so it becomes 4.8Gbit/s=600MB/s

[Bizquick]

Wow first off this post explains so much its hard to take all of it in and takes a few times for me to read and figure out what all is wrong with my setup.

 Second I was looking to get some better speeds I have always liked the idea of using HBA cards vs the standard Sata ports off a main board at least for my spin disc's. SSD's on the other hand not sure that the speed difference mattered much.

But lately I have came into a new issue with speed and a different CPU and main boards.

 

I was thinking my old 9211 LSI cards were just too slow or something. So I search ebay for a smaller upgrade and got 9300 16i cards for not much more than 9211 cards.  with them able to do SAS 12GB I figured it might be closer to true 6GB Sata 3 speeds.

 

Now If I used this card in my AMD 3400G 470x chipset board. The speed were actually terrible vs using in my Intel B250 pro i7-6700.

both had same ram amounts and same ram speed.  But preforace on the AMD was terrible regardless which PCIe slot was used.

 

I'm starting to think maybe the 9300 firmware might be the issue I'll have the check the firmware version when I get home. But I'm wondering how much impact that would cause on the speeds.

 

Also should I have the party drive on the HBA with all the other spin disc's or should I put it on the Sata ports off the main board?

I have 2 cache drives on sata ports and right now the HBA has all 3 other drives. I'm using 3 drive per a channel on the 9300 16i card ports 1 and 3. Speed on intel seams ok I get 200 to 212MB.  but on AMD I cant break above 190MB I avg 165MB to 175MB at best.

I would think the AMD would be better because the 470x chipset is much newer and I'm using the main PCIe slot to get 3.0 16x or so.

 

 

[JorgeB]

47 minutes ago, Bizquick said:

Now If I used this card in my AMD 3400G 470x chipset board. The speed were actually terrible vs using in my Intel B250 pro i7-6700.

There's a known issue with the current kernel Unraid uses and AMD CPUs.

[Bizquick]

Great well maybe I can reload that box with the AMD. What Kernel version should I run?

[JorgeB]

Try with v6.10.3

[Soren Hansen]

On 1/10/2023 at 4:44 PM, JorgeB said:

There's a known issue with the current kernel Unraid uses and AMD CPUs.

I AM new at this, but those issues are, to my understanding, related to dedicated server hardware like Threadrippers and Epyc cpu's, Bizquick are, again as I read it, asking about consumer grade hardware.

[twin_suns]

Is there any chance that running two SSD's on a 5 port (gen3 x2) JMB585 (but in a gen3 x1 electrical slot) would have any performance benefit over a 2 port (gen3 x1) JMB582? 

Edited May 4, 2023 by twin_suns

[JorgeB]

1 hour ago, twin_suns said:

Is there any chance that running two SSD's on a 5 port (gen3 x2) JMB585 (but in a gen3 x1 electrical slot) would have any performance benefit over a 2 port (gen3 x1) JMB582? 

They should perform identically in that scenario.

[Allexz]

On 5/4/2023 at 12:32 PM, JorgeB said:

They should perform identically in that scenario.

If you connect 6/6 sata ports to SSDs and only one ssd transfers at this moment in time will it get the full throughput of the pci-express slot ? or is it locked down due to the controller?

 

In my case all the drives will never be used simultaneously, but i would  like for the ones that are used to get as much bandwith at that time.

 

Edit: Also im hoping these asmedia cards will be nice.

So many other forums/threads completely just hate them and say go buy hba

Edited May 15, 2023 by Allexz