Sata/SAS controllers tested (real world max throughput during parity check)

[JorgeB]

8 hours ago, Allexz said:

If you connect 6/6 sata ports to SSDs and only one ssd transfers at this moment in time will it get the full throughput of the pci-express slot ?

Yes, you'll get around 560MB/s

[coolspot]

On 10/20/2021 at 12:33 AM, rosswaters said:

I am not daisy chaining to another enclosure and I don't need redundancy on the SFF-8088s, I am just looking for better than 100MBs per drive.
 

 

Did you ever figure this out, I have the same 36 bay enclosure with the same SAS2-846EL2 controller, and fully loaded I'm maxed out at ~90MB/s using SATA drives.

[Kirkbuilder]

Love this topic!  I read all the posts here and I am still a bit confused.  Please forgive my ignorance about hardware, but I am trying 🙂  I have been trying to figure out how to increase my disk speeds on my array and was told in a discord channel that I was stuck at my current speeds and there was nothing to be done about it.  BUT, I kept on looking and found this wonderful article which I have learned a great deal from.  My problem, my array of 18 disks is doing read/write at ~100mb and am currently shrinking my array using the second non-volatile procedure of moving the data using unbalance and then 0'ing out the drive to update parity, etc, etc. but am only getting 69.8 MB/s.  As can be seen here: 

Here is my array:

 

Here is my setup Unraid 6.12.3:

GA-7PESH2 GIGABYTE Intel Rev1.0 LGA2011 E-ATX Server Motherboard LSI2008 8x SAS/SATA, 2x 10G X540-AT2, Avocent IPMI 2.0 (GA-7PESH2)

2x Intel Xeon E5-2690 v2 3.0GHz 10 Core 25MB Cache Processors

128GB DDR3 1600MHz ECC Registered Memory 8x 16GB

HP 487738-001 468405-001 24-Bay 3Gb SAS Expander Card

Samsung 860 QVO 1TB 2.5 Inch SATA III Internal SSD (MZ-76Q1T0B/AM) x2 Cache

24 x 8+ TB Drives

 

I want to try to maximize bandwidth since it is taking me ~40 hours to "0" out an 8TB drive.  From what I have read, I am now thinking I need to get 2 SAS cards to do this, but was hoping you might confirm that a good option would be 2x LSI 9305-24i's.

 

[JorgeB]

Please post the diagnostics. 

[Kirkbuilder]

tower-diagnostics-20230905-1721.zip

[JorgeB]

With 18 disks connected to the HP expander max speed will max out at roughly 120MB/s, still should be better then 100MB/s, possibly the CPU bottlenecking, also post the output of:

 

lspci -d 1000: -vv

 

to check HBA link speed/width

[Kirkbuilder]

9 hours ago, JorgeB said:

With 18 disks connected to the HP expander max speed will max out at roughly 120MB/s, still should be better then 100MB/s, possibly the CPU bottlenecking, also post the output of:

 

lspci -d 1000: -vv

 

to check HBA link speed/width

Here you go.  I was hoping to get something over 200MB or more with a change.

lspci.txt

[JorgeB]

The HBA is only linking at x4:

 

Speed 5GT/s, Width x4 (downgraded)

 

That means around 1600MB/s usable total, with 16 drives it's 100MB/s, so it checks out.

 

To be able to get around 200MB/s the HBA would need to be on an x8 slot and you'd also need a faster expander, current one will max out at around 120MB/s as mentioned. 

[Kirkbuilder]

10 hours ago, JorgeB said:

The HBA is only linking at x4:

 

Speed 5GT/s, Width x4 (downgraded)

 

That means around 1600MB/s usable total, with 16 drives it's 100MB/s, so it checks out.

 

To be able to get around 200MB/s the HBA would need to be on an x8 slot and you'd also need a faster expander, current one will max out at around 120MB/s as mentioned. 

What would you suggest to replace it?

[JorgeB]

5 minutes ago, Kirkbuilder said:

What would you suggest to replace it?

First you need to get the HBA linking at x8, since that's the current bottleneck, is that possible with your board?

[Kirkbuilder]

15 minutes ago, Kirkbuilder said:

What would you suggest to replace it?

Thank you for your analysis!  I have had a great deal of luck with HP so probably want HP 12G SAS3 EXPANDER (761879-001) or would you suggest another card?

[Kirkbuilder]

25 minutes ago, JorgeB said:

First you need to get the HBA linking at x8, since that's the current bottleneck, is that possible with your board?

https://download1.gigabyte.com/Files/Manual/server_manual_7pesh_e_1003.pdf?v=102d7cc391a19a751aff685e16f3ee87 I believe so.

[Kirkbuilder]

3 hours ago, JorgeB said:

First you need to get the HBA linking at x8, since that's the current bottleneck, is that possible with your board?

Yes.  The expander was in an 16x PCI slot.  I switched it to an 8x slot.  here is my new lspci. Still shows LnkSta:    Speed 5GT/s, Width x4 (downgraded)

lspci.txt

[JorgeB]

Not clear if you moved the expander only or also the HBA, but if it's still linking at x4 it will still bottleneck. 

[Kirkbuilder]

7 hours ago, JorgeB said:

Not clear if you moved the expander only or also the HBA, but if it's still linking at x4 it will still bottleneck. 

I am having trouble figuring out how to fix this.  Googling and googling with no joy 😞

[JorgeB]

Basically the HBA needs to be on an x8 or x16 electrical slot, some boards have x8 or x16 physical slots that are only x4 electrically.

[ChatNoir]

Possibly if you already have an extension card using x16 (graphics card maybe), your motherboard wouldn't or couldn't provide you x8 but only x4 ?

[Kirkbuilder]

On 9/8/2023 at 12:48 PM, ChatNoir said:

Possibly if you already have an extension card using x16 (graphics card maybe), your motherboard wouldn't or couldn't provide you x8 but only x4 ?

No other cards are installed in the PCI slots.  I have been looking for a way to determine if there is a setting which is keeping the power low on the motherboard or on the expander card.  Any ideas?

[ChatNoir]

If the card is plugged in the first PCIe slot (generally used for Graphics Cards) I wouldn't know why it would not link at full speed.

[tazire]

Curious if anyone knows anything about an LSI SAS3x36R expander? I have been looking at a 2U 24bay chassis that uses this expander with the intention of using it as a JBOD to connect SSD's to my server. I'm just trying to make sure that it will have the throughput to use SSD's. Fairly sure it should be absolutely fine during normal use but in a worst case if I'm hitting all 24 drives at once what's the theoretical throughput?

 

EDIT* 

I would use a 9500-16e HBA to get 3 connections and PCIE4 link to HBA. 

Edited September 20, 2023 by tazire

[JorgeB]

1 hour ago, tazire said:

it as a JBOD to connect SSD's to my server.

SAS roughly 9600MB/s theoretical max, I would expect around 8800MB/s usable.

SATA devices can make use of LSI's Databolt for better than 4800MB/s, but don't expect same as SAS, should be capable of at least 6000 to 7000MB/s.

 

1 hour ago, tazire said:

I would use a 9500-16e HBA to get 3 connections and PCIE4 link to HBA. 

You can just connect two cables from the HBA to the expander.

[tazire]

1 hour ago, JorgeB said:

SAS roughly 9600MB/s theoretical max, I would expect around 8800MB/s usable.

SATA devices can make use of LSI's Databolt for better than 4800MB/s, but don't expect same as SAS, should be capable of at least 6000 to 7000MB/s.

 

You can just connect two cables from the HBA to the expander.

The particular expander allows for up to 3 connections. Would this make any difference?

[JorgeB]

10 minutes ago, tazire said:

The particular expander allows for up to 3 connections.

Only two wide connections are supported by expanders, the other one can be for cascading.

[DuckServer]

Can we get 32 HDDs x 140MB/s by this way? , just plug one HBA card in PCIEX16 slot. 

[JorgeB]

47 minutes ago, DNU said:

Can we get 32 HDDs x 140MB/s by this way? ,

You can, note that speed will be limited to roughly 140MB/s with all disks used simultaneously.