Slow performance?

[theonewithin]

Hello all!

Trying to diagnose why the performance of my setup is slower than expected.

Have 10x 8TB SAS drives connected through a SAS expander to a typical LSI 8 Channel HBA.

Previously when using SATA based driver performance was stellar and during parity operations I could achieve well over 1.5GB/sec

With these SAS drives I appear to be hitting some sort of bottleneck.

Only maxing out at around 900MB/s total transfer rate.

Each drive when tested alone can achieve over 200MB/s.

Adding 4 drives directly to the HBA and bypassing the expander shows no change at all.

If I do a scrub of a set of 3 SAS 15K cache drives running on the same setup the speed of this is very slow.

Once pausing the parity check the performance shoots over 700MB/s which is the expected speed of these drives.

 

Have run out of things to try and am now here to find help!

I hope someone can figure out where the bottleneck of my system is.

Thanks all!

Have attached diagnostics.

big-server-diagnostics-20221022-0038.zip

[JorgeB]

Run the diskspeed docker, it will test disks individually and controller bandwidth with all together.

[theonewithin]

It doesn't give a controller option.

All disks run at expected speed as per my original post.

[JorgeB]

Post the screenshot from the test.

[theonewithin]

[theonewithin]

Not sure why you won't just accept that I have done it and per drive performance is fine.

[JorgeB]

Not that one, the controller test.

[theonewithin]

As I said, It gives no option for that.

Wait...

I just found it.

Why is it hidden behind a button?

 

Edited October 21, 2022 by theonewithin

[theonewithin]

Ok. That has pointed out the issue.

For some reason the card is running at 5GT/s over 2 lanes...

Odd.

Time to start unplugging stuff I guess.

[theonewithin]

Well crap.

HBA card appears to be the issue. Won't negotiate more than 2x lanes.

Swapped in a GPU and it connects fine with 4-16 lanes.