Array slowness, need to adjust disk tunables?

[weirdcrap]

I recently made some substantial hardware changes to my second UnRAID backup machine and since doing so my array speeds have been slower than what they used to be. My array consists of a mix of 2TB disks and a single 4TB parity drive.

 

System Specifications

 

Old specs:

Mobo: ASROCK 880GMLE-FX

CPU: AMD FX-6300 Black Edition

RAM: 8GB (2x 4GB) Corsair Ballistix DDR3

Storage Cards: SuperMicro AOC-SASLP-MV8

Total Drives: 16 x 2TB (1 Parity x 15 data)

Drives were all mounted in CoolerMaster 4x3 bay expanders where each drive had an individual power and SATA connection.

 

PCI/PCIE Layout:

1x PCIE x1<----- IOCrest IO-PCE9215-4I

1x PCIE x16 <---- SuperMicro Card

2x PCI <----- Graphics Card

 

 

my new specs:

Mobo: ASUS M5A97 R2.0

CPU: AMD FX-6300 Black Edition

RAM: 16GB (4x 4GB) Corsair Ballistix DDR3

Storage Cards:

-SuperMicro AOC-SASLP-MV8 6x drives

-SuperMicro AOC-SAS2LP-MV8 6x drives

-SYBA SI-PEX40064 aka. IOCrest IO-PCE9215-4I 2x

Other six drives are connected directly to motherboard.

Total Drives: 19 x 2TB 1 x 4TB (1 4TB Parity x 19 2TB data)

These are all in NORCO SS-500 5-Bay SATA / SAS Hot Swap Rack Module with individual SATA connections per drive and shared power from 2x molex connectors per bay.

 

PCI/PCIE Layout:

1 x PCIe 2.0 x16 (blue)  <---- SuperMicro Card          x6 disks (all 2TB)

1 x PCIe 2.0 x16 (x4 mode, black)  <---- SuperMicro Card          x6 disks (all 2TB)

1 x PCIe 2.0 x1  <-----  Empty

1 x PCIe 2.0 x1  <----- IOCrest IO-PCE9215-4I card      x2 disks (all 2TB last 6 on Mobo).

2 x PCI <----- One Empty, one PCI Nvidia GTX200 series Graphics Card

 

The parity drive, cache drive, and a few random array disks are all on the motherboard ports.

 

I used the Disk Speed test script to test my disks and none of them are reporting terribly slow speeds (sdb & sdh are USB disks):

 

1) Before the hardware changes my array writes and parity checks were maxing out around ~112MB/s which is awesome and where I want to get it back to.

 

2) After the hardware changes and before I messed with the disk tuneable settings at all I was lucky to see over ~50MB/s (usually closer to ~30MB/s) during parity operations.

 

3) After applying some 'sane' numbers from the UnRAID wiki my speeds have improved but are still not up to where they used to be. Now I consistently see speeds around ~70MB/s during parity operations.

 

The two tuneables I modified are:

md_num_stripes: 2048

md_sync_window: 1024

 

 

Should I still use Pauven's tunables script to get a better idea of what my numbers should be?

 

I see Johnny.Black has some advice for those of us who use SAS2LP cards. Is his advice still recommended as well?

 

I also see Squid links to some discussions here that mention some issues with different drive sizes?

 

I haven't flashed any updated firmware to any of them and INT13 is disabled on both cards.

 

I haven't had a chance to read through all of the various discussions I am linking to, I am hoping someone with way more knowledge of this issue will be able to pop in and help me much quicker than it would take for me to read through all the different discussions and pages upon pages of replies.

 

Edit: forgot to get to the point:

 

So bottom line is, should I continue to mess with my tunables? Any recommended values?

 

Should I try something else first?

[John_M]

I think I'd consider moving two of the disks off the SASLP and onto the SAS2LP to give eight on the newer faster card and four on the older slower card. Regarding tunables, I'd use johnnie.black's recommendations.

[JorgeB]

1) Before the hardware changes my array writes and parity checks were maxing out around ~112MB/s which is awesome and where I want to get it back to.

 

If I'm reading your old config correctly that's not possible, if you had 8 disks on the SASLP max speed during a parity check with the best possible settings would be 80MB/s.

 

For your current config there's a couple of things you can try:

 

-make sure the SAS2LP is in the x16 slot and connect 8 disks to it, SASLP on the x4 slot with 4 disks only

 

-set your tunables to these (unRAID v6.2 or v6.3)

 

nr_requests=8

num_stripes=4096

sync_window=2048

sync_thresh=2000

 

[weirdcrap]

The old configuration I believe had 6 disks on the saslp, 6 on the mobo, and 4 on a pcie x1 sata card. Sorry I didn't flesh the specs out better I copy pasted what I had in another thread from last month.

Those parity speed numbers for the old setup are from memory and I don't know if I have any screenshots or anything to back those numbers up (they may be exaggerated.)

 

Yes the SAS2LP is in the x16 and the SASLP is on the x4.

 

I will juggle some drives around and see how that does, then apply the new tunable numbers.

 

Sent from my Nexus 6P using Tapatalk

 

[JorgeB]

The old configuration I believe had 6 disks on the saslp

 

Ok, in that case you could get  up to 105MB/s

[weirdcrap]

Small update, I was slightly off on the disk numbers. The disk numbers before my juggling were:

 

Motherboard (6 ports/6 used): 4x data          1x SSD cache      1x 2TB hot spare

SAS2LP (8 ports/7 used): 6x data 1x parity

SASLP (8 ports/6 used): 6x data

IOCrest IO-PCE9215-4I (4 ports/2 used): 2x data

 

I moved one data disk off the old SASLP card for a layout of:

 

Motherboard (6 ports/6 used): 4x data        1x SSD cache      1x 2TB hot spare

SAS2LP (8 ports/8 used): 7x data  1x parity

SASLP (8 ports/5 used): 5x data

IOCrest IO-PCE9215-4I (4 ports/2 used): 2x data

 

Should I load the fifth drive from the SASLP on to the IOCrest card in the PCI-E 2.0 x1 slot or leave it? I haven't messed with the tuneables yet, server currently half together on my office floor.

 

I could also off load drives onto a second 1x card I have (I still have an open PCI-E 2.0 1x slot). I seem to recall reading somewhere that unlike PCI slots, PCI-E slots each have their own dedicated bandwidth?

[JorgeB]

I«d leave it as it is now, with some tuning it should give >100MB/s.

[weirdcrap]

I«d leave it as it is now, with some tuning it should give >100MB/s.

With the dissk moved and your tunable settings I am now seeing roughly ~105MB/s during parity check. Do you think I can push it any faster with my setup? I'm happy with where it is now but faster is always better right?

 

[JorgeB]

You can try swapping the onboard hot spare with one of the SASLP disks, but doubt that it will make a significant difference, there's a point with so many disks that the CPU/RAM can also be bottlenecks.

 

Also, 105MB/s is a pretty decent speed, and it's already close to the maximum speed of some of your older disks.

[weirdcrap]

got home from work and checked on it and now it seems to be hovering around ~70MB/s.

 

 

[JorgeB]

That's perfectly normal, if you look at the disk speed test graphics you'll see that all disks start much faster on the outer cylinders and speed gradually decreases as it goes through the inner cylinders, parity check speed can't be faster than your slowest disk at any given position.

[weirdcrap]

That's perfectly normal, if you look at the disk speed test graphics you'll see that all disks start much faster on the outer cylinders and speed gradually decreases as it goes through the inner cylinders, parity check speed can't be faster than your slowest disk at any given position.

Makes sense, I did some digging in my documents and found my chart from the speedtest I ran. Thanks for your help.