Array getting slower as disks are added.

[Excessus]

Setup:

 

CPU - i7-6850K

MB - ASUS X99-A II

SAS Controllers - Dell PERC H310 x2

Drives - 11 8TB drives. 1 3TB drive. (10 data, 2 parity).

 

I have been adding drives to this system over time.  As I add more drives, parity checks and writes in general get slower and slower.  For a while, I had 7 8TB data drives and 3 3TB data drives. During the parity check, I would sit around 60-70 megs a second while the 3TB drives were being checked.  Once they were done being checked, the remaining 7 8TB drives sped up to 120 megs a second.

 

It seems like there is a max amount of data that I am allowed to push across the bus. My CPU / MB allows 40 PCIe lanes, so I don't believe that I'm running up against that.

 

Not sure if I have a configuration issue, a hardware issue, something else or if this is just normal in larger drive configurations.  Diagnostics attached.  Any help would be appreciated.

hex-diagnostics-20170714-1557.zip

[SSD]

I can understand that parity checks might get slower, and that so much parallel processing might slow things down with higher drive counts.

 

But individual drive I/O should not be affected.

 

Are you seeing individual drive read and write speeds being affected?

[Excessus]

10 minutes ago, bjp999 said:

I can understand that parity checks might get slower, and that so much parallel processing might slow things down with higher drive counts.

 

But individual drive I/O should not be affected.

 

Are you seeing individual drive read and write speeds being affected?

 

Individual drives are not affected.  So reading a file is okay, but writing to the array (which requires writes to both parity drives and reads from all drives) is as slow as parity check.

 

[SSD]

So turbo writes are slower. Make sense.

[JorgeB]

2 things come to mind, make sure none of the HBAs is on slot 2 (second one from the top), set you're tunables like so:

 

md_num_stripes: 4096

md_sync_window: 2048 

md_sync_thresh: 2000

[Excessus]

2 minutes ago, bjp999 said:

So turbo writes are slower. Make sense.

 

I suppose.  I guess my question is; what is the bottleneck in specific technical terms.  I have plenty of CPU, RAM and PCIe lanes.  Is there any way to get the speed back?

[Excessus]

3 hours ago, johnnie.black said:

2 things come to mind, make sure none of the HBAs is on slot 2 (second one from the top), set you're tunables like so:

 

md_num_stripes: 4096

md_sync_window: 2048 

md_sync_thresh: 2000

 

I made these changes and don't really see any difference.  Do I need to restart to see an effect?

 

Here is my slot configuration:

 

1 - low end graphics card

2 - empty

3 - Dell PERC H310

4 - GTX 1070

5 - empty

6 - Dell PERC H310

[JorgeB]

6 hours ago, Excessus said:

3 - Dell PERC H310

 

By the 2nd slot from the top I meant the 2nd PCIe x16 slot, it works at x1 or x4 max, HBA should be in another slot (or at least make sure it's operating @ x4) or it will be bandwidth limited.

 

1 x PCIe 2.0 x16 (x4 mode) *³

 

3* The PCIe x16_2 shares bandwidth with USB 3.1 and PCIe x1_2 slot. By default, the PCIe x16_2 slot automatically runs at x1 mode.

Edited July 15, 2017 by johnnie.black

[Excessus]

19 hours ago, johnnie.black said:

 

By the 2nd slot from the top I meant the 2nd PCIe x16 slot, it works at x1 or x4 max, HBA should be in another slot (or at least make sure it's operating @ x4) or it will be bandwidth limited.

 

1 x PCIe 2.0 x16 (x4 mode) *³

 

3* The PCIe x16_2 shares bandwidth with USB 3.1 and PCIe x1_2 slot. By default, the PCIe x16_2 slot automatically runs at x1 mode.

 

Thanks!  I moved some cards around and this seems to have made a world of difference!  My write speeds are now around 130 megs a second!

 

I guess it pays to read the damn motherboard manual sometimes.

 

[SSD]

20 hours ago, johnnie.black said:

 

By the 2nd slot from the top I meant the 2nd PCIe x16 slot, it works at x1 or x4 max, HBA should be in another slot (or at least make sure it's operating @ x4) or it will be bandwidth limited.

 

1 x PCIe 2.0 x16 (x4 mode) *³

 

3* The PCIe x16_2 shares bandwidth with USB 3.1 and PCIe x1_2 slot. By default, the PCIe x16_2 slot automatically runs at x1 mode.

 

Excellent catch!