Setup:

 

CPU - i7-6850K

MB - ASUS X99-A II

SAS Controllers - Dell PERC H310 x2

Drives - 11 8TB drives. 1 3TB drive. (10 data, 2 parity).

 

I have been adding drives to this system over time.  As I add more drives, parity checks and writes in general get slower and slower.  For a while, I had 7 8TB data drives and 3 3TB data drives. During the parity check, I would sit around 60-70 megs a second while the 3TB drives were being checked.  Once they were done being checked, the remaining 7 8TB drives sped up to 120 megs a second.

 

It seems like there is a max amount of data that I am allowed to push across the bus. My CPU / MB allows 40 PCIe lanes, so I don't believe that I'm running up against that.

 

Not sure if I have a configuration issue, a hardware issue, something else or if this is just normal in larger drive configurations.  Diagnostics attached.  Any help would be appreciated.

hex-diagnostics-20170714-1557.zip

I can understand that parity checks might get slower, and that so much parallel processing might slow things down with higher drive counts.

 

But individual drive I/O should not be affected.

 

Are you seeing individual drive read and write speeds being affected?

10 minutes ago, bjp999 said:

I can understand that parity checks might get slower, and that so much parallel processing might slow things down with higher drive counts.

 

But individual drive I/O should not be affected.

 

Are you seeing individual drive read and write speeds being affected?

 

Individual drives are not affected.  So reading a file is okay, but writing to the array (which requires writes to both parity drives and reads from all drives) is as slow as parity check.

 

So turbo writes are slower. Make sense.

2 things come to mind, make sure none of the HBAs is on slot 2 (second one from the top), set you're tunables like so:

 

md_num_stripes: 4096

md_sync_window: 2048 

md_sync_thresh: 2000

2 minutes ago, bjp999 said:

So turbo writes are slower. Make sense.

 

I suppose.  I guess my question is; what is the bottleneck in specific technical terms.  I have plenty of CPU, RAM and PCIe lanes.  Is there any way to get the speed back?

3 hours ago, johnnie.black said:

2 things come to mind, make sure none of the HBAs is on slot 2 (second one from the top), set you're tunables like so:

 

md_num_stripes: 4096

md_sync_window: 2048 

md_sync_thresh: 2000

 

I made these changes and don't really see any difference.  Do I need to restart to see an effect?

 

Here is my slot configuration:

 

1 - low end graphics card

2 - empty

3 - Dell PERC H310

4 - GTX 1070

5 - empty

6 - Dell PERC H310

6 hours ago, Excessus said:

3 - Dell PERC H310

 

By the 2nd slot from the top I meant the 2nd PCIe x16 slot, it works at x1 or x4 max, HBA should be in another slot (or at least make sure it's operating @ x4) or it will be bandwidth limited.

 

1 x PCIe 2.0 x16 (x4 mode) *³

 

3* The PCIe x16_2 shares bandwidth with USB 3.1 and PCIe x1_2 slot. By default, the PCIe x16_2 slot automatically runs at x1 mode.

Edited July 15, 20178 yr by johnnie.black

19 hours ago, johnnie.black said:

 

By the 2nd slot from the top I meant the 2nd PCIe x16 slot, it works at x1 or x4 max, HBA should be in another slot (or at least make sure it's operating @ x4) or it will be bandwidth limited.

 

1 x PCIe 2.0 x16 (x4 mode) *³

 

3* The PCIe x16_2 shares bandwidth with USB 3.1 and PCIe x1_2 slot. By default, the PCIe x16_2 slot automatically runs at x1 mode.

 

Thanks!  I moved some cards around and this seems to have made a world of difference!  My write speeds are now around 130 megs a second!

 

I guess it pays to read the damn motherboard manual sometimes.

 

20 hours ago, johnnie.black said:

 

By the 2nd slot from the top I meant the 2nd PCIe x16 slot, it works at x1 or x4 max, HBA should be in another slot (or at least make sure it's operating @ x4) or it will be bandwidth limited.

 

1 x PCIe 2.0 x16 (x4 mode) *³

 

3* The PCIe x16_2 shares bandwidth with USB 3.1 and PCIe x1_2 slot. By default, the PCIe x16_2 slot automatically runs at x1 mode.

 

Excellent catch!