July 14, 20179 yr Setup: CPU - i7-6850K MB - ASUS X99-A II SAS Controllers - Dell PERC H310 x2 Drives - 11 8TB drives. 1 3TB drive. (10 data, 2 parity). I have been adding drives to this system over time. As I add more drives, parity checks and writes in general get slower and slower. For a while, I had 7 8TB data drives and 3 3TB data drives. During the parity check, I would sit around 60-70 megs a second while the 3TB drives were being checked. Once they were done being checked, the remaining 7 8TB drives sped up to 120 megs a second. It seems like there is a max amount of data that I am allowed to push across the bus. My CPU / MB allows 40 PCIe lanes, so I don't believe that I'm running up against that. Not sure if I have a configuration issue, a hardware issue, something else or if this is just normal in larger drive configurations. Diagnostics attached. Any help would be appreciated. hex-diagnostics-20170714-1557.zip
July 14, 20179 yr I can understand that parity checks might get slower, and that so much parallel processing might slow things down with higher drive counts. But individual drive I/O should not be affected. Are you seeing individual drive read and write speeds being affected?
July 14, 20179 yr Author 10 minutes ago, bjp999 said: I can understand that parity checks might get slower, and that so much parallel processing might slow things down with higher drive counts. But individual drive I/O should not be affected. Are you seeing individual drive read and write speeds being affected? Individual drives are not affected. So reading a file is okay, but writing to the array (which requires writes to both parity drives and reads from all drives) is as slow as parity check.
July 14, 20179 yr 2 things come to mind, make sure none of the HBAs is on slot 2 (second one from the top), set you're tunables like so: md_num_stripes: 4096 md_sync_window: 2048 md_sync_thresh: 2000
July 14, 20179 yr Author 2 minutes ago, bjp999 said: So turbo writes are slower. Make sense. I suppose. I guess my question is; what is the bottleneck in specific technical terms. I have plenty of CPU, RAM and PCIe lanes. Is there any way to get the speed back?
July 15, 20179 yr Author 3 hours ago, johnnie.black said: 2 things come to mind, make sure none of the HBAs is on slot 2 (second one from the top), set you're tunables like so: md_num_stripes: 4096 md_sync_window: 2048 md_sync_thresh: 2000 I made these changes and don't really see any difference. Do I need to restart to see an effect? Here is my slot configuration: 1 - low end graphics card 2 - empty 3 - Dell PERC H310 4 - GTX 1070 5 - empty 6 - Dell PERC H310
July 15, 20179 yr 6 hours ago, Excessus said: 3 - Dell PERC H310 By the 2nd slot from the top I meant the 2nd PCIe x16 slot, it works at x1 or x4 max, HBA should be in another slot (or at least make sure it's operating @ x4) or it will be bandwidth limited. 1 x PCIe 2.0 x16 (x4 mode) *3 3* The PCIe x16_2 shares bandwidth with USB 3.1 and PCIe x1_2 slot. By default, the PCIe x16_2 slot automatically runs at x1 mode. Edited July 15, 20179 yr by johnnie.black
July 16, 20179 yr Author 19 hours ago, johnnie.black said: By the 2nd slot from the top I meant the 2nd PCIe x16 slot, it works at x1 or x4 max, HBA should be in another slot (or at least make sure it's operating @ x4) or it will be bandwidth limited. 1 x PCIe 2.0 x16 (x4 mode) *3 3* The PCIe x16_2 shares bandwidth with USB 3.1 and PCIe x1_2 slot. By default, the PCIe x16_2 slot automatically runs at x1 mode. Thanks! I moved some cards around and this seems to have made a world of difference! My write speeds are now around 130 megs a second! I guess it pays to read the damn motherboard manual sometimes.
July 16, 20179 yr 20 hours ago, johnnie.black said: By the 2nd slot from the top I meant the 2nd PCIe x16 slot, it works at x1 or x4 max, HBA should be in another slot (or at least make sure it's operating @ x4) or it will be bandwidth limited. 1 x PCIe 2.0 x16 (x4 mode) *3 3* The PCIe x16_2 shares bandwidth with USB 3.1 and PCIe x1_2 slot. By default, the PCIe x16_2 slot automatically runs at x1 mode. Excellent catch!
Archived
This topic is now archived and is closed to further replies.