Painfully slow parity check speeds

[bdowden]

Hi all.

 

I’ve been trying to figure out why my parity check speeds go from 65-70 MBps to 5, seemingly at random times.

 

all of my drives are shucked 14tb WD element drives, 260TB total, with 2 parity. I have at least 25 drives not in the array, mainly for farming chia (20 or so), and then a few for other things.

 

My specs:

Supermicro MotherBoard MBD-X11DPI-NT-O

256gb ddr4

Supermicro CSE-847 45 bay chassis (JBOD)

Supermicro CSE-846A chassis (main server)

LSI-9201-16e HBA

Suoermicro CSE-216E26 2.5” chassis

 

most drives are in the JBOD enclosure but 5 are in the 846, including one parity drive, connected to the motherboard sata ports. The two chassis are connected via an external SFF8088 from the LSI to the 8088 connectors on the 847.

 

there are only 2 drives in the 216. 
 

No drives have smart errors and disk speed tests are fine.

 

Where my mind is going is the LSI adapter isn’t the best and I should switch to the PERC H200e. I think also the constant spin of the chia drives going over the same 8088 cables are causing a slow down of speeds for parity checks.

 

Is there anything else I should look at? Every time a parity check starts I dread seeing those super slow speeds.

[JorgeB]

Diagnostics saved during a slow period might show something.

[bdowden]

Ok. When I get back home I plan on removing the 2.5” chassis and replacing the LSI with the h200e; while I’m at it I’ll replace a drive that I’ve been meaning to.

 

but I’m general there shouldn’t be issues with unassigned drives and parity speeds as long as the hardware is good without issues?

[JorgeB]

Correct.

[bdowden]

Here are the diagnostics.

From what I saw in the smart reports I see 

Device-to-host register FISes sent due to a COMRESET

 

in almost all of them. I'm not sure which drives are in the 45 bay and which are in the main chassis, but the fact that the majority have this listed I'm going to assume it's the LSI HBA. This weekend I'll try to swap it out for the H200e and see if that fixes it.

 

And yes, I know some of the SSD's have smart errors - I'm slowly migrating off of those old SSD's but I haven't made my way to get rid of all of them.

tower2-diagnostics-20221012-1846.zip

[JorgeB]

There's something else reading from disk2.

[bdowden]

Yes there was. I had some docker app data directories on disk2 instead of on cache. Once I moved that to cache the parity check speeds are averaging 100MB/s. From what I've read that's about what I should be getting.

 

Thanks for your help!