Parity Rebuild extremely slow after replacing data drive? (~5Mb/s)

[Majawat]

My hard drives filled up (<1TB free), so I replaced an 8TB drive (a shucked WD80EFZX) with a 14TB drive (ST14000NT001 from serverpartdeals.com). I stopped the array, unassigned the device, shut down, replaced drive using the same power and SATA cable, started server, assigned drive, and started the array.

 

But the parity rebuild is taking forever. Much longer than it has in the past, which would be a few days. But now it's between 30 and 80 days at times; around 3-5Mb/s. I have no idea how to troubleshoot this and I'm concerned I got a bad drive or something. 

 

Is there an issue I'm overlooking? What can I do to speed this rebuild up?

Any help would be greatly appreciated. Thank you. Dignostics attached.

 

hathor-diagnostics-20240229-1633.zip

[itimpi]

There seem to be a lot of reads and writes to several drives in the array that are not due to the parity operation.  These will severely degrade the parity operation so the way to speed it up is to stop whatever (person or application) is doing that array activity.

[Majawat]

Much appreciated, itimpi! Is there any way to figure out what's doing the reads/writes?

 

26 minutes ago, itimpi said:

stop whatever (person or application) is doing that array activity.

 

[itimpi]

Have you tried stopping all docker containers and/or VMs?

 

the File Activity plugin might give a clue by allowing you to see what files are being accessed.

[Majawat]

4 minutes ago, itimpi said:

Have you tried stopping all docker containers and/or VMs?

 

I'm doing that now, I actually paused the rebuild and the Mover started moving, so I'm going to let that finish before continuing.

 

5 minutes ago, itimpi said:

Have you tried stopping all docker containers and/or VMs?

 

the File Activity plugin might give a clue by allowing you to see what files are being accessed.

I'll check it out, thanks!

[itimpi]

Just now, Majawat said:

I'm doing that now, I actually paused the rebuild and the Mover started moving, so I'm going to let that finish before continuing.

If you have the Parity Check Tuning plugin installed then it has options to automatically pause array operations while mover or appdata backup are running (as they severely degrade performance if run in parallel with an array operation).