Hello,

 

I initiated a parity rebuild due to the replacement of a failing drive. The rebuild started relatively quickly but eroded to ~30-40MB/s after the first day. Unfortunately for the last two days the rebuild has been averaging aroun 2-3 MB/s. I'm not seeing any driver related errors in the logs. I have shutdown all VM and dockers services and have disabled all shares, but am having still seeing this exceptionally slow speed. 

 

I previously ran a drive speed check and did see that disk4 (the zfs disk I have in the array) had an average speed of <20% the speed of the rest of the drives. I assume this is part of the issue and want to test out migrating this disk away from zfs and potentially removing fromt he server entirely, but I need to finish this parity rebuild first.

 

What else should I be considering to try and get this parity rebuild back to a relatively (or at least viable) speed.

citizenur-diagnostics-20240516-2334.zip

 

There may be something else reading from disk4, you can also run the diskspeed docker to confirm all disks are performing well, though it will only test read speed.

10 minutes ago, JorgeB said:

There may be something else reading from disk4, you can also run the diskspeed docker to confirm all disks are performing well, though it will only test read speed.

 

THanks much - I tried exploring that, but the file activity pluggin wasn't showing any disk activity. Also, when I look at the read / write speed per disk it looks like the parity writes are equivalent to the reads on all of the other disks except for disk 4 which is showing a slightly higher read speed.

 

I ran the diskspeed test docker prior to starting the parity build and did find disk 4 operating at a much lower performance than the other drives but nothing even close to this slow. Do you think its prudent to run the diskspeed test again while still running the parity rebuild?

 

One other item of note the CPU load on the system seems quite high given the fact that I don't have docker or vm services on - anything I should look at there that may be related to the slow rebuild speed?

 

 

31 minutes ago, citizen_y said:

Do you think its prudent to run the diskspeed test again while still running the parity rebuild?

You can run it again if you pause the rebuild.

 

31 minutes ago, citizen_y said:

One other item of note the CPU load on the system seems quite high

I agree, and SHFS is using CPU suggesting there's something assessing the shares, or it got stuck, you can also try rebooting.

If I pause the rebuild, and reboot will the build restart or will it maintain its position?

I would try turinng off the Docker service, and making sure nothing on the network is trying to read from it.

 

A reboot will wipe the status and start the check back from the start unless you have the parity check tuning plugin and the resume function enabled in it, but not sure if you can install/enable that and have it work after the check has already been started.

Edited May 17, 20242 yr by Kilrah

Just a quick update - I really struggled to solve this, but read some other posts of people having issues with a slow performing zfs drive in the array so I decided to try and migrate my one zfs drive to xfs. Turns out this was a struggle, even copying the content from the zfs drive to another xfs formatted drive the best performance I could get was 2-3 Mb/s. I finally buckled down and copied only the irreplaceable content and blew up the rest when I erased that drive and reformatted under xfs. After doing so, I ran another speed check and voila, the drive was now showing speeds on speed tests that were inline with the other drives (4-5x what it had been under zfs). 

 

After that, I started another parity build, its not done yet, but its been maintaining speeds of 170-260 MB/s (inly really slowing as it comes to the later portions of any given drive size int he array). I'll report back when complete to confirm this was the fix - but right now its looking like another case of a a zfs drive in an array causing exceedingly slow performance.

 

 

Filesystem in use should never affect the parity sync/check speed, even if it doesn't have a filesystem it will perform the same, so possibly something else going on.