How I reduced my parity check by 4h

[Alphahelix]

Hi all,

 

I don't know if this is common knowledge or not, but I didn't know about it, so maybe it can help one or two here.

 

As a side effect to decommission a bunch of drives (11 in total) I gained a speed bump in parity check. I thought about it and my conclusion (my conclusion my be wrong) is as following. All mechanical drives have a transfer (read and write) curve. From 0-50% full the speed is pretty much the same, from 50-75% the speed has gone down quite a bit and from 75-100% it's close to half the speed at 0%. And as I hope everyone know, It's always the slowest disk that sets the limit. So if you have a  mix different drive sizes you end up being hit by the 75-100% range many times, and that is where I think my bottleneck was...

 

Before 26h (largest disk 8tb):

8x500gb

2x1000gb

1x1500gb

1x6000gb

9x8000gb

so I had a total of 5 speed drops during parity check.

less than 100mb/s (Can't remember the exact speed)

 

After 22h (largest disk 10tb):

1x6000gb

10x8000gb

3x10000gb

now I "only" have a total of 3 speed drops during parity check.

122mb/s 

 

I not only reduced the parity check by approx 4h, I also increased the largest disk by 2tb... So I reduced the time by 15% while I at the same time increased the largest disk by 25%.

 

All this only by removing the speed drops caused by having many different hard disk capacity.

 

I hope it is not to geeky.

 

/Alphahelix

[Harro]

Another tool to help you see what is going on with drives and connections would be  @jbartlett disk speed docker. By using this docker and gradually replacing smaller drives with larger ones I decreased my parity time from 1 day, 7 hr, 57 min, 11 sec @34.8 MB/s on Feb.2017 to 18 hr, 29 min, 10 sec @150.3 MB/s  on Feb 2020.  I am quite happy with how everything is running.

 

[Alphahelix]

16 hours ago, Harro said:

Another tool to help you see what is going on with drives and connections would be  @jbartlett disk speed docker. By using this docker and gradually replacing smaller drives with larger ones I decreased my parity time from 1 day, 7 hr, 57 min, 11 sec @34.8 MB/s on Feb.2017 to 18 hr, 29 min, 10 sec @150.3 MB/s  on Feb 2020.  I am quite happy with how everything is running.

 

It was one of the tools I used. 

[witalit]

I have an old 1.5TB drive I need to replace ASAP as its not far off falling over. This tool has been a good indicator for me, my other drives are not too old. 

[TX-Boiler]

On 2/29/2020 at 1:33 PM, Harro said:

Thanks for this docker recommendation. I've got a 2TB WD Green drive with about 8.5 years on it that I've been thinking about replacing. This utility showed that it is by FAR the slowest drive my setup and probably should be gone - looks like I'll be upgrading. It's fascinating to see how much faster the newer 4TB drives (my max disk size right now) are vs. the old drives.

 

Another tool to help you see what is going on with drives and connections would be  @jbartlett disk speed docker. By using this docker and gradually replacing smaller drives with larger ones I decreased my parity time from 1 day, 7 hr, 57 min, 11 sec @34.8 MB/s on Feb.2017 to 18 hr, 29 min, 10 sec @150.3 MB/s  on Feb 2020.  I am quite happy with how everything is running.