[Plugin] Parity Check Tuning


367 posts in this topic Last Reply

Recommended Posts

  • Replies 366
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Parity Check Tuning plugin   The Parity Check Tuning plugin is  primarily designed to allow you to split a parity check into increments and then specify when those increments should be run.

i have been working through all cases where this can happen in the code and I think I now have them all fixed in the version running on my test server.  There was a number of places in the code where

I am currently working on the code to allow array operations to be restarted (resumed) from where they were as long as: the array was shutdown cleanly there have been no changes to the

Posted Images

For general reference the diagnostic information that @vw-kombisupplied showed that the plugin is not updating the history correctly when errors were reported/corrected during the parity check.  The history will always show 0 errors until I issue a corrected version of the plugin. 

Link to post

The plugin should low update the error count in the parity history for any future checks.

 

In addition I have exposed some functionality at the CLI level that some may find helpful.   

Usage: parity.check <action>
where action is one of
  pause            Pause a rumnning parity check
  resume           Resume a paused parity check
  check            Start a parity check (as Settings->Scheduler)
  correct          Start a correcting parity check
  nocorrect        Start a non-correcting parity check
  status           Show the status of a running parity check
  cancel           Cancel a running parity check

I would be interested any feedback on this feature.

Edited by itimpi
Link to post
13 minutes ago, wgstarks said:

Has the statistics report (time, speed, etc) been updated to carryover after a pause, or is it still just for the currently running check?

 

Just trying to keep track of the changes.😁

Not quite sure what you are asking?   If you are talking about the parity history then the plugin has always updated it to be correct for the run as a whole - not just the last increment.   It was already doing this for duration, speed etc, but there was a bug (now fixed) where this was not being done for the error count.  

 

While writing this it has just occurred to me that the parity check history does not currently show whether a run was correcting or non-correcting - would it be useful to have this displayed in the history as well? It should be an enhancement I could make to the plugin easily enough.

Link to post

I know that originally if a check was paused the elapsed time would reset to zero after the check was resumed. I had assumed this would also effect the averaged speed, but that may be a mistake on my part, and possibly this has all been corrected already.

Link to post
11 minutes ago, wgstarks said:

I know that originally if a check was paused the elapsed time would reset to zero after the check was resumed. I had assumed this would also effect the averaged speed, but that may be a mistake on my part, and possibly this has all been corrected already.

you need to distinguish what the core Unraid system does and what the plugin does.    Core Unraid seems to reset for each increment, but when the plugin is handling the pause/resume process it keeps track of how each increment went and then on completion it calculates the figures for the run as a whole and this is what ends up in the history.

Link to post
17 minutes ago, wgstarks said:

that may be a mistake on my part,

Well then, I was correct. It was a mistake on my part.😀

 

Thanks for the explanation. I know there isn’t much the plugin can do to correct the core functionality, and honestly, the stats for a running check have never been very accurate due to the speed changes across the disk itself (and I’m sure many other factors).

Link to post
47 minutes ago, jmcconathy said:

So far this seems to function as expected. I have issues with heat, particularly during parity checks, on my system and have enabled that feature as well to see if it will help. 

I will be interested to see how well this works.   My tests were a little artificial as my system does not suffer from heat problems.

 

if you have any problems then please enable the debugging log option in the plugin settings and then let me have a copy of the syslog so that I can see what is going on.

Link to post
  • 1 month later...

Hi there. Thanks for your time creating this plugin, I'm excited to try it out. My current parity checks only take about 10 hours, but I'm looking at upgrading to some bigger drives this fall, so this plugin should come in handy.

 

Based on my settings in my screenshot, I'm expecting the parity check to start Sept 30th (last day of the month) at 10pm, then pause at 4am on Oct 1st, then resume Oct 1st at 10pm, and repeat until finished. Does that look right?

parity_tuning1.png

Link to post

Yes - you should get the behaviour you want.   In this case the check should finish during the second increment if there are no problems.   With bigger drives you would get more increments as the total execution time increases.

Link to post
On 9/29/2019 at 6:20 AM, itimpi said:

Yes - you should get the behaviour you want.   In this case the check should finish during the second increment if there are no problems.   With bigger drives you would get more increments as the total execution time increases.

Worked perfectly, thanks!

Link to post
  • 3 weeks later...

I just finished using this plugin to split up a monthly parity check. I have my normal parity check schedule set to correct errors, however, both the notifications during the parity check and the parity.check.tuning.progress.save file are showing non-correcting parity checks.

 

paritychecksettings.png

 

type|date|time|sbSynced|sbSynced2|sbSyncErrs|sbSyncExit|mdState|mdResync|mdResyncPos|mdResyncSize|mdResyncCorr|mdResyncAction|Description
STARTED|2019 Oct 20 00:00:01|1571544001|1570408676|1570493018|0|0|STARTED|0|0|7814026532|0|check P|Non-Correcting Parity Check|
PAUSE|2019 Oct 20 07:30:01|1571571001|1571544001|0|0|0|STARTED|7814026532|2851442360|7814026532|1|check P|Non-Correcting Parity Check|
RESUME|2019 Oct 21 00:00:06|1571630406|1571630401|0|0|0|STARTED|7814026532|2851930044|7814026532|1|check P|Non-Correcting Parity Check|
PAUSE|2019 Oct 21 07:30:01|1571657401|1571630401|0|0|0|STARTED|7814026532|5397770972|7814026532|1|check P|Non-Correcting Parity Check|
RESUME|2019 Oct 22 00:00:06|1571716806|1571716801|0|0|0|STARTED|7814026532|5398227420|7814026532|1|check P|Non-Correcting Parity Check|
COMPLETED|2019 Oct 22 07:30:01|1571743801|1571716801|1571741144|0|0|STARTED|0|0|7814026532|1|check P|Non-Correcting Parity Check|

 

Link to post

I’ll check this out.    Showing the type of the parity check in the history is a recent addition to the plugin and it is possible it could be getting it wrong.  
 

Note that the plugin specifically does NOT use the configured setting but attempts to derive it from other sources during the parity check.   This is because there are scenarios under which the check cannot do exactly what is scheduled and it silently runs a reduced variant of the check.

 

EDIT:  Found the cause - silly mistake on my part.   Will check out the fix and then release an updated plugin

Edited by itimpi
Link to post

@itimpi I do not know if what I want to do is possible, but it was suggested to me in another post that this plugin might allow it. I would like to schedule a parity check every 4 weeks on Saturdays. This would allow me to set it up on a schedule that would not interfere with duplicati's upload schedule. I know they can both run at the same time, but I have been having issues with duplicati hanging, and am trying to reduce any possible interferences. I know there is options for weekly that I can set up as first, last, second, third, fourth, but I think that is referring to the weeks in the month not the number from the last run. What I mean is that if I select weekly and 2nd week it would be the second week of each month, not every 2 weeks. I want it to run specifically on Saturday every 28 days. Is this possible?

Link to post
2 hours ago, jebusfreek666 said:

@itimpi I do not know if what I want to do is possible, but it was suggested to me in another post that this plugin might allow it. I would like to schedule a parity check every 4 weeks on Saturdays. This would allow me to set it up on a schedule that would not interfere with duplicati's upload schedule. I know they can both run at the same time, but I have been having issues with duplicati hanging, and am trying to reduce any possible interferences. I know there is options for weekly that I can set up as first, last, second, third, fourth, but I think that is referring to the weeks in the month not the number from the last run. What I mean is that if I select weekly and 2nd week it would be the second week of each month, not every 2 weeks. I want it to run specifically on Saturday every 28 days. Is this possible?

No, this plugin will not help.    It does not control when a check starts - it merely allows you to control the time slots that are used for running increments of the check.   You can only use the standard built-in scheduling option for controlling the start of checks.

Link to post
On 1/12/2020 at 11:33 PM, jebusfreek666 said:

parity check every 4 weeks on Saturdays.

Set the schedule up as custom.  Will do exactly what you want.  ie: the last saturday of the month, the 1st, 3rd etc.

Link to post
11 hours ago, Squid said:

Set the schedule up as custom.  Will do exactly what you want.  ie: the last saturday of the month, the 1st, 3rd etc.

But the 4th Saturday is not actually 28 days, unless I am missing something? For example, the 4th Saturday in January this year is 1/25. The 4th Saturday in February is 2/22. This is exactly 4 weeks apart. But the 4th Saturday in March is 3/28. This is not 28 days past the last check, as there is actually 5 Saturdays in the month prior. This would throw the schedule off. Unless I am understanding this wrong?

Link to post

Probably need to do the following 2 steps. Watch out for the second step, it's a dozy. 

 

1. Disable parity check schedule.

2. Write your own custom bash script scheduled to run every Saturday. Inside this custom script, determine the week of the year and if divisible by 4 then kick off a parity check, otherwise exit.

 

Hint for day of year (ISO, Monday start, 01-53)

/bin/date +%V

 

Link to post

I'm not certain of the parameter on the parity check, I know it's NOCORRECT for non-correcting parity check, so I'm guessing it's CORRECT for correcting parity check.

 


# get the week number of the year without any padding

weeknumber=$(date +%-V)

# see if it's an even multiple of 4 by using modulo arithmetic, 0 means multiple of 4

forthweek=$((($weeknumber)%4))

if [ "$forthweek" -eq "0" ]; then

    # invoke parity check here

     /usr/local/sbin/mdcmd check CORRECT &> /dev/null

fi

 

Link to post
15 minutes ago, BRiT said:

I'm not certain of the parameter on the parity check, I know it's NOCORRECT for non-correcting parity check, so I'm guessing it's CORRECT for correcting parity check.

 

 


# get the week number of the year without any padding

weeknumber=$(date +%-V)

# see if it's an even multiple of 4 by using modulo arithmetic, 0 means multiple of 4

forthweek=$((($weeknumber)%4))

if [ "$forthweek" -eq "0" ]; then

    # invoke parity check here

     /usr/local/sbin/mdcmd check CORRECT &> /dev/null

fi

 

 

I have never used user scipts, and before unraid had never seen linux. So this is completely greek to me. I will read up on it though. Thanks.

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.