Jump to content
itimpi

[Plugin] Parity Check Tuning

26 posts in this topic Last Reply

Recommended Posts

Posted (edited)

 

Parity Check Tuning plugin

 

The Parity Check Tuning plugin is designed to allow you to split a parity check into increments and then specify when those increments should be run.   It will be of particular use to those who have large parity drives so that the parity check takes a long time and who leave their Unraid servers powered on 24x7.   The idea is that you can specify time slots when the increments should be run and these can be chosen to be at times when the Unraid server is likely to be idle. 

 

As an example on my system I have a 10TB Parity Disk and an uninterrupted Parity Check takes about 30 hours to complete.. I have my normal scheduled parity checks set to run monthly.   By using this plugin to run using 3 hour increments the elapsed time extends to 10 days (10 x 3 = 30) but I do not notice any impact on my normal use as the increments are run when I am not using the system.    Once enough increments are run to complete the scheduled parity check then no further increments will be run until the time for the next scheduled check comes around.

 

The Settings page is added as an extra section to the Settings->Scheduler page (see the screenshot below) in the Unraid GUI as this seemed the most logical place for it to appear.   The initial release of the plugin allows you to specify a single daily time slot for running increments.    This seems to satisfy the basic Use Case but I am amenable to others making Use Cases for something more sophisticated. 

 

Debug feature

If you enable the option for debug logging then you will see reasonably verbose entries appearing in the syslog about how this plugin is functioning internally.   All these entries will include the word DEBUG so It is clear that they have been activated by turning on the Debug logging.  Although this feature is primarily aimed at tracking down any issues that might be reported and developing new features the entries will be meaningful to any users interested in such matters.

 

When this option is set to Yes then you are offered an additional option of Hourly for the frequency at which this plugin should pause/resume parity check increments.   This was added primarily to help with testing and to help track down any issues that users might experience in using the plugin.   Early feedback ahs suggested that users new to this plugin can use this feature as a way of getting a feel for how the plugin operates.

 

Built-in Help

The settings page for this plugin has built-in help to describe the meaning of the various settings.    You can click on the description text for any particular setting to toggle it on-/off for that particular setting or you can turn it on/off at the page level by using the standard Help toggle in the Unraid GUI.  Suggestions for improving the wording or expanding on the provided text are welcomed as it is not intended to produce any separate documentation.

 

Planned Enhancements

There are a few enhancements that are already planned (and on which work has started)

  • The settings screen currently has an entry as to whether parity checks started outside a normal scheduled one (e.g. manually started or started by the system after an unclean shutdown) should also be run in increments.    It is likely that in such a scenario the user may be interested in getting their array back to health as soon as possible and would like the check to run without interruption.   At the moment the you can only specify the Yes as the code to support the No option is not yet complete.
  • Improve the history kept about parity checks that are paused and resumed using this plugin so that actual running time and total elapsed time of the parity check are both tracked. 
  • Pause parity checks if disks overheat and resume them when they cool down.   Ideally an Unraid server has sufficient cooling that such a feature should not be required but anecdotal evidence suggests that a significant number of people have problems with systems over-heating under load.

Suggestions for other possibilities are always welcomed.

 

Wish List

This a holder for "blue sky" ideas that have been expressed for which there is no idea if it is even technically possible.   They are kept here as a reminder and for others to perhaps expand on, and even perhaps come up with ideas for implementation..

  • Auto detect idle periods:  The idea is that instead of the user having to specify specific start/stop times for running parity check increments the plugin should automatically detect periods when the system is idle to resume a parity check.   This would need the complementary option of automatically detecting the system is no longer idle so that the check can be paused.
  • Avoid running parity check if mover is running.  Mover and parity Checking severely degrade each others performance.   Some way of removing (or at least minimising) this conflict would be desirable.   Thee are lot of permutations that need to be thought through to come up with a sensible strategy.
  • Stop docker containers during a parity check.  The ability to schedule the parity check to stop specified docker containers prior to check running and restart the docker containers after the check is paused or completed.  A workaround for this would be to use the User Scripts plugin to do this although an integrated capability would be easier to use.
  • Resume parity checks on array start. The current Limetech implementation of pause/resume does not allow a parity check to be started from any point except the beginning.   If the ability to start at a defined offset is ever provided then this could be implemented.
  • Partial parity Checks: This is just a different facet of the ability to Resume parity checks on array start where you deliberately set the system up to perform part of a parity check with reboots in between the parts.

 

Feedback

Feedback from users on the facilities offered by this plugin is welcomed, and is likely to be used to guide the direction of any future enhancements.  It will be interesting to hear how useful users find this plugin to be in the normal running of their system.   Please feel free to suggest any changes that you think would enhance the experience even if it only rewording of text .

 

Requirements

  • Unraid 6.7 rc3 or later
  • Community Applications (CA) plugin.   It is expected that this plugin will be installed via the Apps tab (i.e. the Community Applications plugin) and the appropriate template has been prepared to allow CA to handle it tidily.

 

Installation

The parity Check tuning plugin is available for installation by using the Community Applications plugin.   If you navigate to the Apps tab and search for 'Parity Tuning' this plugin will show up and it can be installed from there.     Once the plugin is installed then if you go to Settings->Scheduler in the Unraid GUI you will see an extra section has appeared that allows you to specify the settings you want to be used for this plugin.

 

Restrictions/Limitations

  • This plugin does not initiate a parity check - it only provides facilities for pausing/resuming them according to the specified criteria.   If there is no parity check running during the timeslot specified for an increment then this plugin take no action. 
  • If the array is stopped for any reason then the current progress of a running parity check is lost.   This means that the next time the array is started the parity check will need to be restarted from the beginning.    This is a restriction imposed by the current Limetech implementation.   The plugin is designed so that this restriction can easily be removed if Limetech can provide a way of starting parity checks at a specified offset rather than starting all parity checks from the beginning.

 

 

 

paused.jpg

Untitled.jpg

Edited by itimpi
  • Like 1
  • Upvote 1

Share this post


Link to post

This looks really interesting.

 

Just some questions simply because I want to be first. Lol

 

How does it react if you write data to your array every day when it resumes? I'm guessing as long as the data is written to the parity drive and the target drives it doesn't matter.

 

Also does this send the same ques to https://forums.unraid.net/topic/70783-plugin-mover-tuning/

to let it know not to fire off the Mover or should the user simply make sure their schedule is set other than when the periodic mover is running?

 

Also could this be ran in stead of the monthly?

 

Share this post


Link to post

Parity in Unraid is always real time so it does not matter when data is written (although writes can slow down the parity check).    That is also true of the existing way parity checks run.

 

if you do not want mover to run at the same time then you control that in the mover settings as they already have an option to not run if a parity check is running.

 

This plugin is meant to run in conjunction with the existing Parity check settings, not replace them   The existing settings control when a parity check is to be initiated.    The settings for this plugin then determine how to split that check into increments to spread the check over a number of days to avoid impacting busy periods.

Share this post


Link to post
6 hours ago, itimpi said:

The settings screen currently has an entry to allow you to apply the increments to manually started parity checks.   At the moment I have not found a reliable way to detect this so the option cannot be turned off and may be removed in a future release.  It is an idea I intend to explore further before making a decision on this.

Same problem I had with the mover tuning plugin.  My solution was to rename the stock mover script, replace it with this

#!/bin/bash
PPPID=`ps h -o ppid= $PPID 2>/dev/null`
P_COMMAND=`ps h -o %c $PPPID 2>/dev/null`
/usr/local/emhttp/plugins/ca.mover.tuning/mover.php $P_COMMAND $1

and then have the mover.php detect if it was running via a cron schedule or not to determine if it was a manual start or scheduled (which is what the P_COMMAND var is)

$cron = ( $argv[1] == "crond" );

and then take appropriate action by calling it.

 

Probably more complicated for you as you would have to replace mdcmd with the wrapper script, and if it's not parity check related, pass through any and all options when you call the renamed mdcmd.  

 

But, food for thought.

Share this post


Link to post

Thank-you.  Going to start playing with this tonight.

Share this post


Link to post
Posted (edited)

 

On 3/2/2019 at 8:51 PM, Squid said:

Same problem I had with the mover tuning plugin.  My solution was to rename the stock mover script, replace it with this


#!/bin/bash
PPPID=`ps h -o ppid= $PPID 2>/dev/null`
P_COMMAND=`ps h -o %c $PPPID 2>/dev/null`
/usr/local/emhttp/plugins/ca.mover.tuning/mover.php $P_COMMAND $1

and then have the mover.php detect if it was running via a cron schedule or not to determine if it was a manual start or scheduled (which is what the P_COMMAND var is)


$cron = ( $argv[1] == "crond" );

and then take appropriate action by calling it.

 

Probably more complicated for you as you would have to replace mdcmd with the wrapper script, and if it's not parity check related, pass through any and all options when you call the renamed mdcmd.  

 

But, food for thought.

I have copied this idea for detecting when Scheduled Parity checks are initiated.   As you guessed it is the mdcmd command that needs wrapping/replacing.     I have already reached the stage of detecting the automatic start of a parity check but still need to do more work to take advantage of this information.

Edited by itimpi

Share this post


Link to post

Feature Request-

Add an option to pause parity checks if mover is running.

 

I normally run mover on a cron schedule so it’s not too complicated to manipulate the schedules for mover and this plugin so that they don’t overlap, but since I use the mover tuning plugin there are times when mover is run outside of the cron schedule. When mover and parity are both running it tends to bring all other functions on my server to a complete standstill.

Share this post


Link to post
18 minutes ago, wgstarks said:

Feature Request-

Add an option to pause parity checks if mover is running.

 

I normally run mover on a cron schedule so it’s not too complicated to manipulate the schedules for mover and this plugin so that they don’t overlap, but since I use the mover tuning plugin there are times when mover is run outside of the cron schedule. When mover and parity are both running it tends to bring all other functions on my server to a complete standstill.

I can see where you are coming from, but the exact rules you are looking for are not clear.    I will have to put some thought into what appears to be some sensible rules for comment.  Taking into account the cron schedule for mover is probably quite easy. However what if mover is triggered manually - should the parity check detect this and pause the parity check.   What if the user starts both mover and parity check manually - what are the rules for determining who wins?  

 

Once a valid set of rules can be derived then I think something could be arranged.    However I do not want to rush into something like this before the interactions have been thought out and how they will be resolved sorted out.

 

As you will see from the first post there are already some new features I am working on.  I will add this one to the "Wish List" section as I do not want to commit to doing it until I have a better understanding of exactly what it will entail.   That means it will not get forgotten and there is time for for further thought.

Share this post


Link to post
7 minutes ago, itimpi said:

I will add this one to the "Wish List" section as I do not want to commit to doing it until I have a better understanding of exactly what it will entail.

Perfect

Share this post


Link to post

Not sure if this is just a random artifact or if it's supposed to be there. Seeing it in Safari and Firefox.

 

SafariScreenSnapz157.thumb.jpg.44aaa704b4ff80e8182563d352dc5ccf.jpg

  • Upvote 1

Share this post


Link to post
Posted (edited)

Do i understand correct.


Lets say parity should run on 14.03.19 12:00

 

with this plugin i could say, okay, run 3 hours every night after that date, until its finished? Then nothing happens until the next parity is startet?

 

.nice. idea!

Edited by nuhll

Share this post


Link to post
Posted (edited)
3 hours ago, nuhll said:

Do i understand correct.


Lets say parity should run on 14.03.19 12:00

 

with this plugin i could say, okay, run 3 hours every night after that date, until its finished? Then nothing happens until the next parity is startet?

 

.nice. idea!

Yes, that is the behaviour to expect. I have added a more detailed example to the first post in this thread to make that clearer.   I will also add the example to the help text in the GUI when I update the plugin.

 

It does rely on the fact that the interval between scheduled checks is long enough that the daily increnent size Is large enough to add up to enough time to complete one check before the next one is scheduled to start.   I must admit I have not (yet anyway) tested what happens if this assumption is not true.  

 

I had assumed that people did not run scheduled parity checks too frequently because of both the length of time individual checks take and the load on the system while they are running.   I think that the behaviour is likely to be that the check in progress simply continues instead of starting again from the beginning but this does need validating.  If not the assumption needs to be clearly stated to stop people setting the increments to be too small.

Edited by itimpi

Share this post


Link to post
2 hours ago, wgstarks said:

Not sure if this is just a random artifact or if it's supposed to be there. Seeing it in Safari and Firefox.

 

SafariScreenSnapz157.thumb.jpg.44aaa704b4ff80e8182563d352dc5ccf.jpg

I am seeing it too when I looked closely enough (perhaps I need new glasses :) ).     It is something that slipped past my radar and easily fixed.   At one point I had some extra fields present on that screen to help with testing and left a tiny bit of text behind when I removed them.

  • Upvote 1

Share this post


Link to post
On 3/14/2019 at 3:21 AM, itimpi said:

Yes, that is the behaviour to expect. I have added a more detailed example to the first post in this thread to make that clearer.   I will also add the example to the help text in the GUI when I update the plugin.

 

It does rely on the fact that the interval between scheduled checks is long enough that the daily increnent size Is large enough to add up to enough time to complete one check before the next one is scheduled to start.   I must admit I have not (yet anyway) tested what happens if this assumption is not true.  

 

I had assumed that people did not run scheduled parity checks too frequently because of both the length of time individual checks take and the load on the system while they are running.   I think that the behaviour is likely to be that the check in progress simply continues instead of starting again from the beginning but this does need validating.  If not the assumption needs to be clearly stated to stop people setting the increments to be too small.

Yeah, very nice idea, thanks for this.

 

I run parity every 3 months, so i guess, that should be done in 1-3 days 

Share this post


Link to post
Posted (edited)

Is it possible to pause pause the parity check when the disks are getting hot? Best case fans are not working well when the room is too hot (summer).

 

And is it possible to implement a button on the main page to do a partial parity check on demand? My server only runs when needed and sometimes only 1h a day or less

 

Nice feature btw! :)

Edited by Marino

Share this post


Link to post
13 minutes ago, Marino said:

Is it possible to pause pause the parity check when the disks are getting hot? Best case fans are not working well when the room is too hot (summer).

 

And is it possible to implement a button on the main page to do a partial parity check on demand? My server only runs when needed and sometimes only 1h a day or less

 

Nice feature btw! :)

 

In this case I would run Parity when its at the coolest part of the day. Maybe early hours of the morning. I know here where I live its normally the coolest time of the day just before Sunrise. You could set your server up to run a couple of hours for a few days at that time frame.

Share this post


Link to post
10 minutes ago, Marino said:

Is it possible to pause pause the parity check when the disks are getting hot? Best case fans are not working well when the room is too hot (summer).

 

And is it possible to implement a button on the main page to do a partial parity check on demand? My server only runs when needed and sometimes only 1h a day or less

 

Nice feature btw! :)

The idea of taking into account disk temperature is an interesting one, but how practical it might be I have no idea.   I suspect it is one of those that sounds a good idea at first but then not too practical after all, and that the only real solution is to improve the cooling inside your server’s case.

 

Partial paritcy checks is not a feature that can currently be implemented as there is no way to start except at the start.   It will need Limetech to provide a way for developers to start a parity check at a defined offset to make it possible.   Whether this will ever happen I have no idea.

Share this post


Link to post
Posted (edited)

Coolest part of the day are here (germany) in the middle of the night. I'd to run the server then. My 12TB drives are getting hot while checking parity, thats why I wished me to pause while the drives are to hot and when it takes a lot longer but keeps the drives cool.

 

edit:

@ itimpi

the problem is, when the case has 4 fans and the drives are in the coolest area just behind that fans the drives (3x12TB) are getting hot anyway.

To improve that I placed the pc on an open window in the winter. But when the room is too hot (summer) I need an AC to cool it down.

 

But maybe it is not a big of a deal to pause the parity check when one drive reaches an unsecure level of temperature.

 

That is THE reason for me to run parity checks not that often.

Edited by Marino

Share this post


Link to post

Not sure why, but the image for the Plugin isn't loading in my install. Not sure if its a glitch on my end or simply a missing image. I'm aware you don't click on it because you go into settings and use the schedule feature, but just thought it was kinda odd. 

 

Capture.JPG

 

Share this post


Link to post

The plugin works only on schedule right?

 

When my server is not powered on at a specific time it would be nice to have an option to check now for ... minutes/hours. Or better. Check for an hour and shutdown after that :)

Share this post


Link to post
1 hour ago, Marino said:

The plugin works only on schedule right?

 

When my server is not powered on at a specific time it would be nice to have an option to check now for ... minutes/hours. Or better. Check for an hour and shutdown after that :)

Not quite sure what you are asking for here :(  What are you thinking should be checked for?   The plugin currently requires a parity check to be actively running (although it can be paused) before it gets involved

Share this post


Link to post
Posted (edited)

I know, but it requires when adding the schedule that the server is running during that time. My server somtimes does not run for a week or so. For this it would be great to check partly the parity and then shutdown (without schedule).

 

Or am I wrong that the schedule is needed to use this plugin?

Edited by Marino

Share this post


Link to post
Posted (edited)
58 minutes ago, Marino said:

I know, but it requires when adding the schedule that the server is running during that time. My server somtimes does not run for a week or so. For this it would be great to check partly the parity and then shutdown (without schedule).

 

Or am I wrong that the schedule is needed to use this plugin?

Two points:

- This plugin does not require a scheduled parity check to be running when the times for its increments to run come around.  The parity check could have been started manually or for some other reason (such as an unclean shutdown).   The increment pause/resume times do need to be scheduled.

- Unraid does not (yet anyway) provide a mechanism that would allow partial parity checks to be run.   Parity checks always start from the beginning.    All this plugin does is allow a check to be stretched over days only running in scheduled idle periods (typically overnight).

 

The plugin's current capabilities are therefore targeting those who leave their Unraid servers powered on overnight and have large enough parity drives that the check currently takes long enough that it impacts performance at times when the Unraid server is not going to be idle.

 

The sort of capability I think you are asking for can only be implemented if/when Limetech provide a capability for parity checks to be started at a defined offset rather than only from the beginning.

Edited by itimpi

Share this post


Link to post
Posted (edited)

My Bad, sorry, i thought the check will resume at the point it was paused last time.

 

So the schedule is only for pause the parity check and the server has to be powered on until the check is done completely? Otherwise the beginning is very good checked but not the rest and the parity check can not finish, right?

 

Then a pause when one of the drives reaches a defined temperature would be even better. When the server has to be powered on all the time this could be useful to cool the system down a little while in pause.

When there is a possibility to implement this.

 

Thank you for the clarification. 

Edited by Marino

Share this post


Link to post
1 minute ago, Marino said:

My Bad, sorry, i thought the check will resume at the point it was paused last time.

Unfortunately not.   I have designed the plugin so that if Limetech provide the required restart capability the plugin will be able to take advantage of it.

2 minutes ago, Marino said:

So the schedule is only for pause the parity check and the server has to be powered on until the check is done completely? Otherwise the beginning is very good checked but not the rest and the parity check can not finish, right?

Yes.

2 minutes ago, Marino said:

Then a pause when one of the drives reaches a defined temperature would be great. When the server has to be powered on all the time this could be useful to cool the system down a little while in pause.

This is something I am currently trying to implement.   I am not sure how hard it is going to be to get it working well.   The idea is that this would be a completely independent setting to the scheduled pause/resume capability that is already implemented so it would always apply to a parity check at any time regardless of how it was initiated once you have set the option to activate this.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now