[Plugin] Parity Check Tuning


Recommended Posts

This looks really interesting.

 

Just some questions simply because I want to be first. Lol

 

How does it react if you write data to your array every day when it resumes? I'm guessing as long as the data is written to the parity drive and the target drives it doesn't matter.

 

Also does this send the same ques to https://forums.unraid.net/topic/70783-plugin-mover-tuning/

to let it know not to fire off the Mover or should the user simply make sure their schedule is set other than when the periodic mover is running?

 

Also could this be ran in stead of the monthly?

 

Link to comment

Parity in Unraid is always real time so it does not matter when data is written (although writes can slow down the parity check).    That is also true of the existing way parity checks run.

 

if you do not want mover to run at the same time then you control that in the mover settings as they already have an option to not run if a parity check is running.

 

This plugin is meant to run in conjunction with the existing Parity check settings, not replace them   The existing settings control when a parity check is to be initiated.    The settings for this plugin then determine how to split that check into increments to spread the check over a number of days to avoid impacting busy periods.

Link to comment
6 hours ago, itimpi said:

The settings screen currently has an entry to allow you to apply the increments to manually started parity checks.   At the moment I have not found a reliable way to detect this so the option cannot be turned off and may be removed in a future release.  It is an idea I intend to explore further before making a decision on this.

Same problem I had with the mover tuning plugin.  My solution was to rename the stock mover script, replace it with this

#!/bin/bash
PPPID=`ps h -o ppid= $PPID 2>/dev/null`
P_COMMAND=`ps h -o %c $PPPID 2>/dev/null`
/usr/local/emhttp/plugins/ca.mover.tuning/mover.php $P_COMMAND $1

and then have the mover.php detect if it was running via a cron schedule or not to determine if it was a manual start or scheduled (which is what the P_COMMAND var is)

$cron = ( $argv[1] == "crond" );

and then take appropriate action by calling it.

 

Probably more complicated for you as you would have to replace mdcmd with the wrapper script, and if it's not parity check related, pass through any and all options when you call the renamed mdcmd.  

 

But, food for thought.

Link to comment

 

On 3/2/2019 at 8:51 PM, Squid said:

Same problem I had with the mover tuning plugin.  My solution was to rename the stock mover script, replace it with this


#!/bin/bash
PPPID=`ps h -o ppid= $PPID 2>/dev/null`
P_COMMAND=`ps h -o %c $PPPID 2>/dev/null`
/usr/local/emhttp/plugins/ca.mover.tuning/mover.php $P_COMMAND $1

and then have the mover.php detect if it was running via a cron schedule or not to determine if it was a manual start or scheduled (which is what the P_COMMAND var is)


$cron = ( $argv[1] == "crond" );

and then take appropriate action by calling it.

 

Probably more complicated for you as you would have to replace mdcmd with the wrapper script, and if it's not parity check related, pass through any and all options when you call the renamed mdcmd.  

 

But, food for thought.

I have copied this idea for detecting when Scheduled Parity checks are initiated.   As you guessed it is the mdcmd command that needs wrapping/replacing.     I have already reached the stage of detecting the automatic start of a parity check but still need to do more work to take advantage of this information.

Edited by itimpi
Link to comment

Feature Request-

Add an option to pause parity checks if mover is running.

 

I normally run mover on a cron schedule so it’s not too complicated to manipulate the schedules for mover and this plugin so that they don’t overlap, but since I use the mover tuning plugin there are times when mover is run outside of the cron schedule. When mover and parity are both running it tends to bring all other functions on my server to a complete standstill.

Link to comment
18 minutes ago, wgstarks said:

Feature Request-

Add an option to pause parity checks if mover is running.

 

I normally run mover on a cron schedule so it’s not too complicated to manipulate the schedules for mover and this plugin so that they don’t overlap, but since I use the mover tuning plugin there are times when mover is run outside of the cron schedule. When mover and parity are both running it tends to bring all other functions on my server to a complete standstill.

I can see where you are coming from, but the exact rules you are looking for are not clear.    I will have to put some thought into what appears to be some sensible rules for comment.  Taking into account the cron schedule for mover is probably quite easy. However what if mover is triggered manually - should the parity check detect this and pause the parity check.   What if the user starts both mover and parity check manually - what are the rules for determining who wins?  

 

Once a valid set of rules can be derived then I think something could be arranged.    However I do not want to rush into something like this before the interactions have been thought out and how they will be resolved sorted out.

 

As you will see from the first post there are already some new features I am working on.  I will add this one to the "Wish List" section as I do not want to commit to doing it until I have a better understanding of exactly what it will entail.   That means it will not get forgotten and there is time for for further thought.

Link to comment
3 hours ago, nuhll said:

Do i understand correct.


Lets say parity should run on 14.03.19 12:00

 

with this plugin i could say, okay, run 3 hours every night after that date, until its finished? Then nothing happens until the next parity is startet?

 

.nice. idea!

Yes, that is the behaviour to expect. I have added a more detailed example to the first post in this thread to make that clearer.   I will also add the example to the help text in the GUI when I update the plugin.

 

It does rely on the fact that the interval between scheduled checks is long enough that the daily increnent size Is large enough to add up to enough time to complete one check before the next one is scheduled to start.   I must admit I have not (yet anyway) tested what happens if this assumption is not true.  

 

I had assumed that people did not run scheduled parity checks too frequently because of both the length of time individual checks take and the load on the system while they are running.   I think that the behaviour is likely to be that the check in progress simply continues instead of starting again from the beginning but this does need validating.  If not the assumption needs to be clearly stated to stop people setting the increments to be too small.

Edited by itimpi
Link to comment
2 hours ago, wgstarks said:

Not sure if this is just a random artifact or if it's supposed to be there. Seeing it in Safari and Firefox.

 

SafariScreenSnapz157.thumb.jpg.44aaa704b4ff80e8182563d352dc5ccf.jpg

I am seeing it too when I looked closely enough (perhaps I need new glasses :) ).     It is something that slipped past my radar and easily fixed.   At one point I had some extra fields present on that screen to help with testing and left a tiny bit of text behind when I removed them.

  • Upvote 1
Link to comment
On 3/14/2019 at 3:21 AM, itimpi said:

Yes, that is the behaviour to expect. I have added a more detailed example to the first post in this thread to make that clearer.   I will also add the example to the help text in the GUI when I update the plugin.

 

It does rely on the fact that the interval between scheduled checks is long enough that the daily increnent size Is large enough to add up to enough time to complete one check before the next one is scheduled to start.   I must admit I have not (yet anyway) tested what happens if this assumption is not true.  

 

I had assumed that people did not run scheduled parity checks too frequently because of both the length of time individual checks take and the load on the system while they are running.   I think that the behaviour is likely to be that the check in progress simply continues instead of starting again from the beginning but this does need validating.  If not the assumption needs to be clearly stated to stop people setting the increments to be too small.

Yeah, very nice idea, thanks for this.

 

I run parity every 3 months, so i guess, that should be done in 1-3 days 

Link to comment

Is it possible to pause pause the parity check when the disks are getting hot? Best case fans are not working well when the room is too hot (summer).

 

And is it possible to implement a button on the main page to do a partial parity check on demand? My server only runs when needed and sometimes only 1h a day or less

 

Nice feature btw! :)

Edited by Marino
Link to comment
13 minutes ago, Marino said:

Is it possible to pause pause the parity check when the disks are getting hot? Best case fans are not working well when the room is too hot (summer).

 

And is it possible to implement a button on the main page to do a partial parity check on demand? My server only runs when needed and sometimes only 1h a day or less

 

Nice feature btw! :)

 

In this case I would run Parity when its at the coolest part of the day. Maybe early hours of the morning. I know here where I live its normally the coolest time of the day just before Sunrise. You could set your server up to run a couple of hours for a few days at that time frame.

Link to comment
10 minutes ago, Marino said:

Is it possible to pause pause the parity check when the disks are getting hot? Best case fans are not working well when the room is too hot (summer).

 

And is it possible to implement a button on the main page to do a partial parity check on demand? My server only runs when needed and sometimes only 1h a day or less

 

Nice feature btw! :)

The idea of taking into account disk temperature is an interesting one, but how practical it might be I have no idea.   I suspect it is one of those that sounds a good idea at first but then not too practical after all, and that the only real solution is to improve the cooling inside your server’s case.

 

Partial paritcy checks is not a feature that can currently be implemented as there is no way to start except at the start.   It will need Limetech to provide a way for developers to start a parity check at a defined offset to make it possible.   Whether this will ever happen I have no idea.

Link to comment

Coolest part of the day are here (germany) in the middle of the night. I'd to run the server then. My 12TB drives are getting hot while checking parity, thats why I wished me to pause while the drives are to hot and when it takes a lot longer but keeps the drives cool.

 

edit:

@ itimpi

the problem is, when the case has 4 fans and the drives are in the coolest area just behind that fans the drives (3x12TB) are getting hot anyway.

To improve that I placed the pc on an open window in the winter. But when the room is too hot (summer) I need an AC to cool it down.

 

But maybe it is not a big of a deal to pause the parity check when one drive reaches an unsecure level of temperature.

 

That is THE reason for me to run parity checks not that often.

Edited by Marino
Link to comment

Not sure why, but the image for the Plugin isn't loading in my install. Not sure if its a glitch on my end or simply a missing image. I'm aware you don't click on it because you go into settings and use the schedule feature, but just thought it was kinda odd. 

 

Capture.JPG

 

Link to comment
1 hour ago, Marino said:

The plugin works only on schedule right?

 

When my server is not powered on at a specific time it would be nice to have an option to check now for ... minutes/hours. Or better. Check for an hour and shutdown after that :)

Not quite sure what you are asking for here :(  What are you thinking should be checked for?   The plugin currently requires a parity check to be actively running (although it can be paused) before it gets involved

Link to comment

I know, but it requires when adding the schedule that the server is running during that time. My server somtimes does not run for a week or so. For this it would be great to check partly the parity and then shutdown (without schedule).

 

Or am I wrong that the schedule is needed to use this plugin?

Edited by Marino
Link to comment
58 minutes ago, Marino said:

I know, but it requires when adding the schedule that the server is running during that time. My server somtimes does not run for a week or so. For this it would be great to check partly the parity and then shutdown (without schedule).

 

Or am I wrong that the schedule is needed to use this plugin?

Two points:

- This plugin does not require a scheduled parity check to be running when the times for its increments to run come around.  The parity check could have been started manually or for some other reason (such as an unclean shutdown).   The increment pause/resume times do need to be scheduled.

- Unraid does not (yet anyway) provide a mechanism that would allow partial parity checks to be run.   Parity checks always start from the beginning.    All this plugin does is allow a check to be stretched over days only running in scheduled idle periods (typically overnight).

 

The plugin's current capabilities are therefore targeting those who leave their Unraid servers powered on overnight and have large enough parity drives that the check currently takes long enough that it impacts performance at times when the Unraid server is not going to be idle.

 

The sort of capability I think you are asking for can only be implemented if/when Limetech provide a capability for parity checks to be started at a defined offset rather than only from the beginning.

Edited by itimpi
Link to comment

My Bad, sorry, i thought the check will resume at the point it was paused last time.

 

So the schedule is only for pause the parity check and the server has to be powered on until the check is done completely? Otherwise the beginning is very good checked but not the rest and the parity check can not finish, right?

 

Then a pause when one of the drives reaches a defined temperature would be even better. When the server has to be powered on all the time this could be useful to cool the system down a little while in pause.

When there is a possibility to implement this.

 

Thank you for the clarification. 

Edited by Marino
Link to comment
1 minute ago, Marino said:

My Bad, sorry, i thought the check will resume at the point it was paused last time.

Unfortunately not.   I have designed the plugin so that if Limetech provide the required restart capability the plugin will be able to take advantage of it.

2 minutes ago, Marino said:

So the schedule is only for pause the parity check and the server has to be powered on until the check is done completely? Otherwise the beginning is very good checked but not the rest and the parity check can not finish, right?

Yes.

2 minutes ago, Marino said:

Then a pause when one of the drives reaches a defined temperature would be great. When the server has to be powered on all the time this could be useful to cool the system down a little while in pause.

This is something I am currently trying to implement.   I am not sure how hard it is going to be to get it working well.   The idea is that this would be a completely independent setting to the scheduled pause/resume capability that is already implemented so it would always apply to a parity check at any time regardless of how it was initiated once you have set the option to activate this.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.