[Feature Request] Parity check pause

[jbuszkie]

This seems like it would be super simple to implement...

Just a pause bottom with the parity checks.

There are times when the array is just so unresponsive that it's almost unusable during parity checks.

I'd love to be able to pause the parity check so I can do what I need to do with the array and just have it continue when I'm done..

This goes along the lines of the incremental parity checks

And auto throttled parity checks

 

I like the thought of a "low priority button as well"  But a pause should be easy to implement.

 

Jim

[ashman70]

I think this would be a great feature too, but if new data was copied to the array while it was paused, I think we'd have to start all over again, no?

[jbuszkie]

thats the case anyway...  if you use the array while the parity check is going on.

 

Any new stuff will just be checked next time if it's past where the check already was

[garycase]

thats the case anyway...  if you use the array while the parity check is going on.

 

Any new stuff will just be checked next time if it's past where the check already was

 

Actually once an area is checked, it's checked => if you write new data to the array that impacts an area that's already been checked, it does NOT get re-checked in that section .. it just won't be checked until the next parity check.

 

By the way, this idea has been discussed several times in prior threads ... I don't think LimeTech plans to add this.  HOWEVER ... Tom has indicated that he is looking at an auto-throttling feature, which would be a nicely automated way of resolving the performance issue during parity checks => they would effectively "pause themselves" when there was other activity.

 

 

[jbuszkie]

I think the pause button would be easy to implement...  The auto throttle might be more complicated..

Once the auto throttle was perfected then the pause might not be necessary...

 

 

[jbuszkie]

thats the case anyway...  if you use the array while the parity check is going on.

 

Any new stuff will just be checked next time if it's past where the check already was

 

Actually once an area is checked, it's checked => if you write new data to the array that impacts an area that's already been checked, it does NOT get re-checked in that section .. it just won't be checked until the next parity check.

That's what I said! :-)

[garycase]

I think the pause button would be easy to implement...  The auto throttle might be more complicated..

Once the auto throttle was perfected then the pause might not be necessary...

 

I agree it seems easy to implement ... but for whatever reason LimeTech didn't like the idea when it was discussed in an earlier thread => but is definitely considering an automatic throttling feature.  Which is, in fact, a nicer feature anyway ... a casual user (wife, kid, etc.) isn't likely to think about logging into the GUI and hitting "Pause" to fix a stuttering movie they're trying to watch during a parity check => but the auto-throttle would simply eliminate the issue.

 

[jbuszkie]

I agree the auto throttle would be preferred....  but it's been a year since that thread was touched.  probably not a high priority for Tom!

[WeeboTech]

I think the pause button would be easy to implement...  The auto throttle might be more complicated..

Once the auto throttle was perfected then the pause might not be necessary...

 

The code already exists in the current md driver, borrowing that may be easier then pausing and restarting.

To pause and restart requires some kind of update to the superblock.

 

It will probably be easier to lift working code and retrofit it to the final desired functionality.

Tom's a smart guy. I've seen the auto-throttle in action with software raid1 and the md driver. It worked nicely and no one had to do anything.

 

Now having a minimum and maximum throttle like the md driver might be nice as well.

At that point you could set the maximum to be a very low value during waking ours and reset it for idle times.

That worked well also.

[dalben]

I'd like to see the opposite as well.  The ability to pause/stop/suspend dockets and plugins when a parity check kicks off to stop the amount of interruption to the parity check.

[Squid]

I'd like to see the opposite as well.  The ability to pause/stop/suspend dockets and plugins when a parity check kicks off to stop the amount of interruption to the parity check.

I think that's actually doable right now with a simple script.

 

Never tried it, but it looks like you can pause any container at will with the docker pause command.

 

A simple script to check var.ini to see if a check / rebuild is running, then issue the appropriate docker pause commands.

 

EDIT:

 

Very quick and dirty test, and it appears that its no problem to pause a running container at will, and then unpause it later.

[Squid]

I'd like to see the opposite as well.  The ability to pause/stop/suspend dockets and plugins when a parity check kicks off to stop the amount of interruption to the parity check.

Combined with an appropriate cron entry (hourly?), this works as a quick and dirty script.  Pauses and unpauses appropriately if a check / rebuild is running

 

#!/usr/bin/php
<?php
$variables = parse_ini_file("/var/local/emhttp/var.ini");

$running = shell_exec("docker ps");
$runningContainers = explode("\n",$running);
unset($runningContainers[0]);

if ( $variables['mdResync'] )
{
  foreach ($runningContainers as $Container) {
    if ( strpos($Container,"(Paused)") )
    {
      continue;
    }
    $lineSplit = explode(" ",$Container);
    $containerID = $lineSplit[0];
    if ( $containerID )
    {
      echo "Pausing ".$lineSplit[8]."\n";
      exec("docker pause $containerID");
    }
  }
} else {
  foreach ($runningContainers as $Container) {
    if ( strpos($Container,"(Paused)") )
    {
      $lineSplit = explode(" ",$Container);
      $containerID = $lineSplit[0];
      echo "Unpausing ".$lineSplit[8]."\n";
      exec("docker unpause $containerID");
    }
  }
}
?>

[dalben]

Cool, thanks. I'll have a play with it.

[garycase]

While it's nice to be able to auto-pause applications that are running when a parity check or rebuild is in progress, I can think of several cases where that wouldn't be very desirable ... i.e. if you're streaming a movie I suspect you'd rather not have it automatically stop because it's time for a parity check.    An auto-throttled parity check would be far preferable in that instance.    Bottom line is I can't think of any real disadvantage of a throttled parity check (except it takes  a lot longer) ... but I can think of quite a few instances where you wouldn't want applications to be halted without warning.

 

... And for those cases where you DO want to stop apps so there's no throttling -- e.g. if you're doing a disk rebuild you likely want it to complete as quickly as possible -- it's simple enough to just manually pause/stop all of your apps.

 

[dalben]

My monthly parity checks kick off at 1am Monday and go through to the morning.  No one streams in my house at that time and the family is too busy getting ready for work and school to stream in the morning.

 

While it may not suit your use case, it certainly suits mine, hence why I asked. 

[Wimpie]

Want to revive this discussion since, as far that I know, no throttling of the parity check is available.

 

I would like pauseable parity checks.

 

My server MB's have hit an internal transfer limit, so each time I add a drive, my parity check times grow much larger than one would like.

 

At the moment I limit the server to 20 drives, but my parity check time is already around 22 hours.

 

This means I always have a day that viewing movies is impossible.

 

Pauseable parity checks would allow me to start a parity check at 01:00 and pause it around 07:00. Do this 4 nights in a row and parity is also checked.

 

What do you think??

 

[jbuszkie]

I also have issues with latency and such when the parity check is going on.  I'm curious how folks that use dockers and such are effected

by the parity check?  Is my machine just that under powered?

 

I've wanted this feature for a while too..

 

Jim

 

Edit:  Oh wait I was the OP!  lol

[WeeboTech]

I'm not a proponent of a full pause (only).

While I'm not an opponent of that feature request, I'm more of a proponent for a throttle option.

 

The older md code had an option to throttle a minimum and maximum resync speed.

 

There was also code to drop to the minimum speed if the subsystem was being used and raise to the maximum speed on idle subsystem access.

 

If someone were to set that to 0 or a very low but acceptable value, so be it, a full pause might not be the best way to go.

What if someone forgot it was on pause?

 

Another choice might be to do incremental parity checks, but that entails keeping place (like pause).

[garycase]

As I noed earlier in this thread, Tom has indicated that he is looking at an auto-throttling feature, which would be a nicely automated way of resolving the performance issue during parity checks => they would effectively "pause themselves" when there was other activity.

 

While that comment was a long time ago, there's another active thread where Tom has implied he's still planning to do exactly that (auto-throttle parity checks).  [ See https://lime-technology.com/forum/index.php?topic=53120.msg510930#msg510930 ]

 

Hopefully that's a feature that will be incorporated in a not-too-distant release ... it would indeed resolve quite a few performance issues that folks have noted during parity checks -- and would eliminate the need for a "Pause" feature.

 

 

 

 

[DZMM]

Old request but +1 from me - my monthly parity becomes a constant battle to stop my cache pool dying as mover can't move files fast enough to the array at times

[Output]

I really like the idea to be able to pause the parity check, but since I just added 8tb disks to my array my check times has increased. Thus I would instead of a pause be able to set the time when the check is done. For instance be able to set so that the check is only done 1 am - 8 am. It would then continue the ( if triggered by the schedular) the next night 1 am - 8 am and so on until done. 

[DZMM]

1 hour ago, Output said:

I really like the idea to be able to pause the parity check, but since I just added 8tb disks to my array my check times has increased. Thus I would instead of a pause be able to set the time when the check is done. For instance be able to set so that the check is only done 1 am - 8 am. It would then continue the ( if triggered by the schedular) the next night 1 am - 8 am and so on until done. 

+1 not sure if this is possible, but if it is I'm a fan - even if it means the check takes 50% longer.  This is a better solution than pausing as it doesn't require me to be at the console to pause and then resume

[RockDawg]

I would also really like to see something implemented to lessen the impact of parity checks on system performance.  With 8TB drives in my system parity checks now take ~20 hours.  No way to schedule that around viewing times.  Plus nowadays I am running Emby and have some friends and family that stream from my server remotely too so it now affects more than just my household.  It's kind of a bummer having to tell people that the first day of each month it probably won't work very well.

 

 

[itimpi]

The Parity Check Tuning plugin is now available (as long as you are running the 6.7 rc4 release or later) which can automate pause/resume of parity checks to allow increments to be run when the system is likely to be idle.  More details are available in the forum Support thread for this plugin.