Jump to content
jonp

Parity check/sync "auto-throttling" feature

34 posts in this topic Last Reply

Recommended Posts

1 hour ago, rick9814911 said:

even a small blip in power

You should have an UPS on your server. The parity check you get from an unclean shutdown may not be the only consequence of frequent power loss. Filesystem corruption is another possibility if a data write is unable to complete, for example.

Share this post


Link to post
59 minutes ago, rick9814911 said:

This would be a very welcomed feature! Especially after a power outage. If I can schedule the parity check it's not a huge deal. But even a small blip in power makes the server slow for hours because of the parity check. 

Why not use the Parity Check Tuning plugin to only run the check out of prime hours.    The plugin now handles pausing/resuming non-scheduled checks as well (assuming you enable this in the plugin's settings).   The after a power blip if you get a parity check auto-starting because of an unclean shutdown you can immediately pause it and let the plugin handle running it only at the scheduled hours.  That way you avoid having the server running slowly during prime time because of the parity check.

 

I have been looking at whether the plugin could do an automatic pause/resume according to system load but I have not yet found a satisfactory way of detecting when a pause is appropriate and (even harder) when a resume is relevant so at the moment it only handles running increments at scheduled times.   If I ever crack detecting the load appropriately I will add an 'auto' option to the plugin. 

Share this post


Link to post
41 minutes ago, itimpi said:

The after a power blip if you get a parity check auto-starting because of an unclean shutdown you can immediately pause it and let the plugin handle running it only at the scheduled hours.  That way you avoid having the server running slowly during prime time because of the parity check.

This could be dangerous.  Power failure can corrupt the file system, and writing to a corrupt file system can make the corruption worse, even so far as making the file system no longer accessible.  That's if it doesn't throw some other kind of error trying to deal with the corruption.  Always best to let a parity sync run to completion after a power failure.

 

The solution to power failure parity syncs is to get a UPS.  Prevent the corruption in the first place and you won't have to deal with a sync.

Share this post


Link to post
1 hour ago, koyaanisqatsi said:

This could be dangerous.  Power failure can corrupt the file system, and writing to a corrupt file system can make the corruption worse, even so far as making the file system no longer accessible.  That's if it doesn't throw some other kind of error trying to deal with the corruption.  Always best to let a parity sync run to completion after a power failure.

 

The solution to power failure parity syncs is to get a UPS.  Prevent the corruption in the first place and you won't have to deal with a sync.

Sorry for dragging this offtopic.

 

Parity check isn't going to affect filesystem one way or the other, since even a correcting parity check will only write parity, and has no effect on the filesystem of any data disk. If the power failure causes a filesystem corruption, that would have to be dealt with on its own. So whether you complete a parity check now or later is a separate issue.

 

Best to have an UPS as mentioned and avoid these potential issues.

Share this post


Link to post

Ignoring the fact that having a UPS is an excellent idea on an Unraid server (and I have one myself), I have found that more often than not after an unclean shutdown the automatic parity check corrects a few sectors near the beginning of the disk (I presume because some bits to do with unmounting the disks are not correctly set) and then often runs to the end with no further sectors corrected.   In such a case pausing/resuming the parity check has little effect on the overall result.

 

As was mentioned file system corruption would be a much more serious issue and would not be corrected by a parity check anyway.  You also relatedly frequently get the case where a disk is reported as unmountable after an unclean shutdown if you try and start the array and running xfs_repair against it resolves this without any data loss.   I personally have a simple script that runs a file system check against all my array drives and if any report corruption (optionally) attempts a repair.   I tend to always run this before starting the array if I have had an unexpected shutdown as it does not take very long.  This has a side benefit that it frequently stops drives being reported as 'unmountable' after restarting from an unclean shutdown.   I wonder if automatically running file system checks after an unclean shutdown is something that it might be worth raising as a feature request?

Share this post


Link to post
25 minutes ago, itimpi said:

Why not use the Parity Check Tuning plugin to only run the check out of prime hours.    The plugin now handles pausing/resuming non-scheduled checks as well (assuming you enable this in the plugin's settings).   The after a power blip if you get a parity check auto-starting because of an unclean shutdown you can immediately pause it and let the plugin handle running it only at the scheduled hours.  That way you avoid having the server running slowly during prime time because of the parity check.

 

I have been looking at whether the plugin could do an automatic pause/resume according to system load but I have not yet found a satisfactory way of detecting when a pause is appropriate and (even harder) when a resume is relevant so at the moment it only handles running increments at scheduled times.   If I ever crack detecting the load appropriately I will add an 'auto' option to the plugin. 

 

I've used this very Plugin for the past two monthly Parity Checks and it has worked exactly as I filled in the options. Other than filling in the options and launching its been literally setup-and-forget. If it hadn't of been for the Notifications I wouldn't of even noticed an impact on the way I normally use my server.

 

I highly encourage it and would recommend it to everybody. Parity Tuner and Mover Tuner work flawlessly together and if you want even more Icing on your Cake run User.Scripts with the Do not run while parity is active for total control of your scheduled events.

 

From the Hours of 12AM-6AM nothing but Parity runs 1st, 2nd and 3rd day of the month and nobody in my house even notices a thing since everybody is asleep.

Share this post


Link to post

I would greatly welcome this feature, as I store all my movies and tv shows on disk, when a parity sync is running it can take longer for things to load.

 

Not to mention loading SMB takes much longer when a parity sync is running

Share this post


Link to post

@jonp - Am I correct in thinking that we have this in 6.8.0-rc3?

Edit: Actually it looks like a lot of these "unscheduled" things are already present.  Maybe time for some pruning?

Edited by Rhynri

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.