Allow parity check ranges


SSD

Recommended Posts

Option to run a parity check for a specific range (e.g., 0-50%, 51-100%). With drives getting larger, it is becoming difficult to have the parity check not affect prime time use of the array By being able to split the parity check into two halves, thirds, quarters, etc., it would allow users to split it over a few nights.

 

Being able to test a narrow range would also come in handy for certain types of problem isolation.

  • Upvote 4
Link to comment

That was 4 years ago! Need to keep it warm. I'd really prefer to have more control rather than throttling I/O ,but guess I'll take what I can get.

 

I bet if you searched back farther you'd find several earlier impassioned requests (by me!) for this or similar feature.

Link to comment

That's different than my request.

 

Mine has three primary use cases:

 

1 - Spread parity checks over multiple days to avoid affecting performance during periods of heavy demand. Throttling would provide a similar, although much more difficult to implement and inferior under a heavily utilized array. I like the absolute power to control and limit parity checks and spun up drives.

 

2 - Resume a parity check - if a server crashes or needs to be brought down during a parity check, currently a user has no choice but to restart a parity check, even if it is many hours into it. With the feature I suggested, a user could note the percent complete, stop the check, reboot, and then start the parity check to the % complete and resume. Throttling would not solve this..

 

3 - Less common, but we sometimes see strange behavior of sectors that appear to flip flop back and forth causing parity sync errors. We've never had a tool that would let us target a narrow range of sectors to properly investigate. With parity checks taking many hours, this type of analysis is not possible.

 

  • Upvote 2
Link to comment
20 minutes ago, gubbgnutten said:

And maybe also useful for running a correcting parity check on a small part of the array if a non-correcting check found errors.

+1

Linked to a recent non-correcting check, it would be possible to correct only the sectors found in error.

 

What I would like to see...

Every non-correcting check logs errors to a dedicated single use log file overwritten each time, with strict limits on number of errors logged before it gives up. A non-correcting check that exceeds that number fails out with an error, aborting with a message urgently requesting a full correcting pass or further troubleshooting.

If the number of logged errors is low enough, an option to run either a full correcting pass OR a pass utilizing the latest non-correcting logged list, followed immediately by a second non-correcting pass with a special flag that aborts on ANY error, and requests a full correcting pass or further troubleshooting.

  • Upvote 3
Link to comment
6 hours ago, jonathanm said:

+1

Linked to a recent non-correcting check, it would be possible to correct only the sectors found in error.

 

What I would like to see...

Every non-correcting check logs errors to a dedicated single use log file overwritten each time, with strict limits on number of errors logged before it gives up. A non-correcting check that exceeds that number fails out with an error, aborting with a message urgently requesting a full correcting pass or further troubleshooting.

If the number of logged errors is low enough, an option to run either a full correcting pass OR a pass utilizing the latest non-correcting logged list, followed immediately by a second non-correcting pass with a special flag that aborts on ANY error, and requests a full correcting pass or further troubleshooting.

Would love that. Maybe should be extracted to a dedicated feature request?

Link to comment

Blast from the past, I had a pretty advanced set of features to enhance the parity check. Most of it is still valid. My simple request in this thread was in hopes of getting something very basic that Tom would actually implement. Sometimes great is the enemy of good. :)

 

 

 

Link to comment
  • 4 months later...

Now with 8tb drives becoming the norm, parity check times are approx 24h. While one can adjust parity check priority indirecty through tunables (stripes, window, thresh) this is not a reliable solution as it reduces parity speed even if the user application (e.g. streaming a bluray of the array) is no longer running. Also for noise/heat of the server during a parity check some more fexibility for parity checks would be really really nice. Here some ideas:

 

- not only a start/stop feature, but maybe rolling partial parity checks. For example one could set aside one or two hours a day for parity checking (outside main usage time of server) and even if it takes 10 days to complete one cycle, this would be a great improvement to my current check schedule of every three month.

 

- remember errorenous stripes from a past (non-correcting) parity check and allow to only re-check these stripes. Saves a lot of time as i try to run non-correcting checks first - in order to have some flexibility to determine if parity is correct of the data on the array disks

 

- get list of files potentially effected by the errors found in a parity check

 

What do you guys think?

Edited by Videodr0me
  • Upvote 1
Link to comment
  • 3 months later...

I'd be up for this.  With a couple of 8TB drives, my parity checks are about 36 hours (would be shorter, but normal usage slows things down).  I would much rather either run a percentage over night, or run parity checks between certain hours.

 

Say, a window of 2-6am to run parity checks, and it takes as many nights as it needs to.

 

Ideally, parity would run when the system was idle and chip away at the scans, but I'm guessing that would open another can of worms.

Link to comment

Recently switched to dual parity. Checks now take almost two days. (55MB/s vs. ~110 for single)

 

Would love something like this. The array is sluggish as all hell on dual parity during a check. Maybe I'm CPU limited or there's something else going on, but Plex was unusable during this time, as were most other things.

 

Would love to have this run only at night for 3-4 days instead.

Link to comment
Recently switched to dual parity. Checks now take almost two days. (55MB/s vs. ~110 for single)
 
Would love something like this. The array is sluggish as all hell on dual parity during a check. Maybe I'm CPU limited or there's something else going on, but Plex was unusable during this time, as were most other things.
 
Would love to have this run only at night for 3-4 days instead.
Big bottleneck somewhere, my checks with dual parity take as long as they did with single parity.
Link to comment
2 hours ago, johnnie.black said:
2 hours ago, -Daedalus said:
Recently switched to dual parity. Checks now take almost two days. (55MB/s vs. ~110 for single)
 
Would love something like this. The array is sluggish as all hell on dual parity during a check. Maybe I'm CPU limited or there's something else going on, but Plex was unusable during this time, as were most other things.
 
Would love to have this run only at night for 3-4 days instead.

Big bottleneck somewhere, my checks with dual parity take as long as they did with single parity.

 

Could well be. Using a C2750 SoC, but usage is never more than around 25-40% during a check. Maybe it's lacking instruction sets that dual parity finds beneficial or something. 1950X build incoming though, so we'll see soon enough.

Link to comment
On 07/11/2017 at 1:57 PM, -Daedalus said:

 

Could well be. Using a C2750 SoC, but usage is never more than around 25-40% during a check. Maybe it's lacking instruction sets that dual parity finds beneficial or something. 1950X build incoming though, so we'll see soon enough.

 

I have 2 servers with a similar MB (C2550).

 

My experience is that there is an internal (to the C2550 SoC) bus limit we are hitting with many drives.

 

I get a maximum of 1.55GB/s of disc throughput (disc stats while parity checking).  I tried different controllers, splitting drives as much as possible over several controllers (using all PCIe slots),  putting some drives on the SoC sata channels, etc...  CPU shows not more than 40% usage, so the CPU is not the bottleneck.

 

Not a problem with 8-10 8Tb drives with this MB, but in a 20+ drive array it is very noticeable. I get hit by this bus limit for the first 4 Tb (0 to 4Tb section), then the array 'shrinks' from 22 to 17 drives (only 6Tb and 8Tb drives), and I see the 'normal'  speed curve (speed dropping while the parity checks goes to the slowest section of the  6Tb drives, and then it repeats for the 8Tb drives (6Tb to 8Tb section))

 

I still have single parity.

 

So I also would like an option to split a parity check to several nights between eg 02:00 and 06:00 ...

 

 

Link to comment
  • 4 months later...

Has there been any further thoughts as to whether a facility along these lines might be incorporated into unRAID as a standard feature?     Periodic parity checks are seen by many as something one should do as part of maintaining array health.  With the size of large drives (12TB now being freely available) continually increasing the time for parity checks is increasing significantly and thus impacting convenient use of the unRAID array.   Times are now becoming significantly more than a day to carry out the full parity check and thus almost certain to impact array use at inconvenient times.

 

in an ideal world I would like to have several options for splitting a parity check into segments:

  • the ability to do a percentage of the parity check each day so it can be limited to hours when the array is not in use.   This would allow a parity check to be spread across days.
  • In a similar vein the ability to specify start/stop times for parity checking to be carried out.   This would help with making sure you were not running a parity check when mover (or other routine task) might be running
  • manually pausing and then restarting a parity check.    Ideally it should be possible to do a reboot between the pause/restart actions.

i realise this raises issues about how progress would be tracked and reported but at first glance I do not see this as being a particularly difficult issue to resolve.

  • Like 2
Link to comment
  • 1 year later...

The Parity Check Tuning plugin is now available (as long as you' are running Unraid 6.7 rc4 or later) that at least allows a parity check to be broken down into increments that can be run when the system would normally be idle.   More details are available in the forum Support thread for this plugin.

 

At the moment there is no way to start a parity check at a defined offset so the idea of running a partial check for a specified range is not yet possible.    If Limetech ever provide a way to start a parity check at a defined offset then the plugin could be enhanced to allow for partial parity checks that do not have to start from the beginning as mentioned in this thread's OP post .

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.