I'm just not sure if this is a plugin type of function or if it must be incorporated from the top down, so my apologies if I am in the wrong spot.

 

The idea: An easy way (a report, perhaps) to determine which file(s) is/are affected by any error that may show up during a parity verify operation.

 

Why: Parity corrections occur when "errors" are found.

 

The Premise: Since we do not know why the "error" occurred, it's possible a parity "correction" may not leave the parity data correct.

 

Who: I'm throwing this out there for anyone with the requisite skills as I am not at that level.

 

 

By way of explanation:

Revamping the Parity Check thread, specifically http://lime-technology.com/forum/index.php?topic=11515.msg109807#msg109807

 

What can happen -- and I surmise (due to lower costs, higher volume production and higher capacities per platter all) not as rarely today as perhaps it was rare twenty years ago -- is that a drive can report inconsistent data on successive reads.  It is not supposed to happen, but it does.

 

So if a drive is flipping bits, the parity check can "correct" the parity data differently many times over.  Obviously if the file(s) remain(s) unchanged, the parity data can be correct at one point, and incorrect at another.

 

I am imagining a 60TB array at my place of employ with this problem in just three years' time and I wonder how I ever would know that my video archive has a file that is irrevocably damaged.  Because it's an archive it is destined to sit dormant for a long time, but every few months have new files added to it, or perhaps some older files read from it.

 

I would dearly love to have a way to run a monthly parity verification where the end result is a report on which file is affected in the event an inconsistency is detected during the scan.  That way, if nothing else, I can go to that file, and manually verify that it is in good shape.

 

If/when the file is determined to be okay, a parity check-and-correct would be carried out.  Ideally the correction would only need to happen to the "file" in question as well, so a full parity check is not necessary... but perhaps that is asking too much?

 

Byron

 

So the proposal is an output report of each parity check that can then be compared to previous reports?  If so, I like it - simple and elegant.  What I don't know is if it will actually work, since parity checks work on the bit level (1s and 0s) and I don't know how you would know which bits correspond to which files.  Anyone?

What I don't know is if it will actually work, since parity checks work on the bit level (1s and 0s) and I don't know how you would know which bits correspond to which files. 

 

From my limited knowledge, I think you've hit the nail on the head. Essentially, without doing a full rebuild on each disk then checking MD5 checksums on the results, there's wouldn't be an easy way of doing this.

 

Unless I'm completely incorrect.

 

You're both right, I see now.  I hadn't thought it through entirely at the time of my post.  A parity check works independently of the file system.  Finding a file belonging to a parity disparity then requires something to marry the bit-level operation to the file system operations.

 

Just thinking off the top of my head here: Why does a parity check need to be brute-force from beginning to end?  Parity information is already written in the correct spot(s) whenever files are added or changed, correct?

 

So, what if parity could be checked on a file-by-file basis?  I imagine that sector locations for each relevant file are already part of the system (due to above).  Knowing which sectors are involved should make it possible to perform a parity check of only those sectors.  If that is true, then a "file-parity-check" utility could be written instead of a traditional disk parity check.

 

Why would anyone want this, you ask?  Okay, I asked, but here we are:

 

1) There'd be no worries about which file was wrong,

2) no need to prematurely replace media for a couple of faults, because   (this thought is invalid because if a disk needed to be replaced, or upgraded, an array-wide parity set would need to be present and correct)

3) we'd be able to correct problem files by writing new copies to the array (after renaming the faulty ones), and

4) we'd sleep better knowing all our files are valid

 

After all, is it not the case that we don't actually care too much if the array parity is correct, but rather whether or not all our files are safely stored in the array?

 

Joe L has brought to light something I forgot (refer to point 2) - we are concerned both with our files being correct and the parity data being correct.

 

Further thoughts?

 

(edited for clarity and corrections )

You're both right, I see now.  I hadn't thought it through entirely at the time of my post.  A parity check works independently of the file system.  Finding a file belonging to a parity disparity then requires something to marry the bit-level operation to the file system operations.

 

I suppose one way to work it would be a hybrid parity check.  Just thinking off the top of my head here: Why does a parity check need to be brute-force from beginning to end?

 

So following the idea that it may not have to be that way, what if parity could be checked on a file-by-file basis?  A utility would need to determine the physical sector locations of each that is occupied by the file in question.

 

I hypothesize that knowing which sectors are involved should make it possible to perform a parity check of only those sectors.  If that is true, then a "file-parity-check" utility could be written instead of a traditional disk parity check.

 

Thoughts?

It is because parity is not checked on a per file basis, but on a set of bits at equivalent bit positions across all the drives.  On some drives, a given bit position might be in a file, on others, it might be empty space.  To be able to re-construct ANY drive, all the bits on all the drives need to have parity calculated, and subsequently checked.

To be able to re-construct ANY drive, all the bits on all the drives need to have parity calculated, and subsequently checked.

 

Quite right, Joe.  It's imperative that a proper parity set be present for rebuilding an entire drive.  I wasn't thinking of that point specifically, but of course the implications of incorrect parity go beyond a few corrupt files.

 

(I need to go back and re-edit my last post with this in mind)