Parity Check - Parity Drive larger than Data Drives


Recommended Posts

A quick question I couldn't find the answer to...

 

On my system I have an 8TB Parity Drive, while my largest Data drive is 6TB.  Perhaps not uncommon, as I'm allowing for future data drive upgrades.

 

When doing a Parity Check it takes say 2 - 3 days to check all the required Data Drive Parity (ie. up to the 6TB mark), after which all the data drives have spun-down and only the Parity Drive remains active.  The Parity Drive then diligently continues reading through it's remaining 2TB (which is not protecting any data drives).  This continued reading of the remaining 2TB of the Parity Drive adds another 10 - 12 hours to the Parity Check completion!

 

So, my question is: Why does a Parity Check need to continue reading the Parity Drive, beyond the capacity of the largest Data Drive?

 

Could this not be optimised, such that once the calculated Parity has been Checked for all the Data Drives, the Parity Check is complete?

ie. Assuming there is indeed no need to just continue on with reading the Parity Drive, when there is no remaining Data Drive capacity to "Parity Check".

 

Link to comment

Hi Benson.  Thanks, but I'm not questioning the MB/s speed of my system.  I'm questioning why the Parity Check *needs* to continue beyond the Data Disk size.

 

ie. Once all the Data drives have been Parity checked against the Parity Drive(s), then I don't understand why the Parity Drive (alone) needs to continue to be read? There is no further Data Parity to calculate / check!  

 

Speed of the system is not relevant to this question.

 

On any system, if you are performing a Parity Check before another operation (eg. upgrading a Data Drive), then the additional time waiting for the larger Parity Drive to (seemingly unnecessarily) read *all* the way through, just adds to the overall time for completing the whole operation.

 

Link to comment
2 minutes ago, Koolkiwi said:

I'm questioning why the Parity Check *needs* to continue beyond the Data Disk size.

So that if you clear a data disk the same size as the parity disk and add it to the array, parity is already valid instead of having to recalculate.

 

It's much better to always keep parity valid, that way you don't have to code for a bunch of scenarios whether or not to trust that parity is correct.

Link to comment
58 minutes ago, Koolkiwi said:

Hi Benson.  Thanks, but I'm not questioning the MB/s speed of my system.  I'm questioning why the Parity Check *needs* to continue beyond the Data Disk size.

 

ie. Once all the Data drives have been Parity checked against the Parity Drive(s), then I don't understand why the Parity Drive (alone) needs to continue to be read? There is no further Data Parity to calculate / check!  

 

Speed of the system is not relevant to this question.

 

On any system, if you are performing a Parity Check before another operation (eg. upgrading a Data Drive), then the additional time waiting for the larger Parity Drive to (seemingly unnecessarily) read *all* the way through, just adds to the overall time for completing the whole operation.

 

If you intended of why check / sync parity which beyond data disk size than I agree it is not necessary, but this need Unraid parity handling or program code to be change.

 

If you add a larger data disk, then parity also need update but in current handling not need as @jonathenm say.

Edited by Benson
Link to comment
1 hour ago, jonathanm said:

So that if you clear a data disk the same size as the parity disk and add it to the array, parity is already valid instead of having to recalculate.

 

It's much better to always keep parity valid, that way you don't have to code for a bunch of scenarios whether or not to trust that parity is correct.

So you are basically saying that the continued reading is just to verify that the remainder of the Parity drive reflects the Parity (0?) for Zeroed Data (which is in effect what you could say we have when there is no Data).

 

I can see the argument for doing this (ie. easier / quicker to add a pre-cleared new disk that is larger than all your other data disks!).

 

But, in terms of efficiency / parity disk wear, that does seem rather wasteful to be doing this extended zero parity check *every time* you do a Parity Check, just to allow for the single case of adding a new pre-cleared disk that is also larger than any existing Data disk!

 

It would seem more efficient (and logical) to deal with this extra Parity only in the actual case of you adding a new disk that is larger than all other data disks (even when pre-cleared).  So that the added disk size space would only, on that one occasion, then have the extra Parity space initialised / updated (if needed).

ie. In my case, if I added my first 8TB Data disk, then there would just be a *once-off* need to initialise the extra 2TB of the Parity disk.

Noting also that if my new 8TB drive had not been pre-cleared, the added Parity space is going to need to be initialised anyway!

  

Edited by Koolkiwi
correct typos
Link to comment
45 minutes ago, Benson said:

If you intended of why check / sync parity which beyond data disk size than I agree it is not necessary, but this need Unraid parity handling or program code to be change.

Yes, of course.  But the point of discussing questions of this nature, is so that the developers can consider potential improvements to the product in some future version (if it is deemed a valid point). 

Link to comment
44 minutes ago, Koolkiwi said:

Noting also that if my new 8TB drive had not been pre-cleared, the added Parity space is going to need to be initialised anyway!

No, Unraid will clear a drive before adding it to the parity protected array if a valid pre-clear signature is not detected. That way parity always remains fully valid, in the event of a drive failure during the adding process.

 

The only time parity is recalculated when adding a drive is if you execute a "new config" and tell Unraid to rebuild parity from scratch based on the existing content of the data drives.

48 minutes ago, Koolkiwi said:

Zeroed Data (which is in effect what you could say we have when there is no Data).

I know you are going to cry semantics, but it's an important distinction to make when dealing with parity that an empty drive with a valid filesystem has data. The filesystem structure itself is handled by parity as well. So a formatted drive with no user data is still full of content for the purposes of parity. Parity is just ones and zeroes, across the entire partition, whether there is a blank XFS filesystem or a filled to the brim encrypted BTRFS volume, it's all the same for parity.

Link to comment
54 minutes ago, Koolkiwi said:

But, in terms of efficiency / parity disk wear, that does seem rather wasteful to be doing this extended zero parity check *every time* you do a Parity Check, just to allow for the single case of adding a new pre-cleared disk that is also larger than any existing Data disk!

The default position is to be sure parity is always fully correct, and alert if that's not the case, so the user can take action to be sure parity is correct. If you change that logic and assume parity isn't valid for some portion of the parity disk, then there are many changes that would need to be made to account for that. When you make such low level decisions in the code that could effect data integrity and drive recovery, the amount of work to validate all the possible scenarios and ways it could mess up is a monumental task.

Link to comment
4 minutes ago, jonathanm said:

The default position is to be sure parity is always fully correct, and alert if that's not the case, so the user can take action to be sure parity is correct. If you change that logic and assume parity isn't valid for some portion of the parity disk, then there are many changes that would need to be made to account for that. When you make such low level decisions in the code that could effect data integrity and drive recovery, the amount of work to validate all the possible scenarios and ways it could mess up is a monumental task.

Thanks jonathanm, I think the above (and your prior reply) does answer my question.

What I appear to have been overlooking was: 

Quote

Unraid will clear a drive before adding it to the parity protected array if a valid pre-clear signature is not detected

This being the case, I can now understand the logic for ensuring the full Parity Drive represents the full Parity (for the full Parity size space).

 

If I've understood you correctly, you're saying that when a new larger drive is installed, it is (in effect) the new "empty" drive space that is being brought into line with the equivalent empty / cleared space Parity.  ie. It is the new Data space that is cleared, not the Parity being updated to reflect whatever was on the new Data drive space.

 

In terms of the Parity process itself, no I'm not going to cry semantics. :)  I do believe I have a good understanding of the Parity mechanism, and do understand that parity is calculated across the raw content of the drives (yep, I've had too long a career in computers, IT, and software development).

 

Link to comment
15 minutes ago, Koolkiwi said:

 

If I've understood you correctly, you're saying that when a new larger drive is installed, it is (in effect) the new "empty" drive space that is being brought into line with the equivalent empty / cleared space Parity.  ie. It is the new Data space that is cleared, not the Parity being updated to reflect whatever was on the new Data drive space.

Exactly. Keeping parity valid all the time every time is the name of the game. If parity must be deemed invalid and corrected or rebuilt, it had better be for a good reason.

 

If your equipment is healthy and nobody downs the server without stopping the array, parity should never be wrong. Any fault should be investigated and corrected. Incorrect parity means a failed drive will be recreated with wrong data, you can only hope the flipped bits are in an unallocated section or the picture area of a media file, if they are in a filesystem area you are looking at file system corruption.

 

Data integrity is the "You had ONE job" of a NAS.

  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.