A solution for Bit Rot?

Recommended Posts

I have an idea that could potentially solve bit-rot on UNRAID when the user uses a BTRFS file system.


BTRFS checksums data and upon reading, if the data does not match the checksum it will report an error and refuse to read the corrupted data.


If UNRAID parity was changed from real-time to very slightly delayed, if BTRFS threw a checksum error, UNRAID could mark the drive as bad and not include corrupted data into the parity allowing the user to restore the good copy of the data when replacing the hard drive.


Implementation strategies could include using the mirrored cache drives to store real-time changes for the parity BUT NOT IMMEDIATELY COMMIT THEM, such that if BTRFS checksum throws an error, parity could DISCARD the corrupted data from being included into the array.



Edited by ben42153251
Link to comment

If parity can virtualize a failed drive (continued access to data), why couldn't that parity virtualized version of the data be available to a physical working BTRFS formatted disk for the purpose of providing a second copy of the data so that BTRFS scrub could replace corrupted data?


Perhaps a slightly modified version of BTRFS or interoperability layer would be needed for UNRAID but I don't see why UNRAID could not have similar levels of data integrity to ZFS mirror or BTRFS mirror.


The UNRAID parity check could be modified to run BTRFS scrub PRIOR to the parity check and if an inconsistency is found then virtualize the disk reporting corrupt data and run another scrub on the physical disk using the virtualized disk as the mirror source to restore the corrupted data.


Expanding my thought bubble... why couldn't UNRAID (in the background) generate parity virtualized versions of ALL physical disks at ALL times such that BTRFS would always have access to a backup copy of a file if a read checksum failed? Then just kick the corrupt disk from the array and rebuild with a new drive.


To avoid a read speed penalty, BTRFS should only draw on the virtualised parity copy of the data when a checksum fails reading data off the physical disk.

Edited by ben42153251
Link to comment
30 minutes ago, sota said:

so... you want unRAID to basically run the array as a RAID4 setup?

No because each disk would still contain its own BTRFS file system rather than being striped.


It may become necessary to use UNRAID to access those individual file systems if a slightly modified version of BTRFS was used (instead of an interoperability layer) but the idea is for each disk to retain its own file system.

Edited by ben42153251
Link to comment

I just realised that in a dual parity configuration we actually have a total of 3 copies of a file potentially available to BTRFS (including the original physical disk).


When reading data only the disk containing a file would need to be spinning, BTRFS could be told another copy of the file is available but UNRAID would only need to spin up the other drives if a checksum failed and it was necessary to read the file from the parity generated copy.


Essentially my whole idea is to an extent "lie" to BTRFS that it's in a mirror configuration using a virtual parity generated disk when necessary to facilitate the other half of the "mirror".


Limetech please consider writing a BTRFS interoperability layer so it can use a parity generated disk to get a copy of a file if its corrupted on the physical disk.


The first step is starting with the 6.9 release, make BTRFS the default file system.

Edited by ben42153251
Link to comment
1 hour ago, ben42153251 said:

I just realised that in a dual parity configuration we actually have a total of 3 copies of a file potentially available to BTRFS (including the original physical disk

This implies that you do not understand how parity works.   Parity has no concept of files or data - it only understands physical sectors.  It’s purpose in life is to be able to recreate a sector on a failed drive but it has no idea what that sector is being used for.   That is why it can handle disks in any mix of file system types as parity operates at a lower level than file system handling.

Link to comment

We use the parity to generate a raw disk as if it had FAILED and mount it READ ONLY somewhere else so that when BTRFS reads off the physical disk and a checksum fails, it can check if the parity generated version of that drive contains the correct version of the file.


Eg imagine standard RAID 5, 1 disk fails, parity is re-creating the whole disk virtually.

Edited by ben42153251
Link to comment
18 minutes ago, ben42153251 said:

Yes I do understand how parity works, read my comments.

Many of your comments refer to parity having an understanding of what file system it is protecting.  I would think that the changes you are talking about would require substantial changes to the file system layer.   Since this is not Unraid specific I would think that any changes at this level should be undertaken by the maintainers/developers of the file system in question.

Link to comment
  • 2 months later...

I like this idea.


I don't even really see why btrfs needs to be involved though. You just need the concept of a second simulated failed disk that has the file that failed hash check. That dynamix file integrity plugin could probably be linked up to that. Make sure you keep "write corrections to parity disk" off


I suppose you could do it now by temporarily unplugging the disk but that is hardly seamless.


Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.