ben42153251 Posted October 10, 2020 Share Posted October 10, 2020 (edited) I have an idea that could potentially solve bit-rot on UNRAID when the user uses a BTRFS file system. BTRFS checksums data and upon reading, if the data does not match the checksum it will report an error and refuse to read the corrupted data. If UNRAID parity was changed from real-time to very slightly delayed, if BTRFS threw a checksum error, UNRAID could mark the drive as bad and not include corrupted data into the parity allowing the user to restore the good copy of the data when replacing the hard drive. Implementation strategies could include using the mirrored cache drives to store real-time changes for the parity BUT NOT IMMEDIATELY COMMIT THEM, such that if BTRFS checksum throws an error, parity could DISCARD the corrupted data from being included into the array. Thoughts? Edited October 10, 2020 by ben42153251 Quote Link to comment
itimpi Posted October 10, 2020 Share Posted October 10, 2020 I think that you will find that in practice there is no practical algorithm that can be programmed to support something like this. Also it would restrict one to using BTRFS file system - again not something that is necessarily desirable. Quote Link to comment
trurl Posted October 10, 2020 Share Posted October 10, 2020 And it would likely be a major change to the way parity currently works for little or no benefit. Changing the core NAS functionality at this level would require more testing than almost anything else attempted. Expect this to be so far down the priority list to never happen. Quote Link to comment
ben42153251 Posted October 10, 2020 Author Share Posted October 10, 2020 (edited) If parity can virtualize a failed drive (continued access to data), why couldn't that parity virtualized version of the data be available to a physical working BTRFS formatted disk for the purpose of providing a second copy of the data so that BTRFS scrub could replace corrupted data? Perhaps a slightly modified version of BTRFS or interoperability layer would be needed for UNRAID but I don't see why UNRAID could not have similar levels of data integrity to ZFS mirror or BTRFS mirror. The UNRAID parity check could be modified to run BTRFS scrub PRIOR to the parity check and if an inconsistency is found then virtualize the disk reporting corrupt data and run another scrub on the physical disk using the virtualized disk as the mirror source to restore the corrupted data. Expanding my thought bubble... why couldn't UNRAID (in the background) generate parity virtualized versions of ALL physical disks at ALL times such that BTRFS would always have access to a backup copy of a file if a read checksum failed? Then just kick the corrupt disk from the array and rebuild with a new drive. To avoid a read speed penalty, BTRFS should only draw on the virtualised parity copy of the data when a checksum fails reading data off the physical disk. Edited October 10, 2020 by ben42153251 Quote Link to comment
sota Posted October 10, 2020 Share Posted October 10, 2020 so... you want unRAID to basically run the array as a RAID4 setup? Quote Link to comment
ben42153251 Posted October 10, 2020 Author Share Posted October 10, 2020 (edited) 30 minutes ago, sota said: so... you want unRAID to basically run the array as a RAID4 setup? No because each disk would still contain its own BTRFS file system rather than being striped. It may become necessary to use UNRAID to access those individual file systems if a slightly modified version of BTRFS was used (instead of an interoperability layer) but the idea is for each disk to retain its own file system. Edited October 10, 2020 by ben42153251 Quote Link to comment
ben42153251 Posted October 11, 2020 Author Share Posted October 11, 2020 (edited) I just realised that in a dual parity configuration we actually have a total of 3 copies of a file potentially available to BTRFS (including the original physical disk). When reading data only the disk containing a file would need to be spinning, BTRFS could be told another copy of the file is available but UNRAID would only need to spin up the other drives if a checksum failed and it was necessary to read the file from the parity generated copy. Essentially my whole idea is to an extent "lie" to BTRFS that it's in a mirror configuration using a virtual parity generated disk when necessary to facilitate the other half of the "mirror". Limetech please consider writing a BTRFS interoperability layer so it can use a parity generated disk to get a copy of a file if its corrupted on the physical disk. The first step is starting with the 6.9 release, make BTRFS the default file system. Edited October 11, 2020 by ben42153251 Quote Link to comment
itimpi Posted October 11, 2020 Share Posted October 11, 2020 1 hour ago, ben42153251 said: I just realised that in a dual parity configuration we actually have a total of 3 copies of a file potentially available to BTRFS (including the original physical disk This implies that you do not understand how parity works. Parity has no concept of files or data - it only understands physical sectors. It’s purpose in life is to be able to recreate a sector on a failed drive but it has no idea what that sector is being used for. That is why it can handle disks in any mix of file system types as parity operates at a lower level than file system handling. Quote Link to comment
ben42153251 Posted October 11, 2020 Author Share Posted October 11, 2020 (edited) We use the parity to generate a raw disk as if it had FAILED and mount it READ ONLY somewhere else so that when BTRFS reads off the physical disk and a checksum fails, it can check if the parity generated version of that drive contains the correct version of the file. Eg imagine standard RAID 5, 1 disk fails, parity is re-creating the whole disk virtually. Edited October 11, 2020 by ben42153251 Quote Link to comment
itimpi Posted October 11, 2020 Share Posted October 11, 2020 18 minutes ago, ben42153251 said: Yes I do understand how parity works, read my comments. Many of your comments refer to parity having an understanding of what file system it is protecting. I would think that the changes you are talking about would require substantial changes to the file system layer. Since this is not Unraid specific I would think that any changes at this level should be undertaken by the maintainers/developers of the file system in question. Quote Link to comment
scorcho99 Posted December 16, 2020 Share Posted December 16, 2020 I like this idea. I don't even really see why btrfs needs to be involved though. You just need the concept of a second simulated failed disk that has the file that failed hash check. That dynamix file integrity plugin could probably be linked up to that. Make sure you keep "write corrections to parity disk" off I suppose you could do it now by temporarily unplugging the disk but that is hardly seamless. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.