Bitrot detection, through filesystem or software

[Jaybau]

2 minutes ago, BRiT said:

I thought you can already get this by using BTRFS as your filesystem in your data drives. That way your array drives are parity protected and the contents of your files have Metadata checksums for the file contents.

 

Unraid has manual options/capability, but not well integrated, automated, stress free, or confidence inspiring when I'm reading posts.  Also, XFS is the preferred/recommended/default filesystem, therefore a lot of users (especially new) are not going to opt for BTRFS.

 

From my understanding you have to do a BTRFS scrub to detect a file error (manually or scheduled).  If error (you get a notification), then you have to get the parity to report an error, and select to repair the data drive from parity.  You  also need to hope your parity didn't get overwritten.  I'm also not sure what happens if there's been updated data to the same block as a corrupt file.  This may be an area between the advantages/disadvantages between realtime and snapshot RAID.  My guess is that Unraid relies on traditional backups for "snapshots", while SnapRAID and BTRFS can use a parity algorithm to accomplish the same goal via snapshots+parity.  If my guess is correct, then restoring changing data can become complicated, but simple for static data such as movie/tv.

 

Or parity report reports an error, then you have to do a full BTRFS scrub to determine if there's a problem with your data, then correct the problem with your parity.

 

If BTRFS (or DFI) do not report a problem, then you have to fix/rebuild your parity.  

 

I have more experience experimenting with SnapRAID, and so far its very easy and a lot more intuitive.  I have very little experience with BTRFS and its been too easy to the point of black box magic.  Basically with SnapRAID, and with my little BTRFS experience, you just scan for hash/checksum errors, and if error you click a button to restore.  With Unraid, lots of posts about "have parity error, what do I do?", and then a bunch of hopeful analysis steps.

 

Synology Hybrid Raid is another option that integrates BTRFS, so you can have parity + integrity + snapshots.

[Jaybau]

8 minutes ago, primeval_god said:

Personally i have no more interest in SnapRAID ... and would not trade any of its great features like realtime parity, and independent disk file systems for any other solution.

 

You don't trade any of Unraid's features with SnapRAID.  With SnapRAID, it adds integrity+snapshot parity, multiple snapshots/parity if you want, and you can store the parity wherever you want.

 

Snapraid is an application, not a file system.  There's zero risk to use for sync/scrubs.

 

Quote

waiting for someone more clever 

 

SnapRAID is pretty clever, and I don't think it would take someone clever to integrate with Unraid.

You simply configure SnapRAID what to snapshot, where to store the snapshot, where to store the metadata.  Then you just run syncs and scrubs.

 

I haven't tested Snapraid with restores and making sure there's no conflict with Unraid parity or BTRFS or DFI.  It's possible the other subsystems would need to get updated if I did a restore from Snapraid.  But for now, at least I have some additional easy and cheap protection, if I ever need it.