While doing a preclear of my drives for my setup, I thought I'd do a little reading about the safety precautions for bit-rot that Unraid might have in place. I was horrified to see that there is no such functionality to use the (2) Parity drive(s) to scrub and repair silent corruption. I have seen debates about ECC memory being the source (which I do not believe based on discussions I've had with engineers who worked at Sun and Apple who worked extensively in this space with ZFS in the early days and based on their evaluations where they lined up a lot of other scenarios that simply demonstrate a big "NO" to that thought process - bit-rot unequivocally exists and happens), etc. but in the end the result is that Unraid simply doesn't appear to facilitate silent corruption detection + repair.  

 

Even tools like DYNAMIX FILE INTEGRITY PLUGIN doesn't solve the problem since it doesn't allow you to recover from a corrupted file. What if your backup(s) has captured a corrupted snapshot and you simply no longer have an uncorrupted file anymore?

 

I suppose BTRFS could be used on an individual disk basis, but again, how do you recover things quickly enough to avoid all bad scenarios? I don't see any automated solution there, but that would be nice if it exists. I think if it does exist this might be useful and I would definitely use BTRFS instead of XFS for this sanity saver. To state this even more clearly, I would LOVE to see ZFS be used in conjunction with Unraid to facilitate this perpetual scrubbing and recovery! 💙  But alas, it doesn't appear that this exists......

 

Anyone have a solution?  I'm willing to buy an additional drive or two to facilitate perpetual scrubbing recovery if I could just use Unraid due to it's expandability attribute, but at this rate I just cannot tolerate the risk of silent corruption and will have to go with a ZFS based solution... 😿

With btrfs you can detect corruption, but since Unraid uses a separate filesystem for each array data device it can never be automatically fixed, you'd need to restore from backups, if/when Unraid starts allowing zfs it will be the same, both will be able to check and restore redundant pools though.

5 hours ago, JorgeB said:

With btrfs you can detect corruption, but since Unraid uses a separate filesystem for each array data device it can never be automatically fixed, you'd need to restore from backups, if/when Unraid starts allowing zfs it will be the same, both will be able to check and restore redundant pools though.

 

When is unraid planning to move to zfs? Is that on the roadmap? 

Edited July 25, 20214 yr by johnwhicker

With ZFS one could do the (perhaps dreaded) `copies=2` which stores two copies. Definitely suboptimal, but something has to be done otherwise I don't see Unraid as being tenable now 😭

 

This makes me very very sad after spending an inordinate amount of time to get it to work on the servers I have.

17 hours ago, johnwhicker said:

When is unraid planning to move to zfs? Is that on the roadmap? 

It's been mentioned multiple times, but no ETA.

 

13 hours ago, unraidfan said:

With ZFS one could do the (perhaps dreaded) `copies=2` which stores two copies.

You could do the same with btrfs, but that is far from ideal, much better to just have backups since you can need them to recover from other situations anyway.

If you'll forgive me, your post sounds like an open threat that developers must provide xyz or you will buy something else.  If that's not what you're trying to say then you might reword it...  if this is what you meant, well, best of luck to you mate.

 

It seems to me that a second server, providing backup, is a solution.  There must be hundreds of ways to use a backup server effectively.

I don't understand you @PeteAron, are you talking to me?

On 7/24/2021 at 8:25 PM, unraidfan said:

Even tools like DYNAMIX FILE INTEGRITY PLUGIN doesn't solve the problem since it doesn't allow you to recover from a corrupted file. What if your backup(s) has captured a corrupted snapshot and you simply no longer have an uncorrupted file anymore?

The idea with the DYNAMIX FILE INTEGRITY PLUGIN is that you would run a check on a schedule often enough to detect issues before good copies are rotated out of your backup. 

Right @primeval_god but apparently SnapRAID is a more refined process and includes it in the parity storage from what I'm hearing?

Edited July 27, 20214 yr by unraidfan

On 7/26/2021 at 7:34 PM, PeteAron said:

It seems to me that a second server, providing backup, is a solution.

This simply is not true. You could be backing up corrupted files.