Bit-Rot

[gumby327]

Hi, yesterday I got an alert off of my unRAID servers during an rsync to my backup server.  It was the last of my really old hard drives.  Back story on that was I wanted to store videos for Plex on a windows box and bought a few refurbished 4tb drives off of Amazon.  Then a few more, then a few more.  One day I decided to try unRAID out and that is when my eyes were opened to servers and their silent killers.  Over the next few months those old drives overheated and suffered a LOT of sudden death.  Now, I am on 100% new drives and life is a little easier.  Anyway my initial backup share was the last of those old drives and when it went out I thought no biggy, pop in a new one and it will restore from parity.  I have been trying out Dynamix File Integrity plugin this past week and it had completed no errors.  I wanted to inspect my Acronis backup files to see that they were still viable.  Sure enough I had massive bit-rot.  None of the PC's requiring restoration (were still working), I was happy to have learned it now rather than later.  I had lost a quarter of my backup files.  When?  was it Windows, or unRAID?  who knows?  All I can say is bit rot is real and never fall into the happy ignorance that you have backups so you are good.  Test your data quality as part of a regular routine.  Over the 4 decades I have been at this I have lost digital content, lots of it.  The stuff most precious to you and your family is normally the first to go.  In my professional life we often call it digital Alzheimer's when the data is right in front of you, but you have no idea some is missing.

Edited February 20, 2022 by gumby327

[Ystebad]

Doesn't ZFS fix this issue?  This is one of my big concerns with using Unraid as my data storage is they appear to be of the mindset that bitrot isn't a significant concern.  scary stuff

[bonienl]

I can only speak for myself, I use the Dynamix File Integrity plugin to monitor bitrot.

Never had any occurrence in the last 6 years, besides some false positives due to file processing.

Though I keep monitoring, bitrot isn't a real issue for me.

 

[itimpi]

50 minutes ago, Ystebad said:

Doesn't ZFS fix this issue?  This is one of my big concerns with using Unraid as my data storage is they appear to be of the mindset that bitrot isn't a significant concern.  scary stuff

In my experience bit rot is very rare.   If you are really concerned about bit rot then you can use btrfs as the file system on the array disks as btrfs has automatic built-in bit rot detection.   The alternative is to use XFS (which I find more resilient) in conjunction with the file integrity plugin.   

[Ystebad]

Never heard of the file integrity plugin.  Will have to investigate.  If this is a good thing, why would it not be incorporated into Unraid itself??

 

[itimpi]

1 minute ago, Ystebad said:

Never heard of the file integrity plugin.  Will have to investigate.  If this is a good thing, why would it not be incorporated into Unraid itself??

 

Every feature added to Unraid adds cost from a development and testing prospective so a trade-off has to be made.   Since btrfs is provided as standard and already includes bit rot protection that is probably deemed sufficient.

[Ystebad]

Understood, but I selected XFS based upon recommendations that it's what Unraid recommends for improved ability to recover from unclean shutdowns (which have happened to me multiple times due to unraid crashes).

 

So now I guess I'm feeling trapped in the middle - should I change disks to btrfs for bitrot protection but risk unclean shutdown corruption?  Or stay with recommended XFS and not have it.

 

I don't know enough to know what to do, just wish there was a solid solution that didn't have the downsides.

[itimpi]

Just now, Ystebad said:

Understood, but I selected XFS based upon recommendations that it's what Unraid recommends for improved ability to recover from unclean shutdowns (which have happened to me multiple times due to unraid crashes).

 

So now I guess I'm feeling trapped in the middle - should I change disks to btrfs for bitrot protection but risk unclean shutdown corruption?  Or stay with recommended XFS and not have it.

 

I don't know enough to know what to do, just wish there was a solid solution that didn't have the downsides.

I have never had bit rot in the 10 years or so that I have used Unraid as far as I can tell so I certainly do not think it is common.    However, I HAVE installed the file integrity plugin so if it does occur it will hopefully be detected.   I prefer to stick with XfS for the main array as being more robust and probably more performant.

[Michael_P]

4 hours ago, itimpi said:

I have never had bit rot in the 10 years or so that I have used Unraid as far as I can tell so I certainly do not think it is common.

 

Indeed, and far less than that would be from "silent" corruption. Drives have ECC already for just such things and would start spewing SMART errors if it could not be corrected on the fly. 

 

If they lost a quarter of their data, it ain't from bit-rot lol 

[TheLinuxGuy]

On 2/21/2022 at 11:15 AM, itimpi said:

 If you are really concerned about bit rot then you can use btrfs as the file system on the array disks as btrfs has automatic built-in bit rot detection.  

 

Can you expand this a bit, is there any documentation explaining how unraid's "bitroot protection" using btrfs works?

I assume that in order for this to work a 'btrfs scrub' needs to be scheduled and run often? Does unraid throw an error notification in UI when btrfs detects a checksum error or bitrot?

[JorgeB]

7 hours ago, TheLinuxGuy said:

I assume that in order for this to work a 'btrfs scrub' needs to be scheduled and run often? Does unraid throw an error notification in UI when btrfs detects a checksum error or bitrot?

You need to schedule a scrub to check all data, any time a file is accessed it's also checked, currently the GUI doesn't notify you of corruption errors, you can create a script like here:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=700582

 

 

 

[je82]

On 2/21/2022 at 4:42 PM, bonienl said:

I can only speak for myself, I use the Dynamix File Integrity plugin to monitor bitrot.

Never had any occurrence in the last 6 years, besides some false positives due to file processing.

Though I keep monitoring, bitrot isn't a real issue for me.

 

I really wish there was a way to only run this plugin on selected shares, right now (if i remember correctly) it will run on the entire array and having nearly 150tb of data and over 10 million files yeah that's not going to happen, i installed the plugin and it would take nearly a month to just process all the files once, id be fine with just protecting certain files from bitrot which are located in particular shares

[ChatNoir]

2 hours ago, je82 said:

I really wish there was a way to only run this plugin on selected shares

I'm not in front of my server to provide screenshots but I am pretty sure that you can. If not select the one you want, you certainly select the shares and file type you want to exclude.

Note that there are 2 different panels for the plugin, one in settings and one on the top tabs. I think that the fine settings are only accessible on the former.