Bit Rot questions

[wgstarks]

I understand what bit rot is and why it's a problem, but I'm not quite sure of the overall strategy? I've seen a few discussions regarding detecting data corruption using MD5 checksums and I'm willing to accept that this works (magic ). What do you do to repair the corruption if it's detected? I haven't seen any discussion of this part of the game plan. Also, is there some very simple plugin that I can install to be able to run checksums on a cron schedule or something similar and compare them to the previous checksums with some sort of notification feature?

[JonathanM]

What do you do to repair the corruption if it's detected? I haven't seen any discussion of this part of the game plan.

Either restore the file from your backup, or if you added a par2 recovery set for those files it can check for and repair corruption up to the amount of error recovery that you saved in the par2 set.

[Brucey7]

Bitrot can be catastrophic or it can be no problem at all.

 

It all depends on the nature of your data.

 

If your data is typically media (typically the main use of unRAID), then it's largely unimportant, your movies will still play, likewise your audio files and your photos.  You want a filesystem that is unaware of bitrot.  Some filesystems will prevent access to your data with CRC checksum errors and then you are really screwed.

 

A similar analogy is old celluloid movies, the odd frame might detiorate  but you can still watch them.  ReiserFS is friendly to bitrot, I don't know about the newer filesystems, but it's definitely a consideration.

[wgstarks]

What do you do to repair the corruption if it's detected? I haven't seen any discussion of this part of the game plan.

Either restore the file from your backup, or if you added a par2 recovery set for those files it can check for and repair corruption up to the amount of error recovery that you saved in the par2 set.

My unRaid is at 13TB and growing. I haven't really found a practical backup option. Not sure what par2 is so I doubt I have it installed.

---

Bitrot can be catastrophic or it can be no problem at all.

 

It all depends on the nature of your data.

 

If your data is typically media (typically the main use of unRAID), then it's largely unimportant, your movies will still play, likewise your audio files and your photos.  You want a filesystem that is unaware of bitrot.  Some filesystems will prevent access to your data with CRC checksum errors and then you are really screwed.

 

A similar analogy is old celluloid movies, the odd frame might detiorate  but you can still watch them.  ReiserFS is friendly to bitrot, I don't know about the newer filesystems, but it's definitely a consideration.

So, if I'm understanding correctly, I don't need to worry about bitrot with my movies and audio tracks?

[Brucey7]

Unless the data loss is in the header, you won't even notice bitrot.

 

[wgstarks]

Thanks for the replies.

[WeeboTech]

Eventually I'll have a tool to sweep a disk and create a folder.par2 per folder.

This can be used to detect bitrot and possibly repair up to 10% corruption.

It will add a lil over 10% more to the size of each particular directory, but it can possible help get past a bad sector.

 

I have a few other tools I'm working on to catalog files in a SQLite table and also store the hash sums.

There's couple others I have one of which is to store the hash in the filesystem extended attributes.

 

Each has it's pros and cons in speed warehousing and functionality.

I started a conversation here if you want to review options or suggest methodology.

 

RFC: MD5 checksum/Hash Software

http://lime-technology.com/forum/index.php?topic=34988.msg325400#msg325400

[wgstarks]

Thanks WeeboTech. I've actually been following that thread. Waiting to see if your tool was something I could use. Most of the discussion has been well above my level of expertise though.