bitrot

[tucansam]

Interesting article.

 

http://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/#image-3

 

How are you combating it?

[garycase]

That's why it's a good idea to maintain checksums of all your data.  I run a full validation of the checksums about once/year.

 

[tucansam]

Are there any howto's on how to accomplish this?  I've been reading up on it a little, but I've not seen anything that talks about what to do if and when an error is found.

[sureguy]

If it's bad check your backup and restore from there

 

Sent from a mobile device, sorry for any typos.

 

 

[garycase]

... what to do if and when an error is found.

 

Clearly you simply restore the file from your backup.  Of course you should also confirm the checksum of the backup is good ... but the likelihood of BOTH the original file and the backup having the same failure due to bitrot is VERY low !!    Especially if you validate all your checksums on a periodic basis [As I noted, I do this once/year -- on both the server and all my backup disks].

 

[tucansam]

How frequently should one be doing checksums in order to stave off bitrot?  If I do it once a year, and I back up my server once a week, there is a high degree of probability that I will not catch it until its too late.  On the other hand, it seems that doing the checksums weekly is possibly overkill? 

 

I don't want to bring up any bad words here, but I had been thinking about building a NFS-based server for just my important data, and letting unraid continue its excellent job of handling my media.  But some of what I read last night about some of the different NFS products (specifically FreeNAS) led me to believe that I had to have God's own hardware (server grade everything and a startling amount of ram) for it to be safe, and even then, if I sneezed near the server I would become one of the many people who have lost it all.

 

[garycase]

Are there any howto's on how to accomplish this?

 

There is some discussion in the forum on ways to do this in Linux (e.g. on your UnRAID box).  Personally, I do it all from Windows, using the excellent Corz Checksum utility.

http://corz.org/windows/software/checksum/

[tucansam]

... what to do if and when an error is found.

 

Clearly you simply restore the file from your backup.  Of course you should also confirm the checksum of the backup is good ... but the likelihood of BOTH the original file and the backup having the same failure due to bitrot is VERY low !!    Especially if you validate all your checksums on a periodic basis [As I noted, I do this once/year -- on both the server and all my backup disks].

 

 

Makes, sense, yes.  Are you doing this with scripts, or does unraid have an add-on that handles this specifically?  I know the checksum add-on is there, but I'm wondering about specifically comparing a copy to an original (in mass)

[tucansam]

Are there any howto's on how to accomplish this?

 

There is some discussion in the forum on ways to do this in Linux (e.g. on your UnRAID box).  Personally, I do it all from Windows, using the excellent Corz Checksum utility.

http://corz.org/windows/software/checksum/

 

 

Perfect, thank you, I will be looking into this!!!

[garycase]

How frequently should one be doing checksums in order to stave off bitrot? 

 

Depends entirely on how dynamic your data is.  But basically you always want your checksums to be current -- I compute and save the checksums along with all new data I store on the server.    The once/year validation is simply when I run a full "Validate checksums" for the entire server.

 

[tucansam]

How frequently should one be doing checksums in order to stave off bitrot? 

 

Depends entirely on how dynamic your data is.  But basically you always want your checksums to be current -- I compute and save the checksums along with all new data I store on the server.    The once/year validation is simply when I run a full "Validate checksums" for the entire server.

 

OK.  May I ask again, how are you doing that on the fly?  Or is it a manual process each time you copy a file(s)?  My unraid server is the primary file store for everything in the house, including My Documents redirects for all clients, so there are times when a lot of data is going back and forth, between kids saving homework, media files being copied up, etc.

[garycase]

My server is not used for redirects of My Docs, etc. ==> it's purely for media storage; and for backups.  All of the systems in the house are automatically backed up in the wee hours of the morning (3:30) via a scheduled task that (a) updates the checksums for the folders being backed up; and then backs those up.    Whenever I add new media to the server, I simply compute the checksums first ... copy the files to the server.  The process is very simple -- I have a folder "Add to Server" on my main PC.  Anything I want to add I just move to that folder.    Before I actually write them to the server I just right-click on "Add to Server", select "Create Checksums" ... and let it do that.  Then I just copy everything to the server.

 

[jumperalex]

Are there any howto's on how to accomplish this?

 

There is some discussion in the forum on ways to do this in Linux (e.g. on your UnRAID box).  Personally, I do it all from Windows, using the excellent Corz Checksum utility.

http://corz.org/windows/software/checksum/

 

Gary, not sure if you've noticed this or not but Corz now has an alpha linux version of his checksum utility.  It is included in the extras folder of his download.  The "problem" is that it is a manual install for KDE.  I'm not ready to build a VM to run this in, but that would make it SO much faster than going over the network. 

 

Not real point other than raising awareness.

[vm]

Gary, not sure if you've noticed this or not but Corz now has an alpha linux version of his checksum utility.  It is included in the extras folder of his download.  The "problem" is that it is a manual install for KDE.  I'm not ready to build a VM to run this in, but that would make it SO much faster than going over the network. 

 

Not real point other than raising awareness.

 

Thanks for the tip!  I've been experimenting with the linux version. Runs fine from the command line without needing KDE -of course one loses the one-click GUI integration when using the shell, but that does not matter to me. 

 

The linux version is a bash script, that calls various pre-installed linux programs depending on the algorithm you select (e.g. md5sum, sha1sum,  sha256sum, sha512sum etc).

 

It does not appear to support the same level of configuration options as the windows version, but one can readily edit the bash script to customize it (GPLv2+)

 

By the way, I only copied the single "checksum" script to /usr/local/bin. I did not use any of the other files in the zip file. To use the verify function, you need to use the same script but the script must be named "verify", so I created a softlink  ( ln -s checksum verify ). When the script runs it check the name under which it was invoked, so if you run it as "verify" it performs the verify function.

 

Edit: it appears you can also do a verify by invoking checksum with the --verify switch, but I cannot get it to work (maybe my command syntax is wrong)

 

victor

[jumperalex]

Running a vm now has me considering reengaging on this now.  Add a KDE gui to arch. Assuming it won't run under xfce

 

Sent from my Nexus 7 using Tapatalk

 

 

[vm]

Running a vm now has me considering reengaging on this now.  Add a KDE gui to arch. Assuming it won't run under xfce

 

Sent from my Nexus 7 using Tapatalk

 

I'll be interested to hear how you make out if you decide to try it. I've been running the checksum script through a few scenarios and have found some performance issues.

 

For me, the killer feature of this utility is the "append" feature where it will detect new files and add those incrementally to an existing hash. This is what attracted me to trying out the windows version in the first place.  But the implementation in the linux version needs some optimization. Essentially it loads the existing hash file into an array, and then for each file on your disk, compares it to see if it's in the array, if not then it's a new file that must be checksummed. Perfectly reasonable.

 

However, by design, it stores all the hashes in one file in the parent folder you specify (for example, if you run "checksum /mnt/disk1", it will put *all* the checksums for disk1 into one file at the root of disk1- it does not have the capability to create an individual hash file in each folder the way you can in the windows version. This means the comparison loop is very expensive because for each file on your disk, it has to search through the entire hash file.

 

For example, I created hashes for disk1, and it took about 4 hours and ran at about 20% CPU to create SHA1 hashes. Immediately afterward, before adding any new files to the disk, I ran checksum again. It ran for about 90 minutes, pegging one core at 100%, when it was finished it had not added anything as expected (since nothing changed on the disk). As far as I can tell the whole time was spent in the loop that does the comparison to see if a file on your disk already exists in the hash file. No time was spent on actually computing hashes of course.

 

I have considered modifying the script to improve the loop performance, but I'm not a bash optimization wizard (and maybe bash is not even the right tool for this type of work). Alternatively, I've thought about making it store folder hashes in *each* folder the way you can do on the windows version, this would minimize the overhead of the array comparison as it would only be checking a directory's worth of files and not the entire disk over and over.

 

I may just use the windows version   Although the linux version is much faster to initially build the hashes (because it can run at near disk speed as opposed to over the network), the subsequent "append" runs are very slow.

 

I'm not being critical of the author who has clearly marked this script as an "alpha" release.  I am very appreciative of the script and the ideas it has given me. Prior to discovering it I was working on my own checksumming implementation (like a number of other people in these forums).

 

It's also possible I'm way off base and using the tool improperly  

 

victor

 

 

[vm]

I've made some progress in getting the corz checksum script to create hash files per folder. This is a quick and dirty fix. Not sure if it will work with GUI integration.

 

I modified two find commands so that they don't recurse by adding "-maxdepth 1"

 

in the "create" section

eval "find . -maxdepth 1 -type f ${fmask} -print" | while read i; do

 

in the "verify" section

find . -maxdepth 1 -type f  \( -name "*.$hashext" -o -name "*.${hash_types[0]}" \

 

Then I invoke it as follows, effectively feeding it the directories I'm interested in one-by-one. In this example I'm feeding it the entire disk1, but I could of course use a subtree.

 

# create checksums
find /mnt/disk1 -type d -exec checksum "{}" \;

# verify checksums
find /mnt/disk1 -type d -exec verify "{}" \;

 

The other detail I have to solve is that, with these changes, the script is now creating "empty" hash files when it encounters an empty folder. Not fatal, just untidy.

 

EDIT:

This modification has solved the performance problem. It took about 7 hours to generate the initial hashes on my entire array (one hashfile per directory), and about 8 minutes on the subsequent run to look for any *new* files that needed to be hashed (appended). Of course that doesn't include any time actually spent creating new hashes.

 

One caveat if you decide to use this one-hashfile-per-directory approach - the script overwrites its error logs on each run, and although there is a switch to preserve the log, it will place it in the parent folder of where it was invoked. This means you could potentially have error logs throughout your file system. Hard to manage. In my case, I edited the script to store logs in a fixed location.

 

victor

[BobPhoenix]

Here is the script I've been using that I cobbled together out of various posts I've read.  Not as advanced as many and has hard coding that has to be changed for each unRAID server I'm using it on but works for me.

# Script to Create MD5 Hashes of Data Files on Array Disks
#!/bin/bash 
dt=$(date +"%Y-%m-%d")


mkdir /mnt/Backup
mount -t cifs -o user=backup,password=backup //192.168.1.177/Backup /mnt/Backup
mkdir /mnt/Backup/N40L/MD5_${dt}


ARRAY="$(find /mnt/ -type d -maxdepth 1 -name disk* -print)" 
for DIR in $ARRAY
do	
DISK="${DIR##*/}"
find /mnt/${DISK} -type f -exec md5sum {} \;>> /mnt/Backup/N40L/MD5_${dt}/${DISK}.txt
done


tar czf /mnt/Backup/N40L/MD5_${dt}.tgz /mnt/Backup/N40L/MD5_${dt}/*
rm /mnt/Backup/N40L/MD5_${dt}/*.txt
rmdir /mnt/Backup/N40L/MD5_${dt}

 

 

It creates a separate file for each drive (empty file if drive empty) then archives the files in the Dated directory to an archive and deletes the md5 text files.  I just copy the script in my go file currently but am planing on changing to a cron table entry so that it runs in the middle of the month since it runs for days and I don't want it to conflict with parity checks.

 

 

cp /boot/MD5Auto.sh /etc/cron.monthly/MD5Auto.sh
chmod +x /etc/cron.monthly/MD5Auto.sh