Backups & bit-rot

[Spectrum]

Now that I have my unRAID box set up and stable, I'm setting up some rsync scripts to back up data from my primary workstation.  I have a pretty extensive photo library of Canon raw files (that keeps growing )and want to keep them mirrored on the unRAID box for security.  In the past, I have almost lost some images due to bit-rot.  Example: I pulled up a photo I knew was good and it had fluorescent streaks and patches.  I pulled a copy from my offsite backup and it was fine.

 

My concern here that if I get bit-rot on my primary workstation, I don't want it mirrored to the unRAID box.  As a test I did the following:

• Created a 15MB binary file on the unRAID box
  
• Copied the file to my windows box
  
• Changed 1 bit in the file
  
• Copied it back to the unraid box
  
• Used touch to make the modified file's timestamps match the original file's
  
• Copied the file back to my windows box
  
• Ran rsync with the modified file as the source and pointed the dest to the location of the original file on the unRAID box
  

 

Doing a cmp on the modified file on the windows box and the original file on the unRAID server showed the files as being different after the rsync, so rsync did not copy the "bad" file over the "good" file.  Based on this I think it is safe to proceed with scripting and forgetting, but I wanted to check in here to see if anyone else had any thoughts or insights.

[Joe L.]

Now that I have my unRAID box set up and stable, I'm setting up some rsync scripts to back up data from my primary workstation.  I have a pretty extensive photo library of Canon raw files (that keeps growing )and want to keep them mirrored on the unRAID box for security.  In the past, I have almost lost some images due to bit-rot.  Example: I pulled up a photo I knew was good and it had fluorescent streaks and patches.  I pulled a copy from my offsite backup and it was fine.

 

My concern here that if I get bit-rot on my primary workstation, I don't want it mirrored to the unRAID box.  As a test I did the following:

• Created a 15MB binary file on the unRAID box
  
• Copied the file to my windows box
  
• Changed 1 bit in the file
  
• Copied it back to the unraid box
  
• Used touch to make the modified file's timestamps match the original file's
  
• Copied the file back to my windows box
  
• Ran rsync with the modified file as the source and pointed the dest to the location of the original file on the unRAID box
  

 

Doing a cmp on the modified file on the windows box and the original file on the unRAID server showed the files as being different after the rsync, so rsync did not copy the "bad" file over the "good" file.  Based on this I think it is safe to proceed with scripting and forgetting, but I wanted to check in here to see if anyone else had any thoughts or insights.

rsync has two different methods to detect changed files.  The default is to match time-stamps and file sizes.  That is why it did not overwrite the server version of the file.  You can use an option to have it do a bitwise compare and update.  This takes a lot longer (and why it is not the default)  Your test confirms the first method works as you would desire for your use.

 

A bigger issue is the underlying cause of your "bit rot"  You might want to run a memory test on your PC to see if it is working properly.  (You can boot the Window's box on the unRAID flash drive and select the memory test.  Just don't boot into unRAID and assign drives.  You can boot unRAID and might want to to be able to run "smartctl"  and get smart reports on the window's drives to detect it they are the cause of your "bit rot" )

 

Joe L.

 

Joe L

[Spectrum]

rsync has two different methods to detect changed files.  The default is to match time-stamps and file sizes.  That is why it did not overwrite the server version of the file.  You can use an option to have it do a bitwise compare and update.  This takes a lot longer (and why it is not the default)   Your test confirms the first method works as you would desire for your use.

Good to hear

 

A bigger issue is the underlying cause of your "bit rot"  You might want to run a memory test on your PC to see if it is working properly.  (You can boot the Window's box on the unRAID flash drive and select the memory test.  Just don't boot into unRAID and assign drives.   You can boot unRAID and might want to to be able to run "smartctl"  and get smart reports on the window's drives to detect it they are the cause of your "bit rot" )

 

The hard drive that those files lived on when that happened finally gave up and died and the other guts are actually in my HTPC now.  I memtest and prime95 test for at least 72 hours before I consider a new box "stable" so I'm really not that concerned with memory/cpu corruption, more if the underlying drive starts going out.  Not to say mem/cpu won't go south, but I can't spend *all* my time worrying, I have too many things to do!