Sorry for a long post, but I think you will find it useful.

 

INTRO

I think we can all agree that unRAID is an incredible tool to help safeguard our data.  Although we all understand and agree that it is not a backup solution, we all realize that backing up terabytes, or even tens of terabytes of information is just not practical.  Re-ripping hundreds of CDs, DVDs, and/or BluRays could easily take 100s or even 1000s of hours for some people.  Recovering lost TV shows recorded through PVR or downloaded might be impossible.  Although not a backup, the truth is for 90% of our unRAID data – losing it would be, at best, a huge PITA!

 

So we begin to ask ourselves – what are our exposures to data loss?  What can we do to protect it?  There are the physical things – like keeping it cool and using a UPS – there are the preventative things – like running parity checks and smart reports.  But how can we protect ourselves EVEN more?  What are the real risks for data loss?

 

CAUSES OF DATA LOSS / CORRUPTION

My experience is that the greatest risk, right now, involves situations that causes unRAID to crash or report disk related errors in the syslog.  Read the forums very long and you will read about people running a parity check after such a situation and getting parity sync errors.  Sometimes there are 1000s of them, sometimes just a handful.  They can often be traced back to bad memory, lose cables, a bad drive, a power outage, or a server crash but sometimes they are a mystery.  And usually an explanation of why it happened is not nearly as important to the affected individual as what impact it had on data integrity.

 

Tom’s offer to create a non-destructive parity check will help enormously to prevent situations where good parity is destroyed due to a memory error or other nasty problem that would otherwise cause the parity drive to be overwritten by garbage.  It cannot, however, address a smaller but insidious possibility of a few misguided sectors getting corrupted.   This becomes particularly concerning when these things happen in concert with the need to rebuild a disk.  

 

Here is a recent example (sorry jimwhite for bringing this up) of the kind of issue I am describing, but the forums likely get one of these every month or two.  The dialog usually goes something like this?

- What happened to cause this?

- Bad cables, lost power, bad drive, bad memory, server crashes …

- Okay, I fixed the problem.  Did I lose any data?

- I doubt it.   ResiserFS is very good at keeping the data accurate.

- But I did a drive rebuild - how would I know for sure?

- Compare all of your data with your backups and see.  

- Ugh!  I guess I’ll just have to live with it.

 

Another situation that arises less frequently, but is often traumatic to the affected person, is the need to run ReiserFsck to recover data.  When it is done it can be very hard to know what data was accurately recovered, what files are misnamed, what files are missing, and what files are corrupted.  Answering these questions is impossible without a recent backup.

 

SOLUTION

It is important to understand that these issues are no fault of unRAID.  RAID5 treats even a hint of data corruption very seriously and will allow the entire RAID array to be lost rather than risk corrupting any data.   unRAID is not that strict – it tells you when it gets errors but does its best to go on.  That’s how we want it.  But a byproduct is the exact situation we're discussing.

 

I have thought about this problem for a while, and have come up with (what I hope) is a good solution.  The solution is to create PAR (PAR2 to be more accurate) recovery sets on my drives.  For those that don’t know what this means, read about it here.

 

I ran an experiment on a near full 1T drive that had a bunch of very large files on it.  I had just over 1G available on the disk (it was almost full), and decided to create enough recovery blocks to fill up that space.

 

PAR blocksize (the size of a recoverable unit) can vary in size, so I initially wanted the blocks to be very small.  1G worth of 4K blocks could fix a lot of errors.  But PAR limits you to a maximum of about 32K blocks total.  That meant that my blocksize had to be about 30Meg (1T / 32K ~= 30M).  At 30Meg per block, I was able to have 35 blocks for recovery within my 1G of free space.

 

The recovery blocks are not nearly as important as just the ability to DETECT exactly which files are bad, and PAR could do that even with ONE recovery block.  But the 35 30Meg blocks would likely recover the kind of damage I frequently see, and if it weren’t enough, at least I’d know what files were affected.  I decided 1G is enough to do a reasonable PAR set, but you can decide for yourself.  If you want 10G of PAR files you could RECOVER from 10x the amount of damage.    With PAR space >= to the size of your data files, you could even protect yourself from an accidental deletion of a file.  You can keep the PAR files on a different disk if you like.

 

Computing the PAR blocks is a bit time consuming.  My experiment took close to 3 DAYS to build the PAR files on my 1T WD EACS GP drive with 1.6GHz E2140 dual core CPU.  But you only have to build once.  Checking it is faster (a little slower than a parity check - about 8 hours on my system – and I did it over the network using Windows quickpar).  PAR also has issues with traversing subdirectories, but I have some ideas as to how to overcome that.

 

An alternative to this whole PAR based scheme would be some type of process that computes and saves checksums.  There is md5sum built into Unix but also Simple File Verification (SFV) format.  This would be a good way to detect errors, but not correct them.   (Detection is the biggest problem for most of us.)  It would likely be quite a bit faster too.  But I like the PAR files.  It lets me fill each disk with useful redundant into for recovery, space that otherwise I likely would not use.  It has cross platform tools.   And it is darn cool!

 

Combining unRAID for drive level recovery with PAR2 for data level recovery is a pretty awesome combination that will protect you from multiple angles.  My next post will be about how to install a halon system in your home.  

 

The thing about par based recovery, is if you change 1 bit in the source, all the pars potentially have to be recalculated.  Theoretically, an algorithm can be developed to mitigate that burden, but it isn't within the paradigm of what PARs were developed for.

 

It only works for static data.  Now the way a lot of people use unRAID, there is a lot of static data, but it has to be absolutely static for PARS to be effective.

 

Tom's idea of a second parity disk would work dynamically and accomplish the same thing.

 

Even for my "static" content, it sometimes has to change.  For example, my media manager pills in cover art.  Every so often, it will recheck for cover art for things that don't have cover art.  Another change is when subtitles get added when relatives visit (are ore going to visit).

Computing the PAR blocks is a bit time consuming.  My experiment took close to 3 DAYS to build the PAR files on my 1T WD EACS GP drive with 1.6GHz E2140 dual core CPU.  But you only have to build once.  Checking it is faster (a little slower than a parity check - about 8 hours on my system – and I did it over the network using Windows quickpar).  PAR also has issues with traversing subdirectories, but I have some ideas as to how to overcome that.

 

I've used QuickPAR in the past to protect photo sets and mp3 files.  The subdirectory traversal issue has kept me from using it more extensively... I just don't like the idea of having PAR files laying around in every media subdirectory, so as for whatever ideas you might have, I'll look forward to seeing them.

 

I'm really looking forward to the idea of a second Reed-Solomon parity drive being added to unRAID, but of course this doesn't address the concern about data getting corrupted in transit from a workstation to the unRAID server and the unRAID server thinking everything is OK and creating parity information consistent with the bad data.

 

An alternative to this whole PAR based scheme would be some type of process that computes and saves checksums.  There is md5sum built into Unix but also Simple File Verification (SFV) format.  This would be a good way to detect errors, but not correct them.   (Detection is the biggest problem for most of us.)  It would likely be quite a bit faster too.  But I like the PAR files.  It lets me fill each disk with useful redundant into for recovery, space that otherwise I likely would not use.  It has cross platform tools.   And it is darn cool!

 

If I do generate PAR sets for data before moving to the unRAID server, I can always take just the base .par file, around 1K in size, and check the data after copy.  Obviously I need the additional PAR files if I need to reconstruct the data.

 

If the file(s) originated from a torrent site, I often force a re-check of the data once it's on the server using the client.

 

It would be nice to have an unRAID friendly network move utility that does a copy then sha-1 (better than md5) verify before local delete.  I know there were other forum threads about alternative network copy clients, but I think it would be cool to see one that is optimized for us. 

 

Combining unRAID for drive level recovery with PAR2 for data level recovery is a pretty awesome combination that will protect you from multiple angles.  My next post will be about how to install a halon system in your home. 

 

Actually Halon for new installations has been banned by the Montreal Protocol, which I remember only because it was on my CISSP exam.    The data center where I work, built recently, uses Inergen.

 

Great post and thanks for all your hard work in these forums!

 

Actually Halon for new installations has been banned by the Montreal Protocol, which I remember only because it was on my CISSP exam.    The data center where I work, built recently, uses Inergen.

 

Great post and thanks for all your hard work in these forums!

 

It will take a little time to respond to your post but wanted to say a quick thank you for your comments and kind words.

 

The real problem with Halon (and likely Inergen as well although I am not sure), is that it sucks all of the oxygen out of the air.  That's how it puts out a fire.  So if you had it in your house, your house and equipment might survive, but you'd be a goner. 

So if you had it in your house, your house and equipment might survive, but you'd be a goner.

 

Not at all... that's a common myth.

 

Combustion takes a higher O2 level than human respiration.  Halon will lower the O2 level so it won't support combustion, but will support human respiration.

 

I've been in a machine room (1980's) when the Halon system discharged (fire on a tape reader).  No problems.

Did you guys start a new topic... or is that as far as you got ?

I think you shoudl change where it says raid5, to 'some traditional raid levels'

 

as raid5 has long since overcome data currptions stopping a rebuild... like, 10-15 years ago.

I'm currently building a new unraid server and will try ICE-ECC for additional protection. Any experiences with this one ?

 

It uses Reed-Solomon but has the ability to traverse folder structures etc.

 

Tom's idea of a second parity disk would work dynamically and accomplish the same thing.

 

I don't mind the idea of a second parity disk, but my reality is as many others, I suspect... I have many TB of data that are NOT backed up.  They are my DVD/Blu-ray and CD audio collections, some personal video projects, and there are some data that are there as backups for my personal files.

 

So a second parity drive would be fantastic in certain cases.  One I could think of would be if one parity drive failed.  When the replacement was put in, the second parity drive would still be "good" and a simple bit-for-bit copy of it could be transferred to the replacement drive in a heck of a lot less time than a full rebuild.  (Tom that's a hint to make that functionality available if/when you do get a second parity drive option, please with a cherry on top!)

 

But that's a far-fetched scenario.  And PARs can be useful in certain circumstances, too.  But here's the thing - neither PAR data nor multiple parity drives provide one with BACKUP - they simply provide recovery options (and corrupt files detection in the case of PAR).

 

Ultimately, I think the solution is to actually have a backup in the proper sense of the word.  Corrupt files may still be a nuisance, but there are ways of making them stand out.

 

My solution for backup currently is a little ditty called Second Copy.  It isn't perfect but since most of my work is done on the Windows platform, it works for what I need.  To backup my entire UnRAID array, I need to build a second UnRAID server.  I plan to put it in my shed, which has a bit of distance between it and the house - should ever we be so unfortunate to experience a house fire we can't control, I'm hopeful the shed will contain our digital lives in the aftermath.

 

So Second Copy would analyze which files on my UnRAID server had changed periodically, then copy them to the UnRAID box in the shed.  To mitigate file corruptions, SC has an option to save "previous versions" of files.  So I could have 30 previous versions of any files that change and just go back to the one that still works.  This protects against me being careless and permanently deleting a file from my array also.

 

In the end, I think the thread title is important - and I fully believe that a backup is needed when TB of data are "online".  In my case, just re-ripping my CD collection takes 6 solid weeks of almost every waking hour after work.  I don't need to do that too many times - life's too short, and UnRAID is affordable enough that two are better than one!

 

 

I don't mind the idea of a second parity disk, but my reality is as many others, I suspect... I have many TB of data that are NOT backed up.  They are my DVD/Blu-ray and CD audio collections, some personal video projects, and there are some data that are there as backups for my personal files.

True.  An unfortunate reality.

simple bit-for-bit copy of it could be transferred to the replacement drive in a heck of a lot less time than a full rebuild.  (Tom that's a hint to make that functionality available if/when you do get a second parity drive option, please with a cherry on top!)

Actually, that is not true. 

  - First a drive is removed from service (disabled) when a "write" to it fails.  Therefore, even if a drive is currently readable, it does NOT have the correct contents. 

  - Second, most of the time when a disk fails it is un-readable. You cannot copy from it. 

  - Third,  The time to re-construct a disk on a modern motherboard (where the PCI bus does not limit throughput) is not going to be much more than the time to copy from one disk to another.  The bottleneck is the speed at which you can write to the replacement drive.  That is slower than the read speeds of the other drives.

But that's a far-fetched scenario.  And PARs can be useful in certain circumstances, too.  But here's the thing - neither PAR data nor multiple parity drives provide one with BACKUP - they simply provide recovery options (and corrupt files detection in the case of PAR).

True too.

Ultimately, I think the solution is to actually have a backup in the proper sense of the word.  Corrupt files may still be a nuisance, but there are ways of making them stand out.

 

My solution for backup currently is a little ditty called Second Copy.  It isn't perfect but since most of my work is done on the Windows platform, it works for what I need.  To backup my entire UnRAID array, I need to build a second UnRAID server.  I plan to put it in my shed, which has a bit of distance between it and the house - should ever we be so unfortunate to experience a house fire we can't control, I'm hopeful the shed will contain our digital lives in the aftermath.

 

So Second Copy would analyze which files on my UnRAID server had changed periodically, then copy them to the UnRAID box in the shed.  To mitigate file corruptions, SC has an option to save "previous versions" of files.  So I could have 30 previous versions of any files that change and just go back to the one that still works.  This protects against me being careless and permanently deleting a file from my array also.

That will work.  The "rsync" command built into unRAID will also do as you described if invoked with a bit of scripting.  It is smarter in that it will only "sync" those files that have changed, and it does verification checksums to make certain files are copied accurately.

In the end, I think the thread title is important - and I fully believe that a backup is needed when TB of data are "online".  In my case, just re-ripping my CD collection takes 6 solid weeks of almost every waking hour after work.  I don't need to do that too many times - life's too short, and UnRAID is affordable enough that two are better than one!

I ripped my collection over a period of several months.  My "backup" of my movies are the physical disks sitting on the shelves in the theater.  I too would feel the pain if I had to do it again.

 

I recently built a second server, exactly as you described.  My plans are to keep the two in different locations and periodically sync the two.

 

On the new server, I have the "rsync" daemon process running.  It is started by this command:

rsync --daemon --config=/boot/config/rsync.conf

 

The rsync.conf file looks like this:

uid            = root

gid            = root

use chroot      = no

max connections = 4

pid file        = /var/run/rsyncd.pid

timeout        = 600

log file        = /var/log/rsyncd.log

 

[mnt]

    path = /mnt

    comment = /mnt files

    read only = FALSE

 

On my first server, I can run these commands to sync everything to the new server. Only those files that are new or have changed are copied.  I just ran it, and it took all of 6 minutes to sync the two servers (5.5TB of data):

cd /mnt

rsync -avrH user/Pictures tower2::mnt/user/

rsync -avrH user/Movies tower2::mnt/user/

rsync -avrH user/Mp3 tower2::mnt/user/

rsync -avrH user/TV tower2::mnt/user/

rsync -avrH user/data tower2::mnt/user/

 

I do not keep multiple incremental copies of files, but I've read of rsync" scripts that do.

 

Joe L.

I really like the rsync idea, since it's "bulit-in" and I could feasibly set it up as a cron job to have one server sync itself with the other.

 

SecondCopy is my rsync right now because I understand it a whole lot easier.  It also only copies files that have changed, and the option to copy only (rather than fully synchronize) means previously deleted files on the main server will still exist on the backup.

 

This is not always desireable of course, and the fact that I need a 3rd machine running to do the sync with SecondCopy is silly in the big picture.

 

But I shall learn more about rsync because I need to do similar backups at work where I have an archive server, and a backup archive server.... and they need to sync themselves.  So I'll get there 

My solution for backup currently is a little ditty called Second Copy.  It isn't perfect but since most of my work is done on the Windows platform, it works for what I need.  To backup my entire UnRAID array, I need to build a second UnRAID server.  I plan to put it in my shed, which has a bit of distance between it and the house - should ever we be so unfortunate to experience a house fire we can't control, I'm hopeful the shed will contain our digital lives in the aftermath.

 

That gives me a good idea about creating a second unRAID server but keeping it at a relative's house and making

their access read-only to the videos. Although distance is further, using rsync to update new additions won't be that

bad if done overnight.  This way there's backup in case of fire or theft.

 

Now if I can only convince them they need to pay for the 2nd server and 10TB of HDDs

That gives me a good idea about creating a second unRAID server but keeping it at a relative's house and making

their access read-only to the videos. Although distance is further, using rsync to update new additions won't be that

bad if done overnight.  This way there's backup in case of fire or theft.

 

Now if I can only convince them they need to pay for the 2nd server and 10TB of HDDs

 

I've been considering a friend's house for the same reason!  If I put an old wok on my wireless router's antenna, I could easily make a point-to-point to my pal's house, and store a second server there.  Same idea, rsync overnight, read-only access to my files on their end, and on my end I'd back up their files, too with read-only access for me.

 

Win/win methinks... But 20TB is approaching quickly here, so full mirroring is getting more costly