So I just finished reading a nice little fanboy war over at XBMC about freenas vs unraid and it got me thinking ... for our standard media files 1 parity worth of protection is enough (otherwise we wouldn't be using unraid), but what if we wanted more for a "small" subset of our data?  "Small" in this case is relative in so much as family photos, word documents, email backups, etc are likely orders of magnitude smaller than the terabytes of video and music files many of us have.

 

My thought was, why not keep multiple copies of that important data on the unraid box ... like ... say ... a copy on every single drive in the system, or half, or even just three or four?  That way even losing two, three, or more drives doesn't mean the loss of that important data.

 

Don't get me wrong, for the really paranoid this isn't a replacement for off-line and/or off-site backups, but it seems like a reasonable strategy to add another layer or two of redundancy for some data we might really like protected; especially as a bridge between backups.

 

Sooo all that said, is there an easy way to do this in unraid?  Set up a user share "My Important Data" with split level 0 and then have it replicated across multiple drives?  I figure a script could be used to periodically rsynch (?) between “My Important Data” and unexported shares created on the other disks? I'm only a duffer but I’d be willing to give it a shot.

 

Ideally this might someday be added as a feature.  Perhaps when you create the share there is also a replication check box with associated drive selection drop-down.  Then unraid would be able to write the duplicates in real time, or with some reasonable user selectable delay.

 

I’m purposely avoiding using the term mirror here since in my head that applies to entire drives and not subsets of data.

 

Thoughts?  Suggestions?  Pitch Forks?  RTFM’s?

 

Not sure how it would go with usershares and splitlevel. You could however copy stuff to \\Tower\disk1. Replication could be as simple as

 

rsync -av --del /mnt/disk1/ImportantStuff /mnt/disk2

 

in a script run from cron.

 

I haven't actually tested this command, you should probably check for the exact switches to use in http://ss64.com/bash/rsync.html

STFF  

 

Ideas like this have been discussed here many times in the past. Basically, set up cron to run rsync on a regular basis or something similar to that.

 

Peter

 

STFF  

 

Ideas like this have been discussed here many times in the past. Basically, set up cron to run rsync on a regular basis or something similar to that.

 

Peter

 

 

I swear I searched at work.  Then I get home, read the two responses, try another search and find the exact same question asked in March [sigh] ... thanks for indulging me guys.  Still seems like this is a "feature" worth adding because right now I'm still struggling to figure out rsync, cron, crontab, and how to make it work through reboots. :\  But I like to learn, so I have no problem searching more and reading man pages.  Cheers.

If you're going through all the trouble - maybe it's time to consider some off site backup?

 

I'm still looking for the "easy to use" UnRaid to UnRaid Offsite solution.  :-)

 

Russell

If you're going through all the trouble - maybe it's time to consider some off site backup?

 

I'm still looking for the "easy to use" UnRaid to UnRaid Offsite solution.  :-)

 

Russell

 

As I said, this isn't meant to be a replacement for off-line or off-site backup.  Just another layer of protection.  I've got the HD space, I've got the drive, and I've got the paranoia ;-)  I already backup to an external HD from my PC, and to Unraid from my PC.  But I'd like to make an unraid double drive failure less likely to take out my most recent data.  Unraid takes daily snap shots, the external is about weekly ("about" because it is not physically connected or powered to prevent electrical surge concerns so I have to do it manually).

 

For off-site I'm most likely just going to set up a sync to my FTP server hosted in Utah.  I'm in VA, so I figure I have enough geographical separation.  If both sites are taken down ... well ... let's just say I figure I'll have more important things to worry about than if I lost those pictures from that drunken night in bangkok >;-)

Sorry to bumb an old thread, but the code works Im confised though on how to set up a cron with this setting. I am basically making a duplicate copy of my important documents and phots on a black disk. This is code im using [rsync -av --del /mnt/user/Photos /mnt/disk3/BK][/font][/font][/font]. I have searched the forum for a guide in setting up cron, but I don't have the etc folder on my flash? Would just like to automate the script instead of running it from putty.

etc would exist in ram once unRAID has booted. So, you have to put the script file on the flash and then copy it to the ram filesystem once booted. Put the copy command in the go script, which is run on boot.

 

Peter

 

Be kind with my ignorance, but how would the cron file look like command wise? And would it be saved as a .sh file? And what else would be added to the go script? This is my go file:

 

 

#!/bin/bash

# Start the Management Utility

/usr/local/sbin/emhttp &

/boot/unmenu/uu

 

cd /boot/packages && find . -name '*.auto_install' -type f -print | sort | xargs -n1 sh -c

 

#!/bin/bash                                                                                   
# Copy Critical Files to unRAID Server                                                       
#                                                                                            
echo "Backing up critical files" 
rsync -av --del /mnt/user/Photos /mnt/disk3/BK
rsync -av --del /mnt/user/Music /mnt/disk3/BK
rsync -av --del /mnt/user/Jordan /mnt/disk3/BK

 

You'd make an executable script. Put in on the flash somewhere. put a copy command in the go file to copy the script to the /etc/cron.daily or /etc/cron.hourly directory. I believe something like "cp /boot/backup.sh /etc/cron.daily/" would work to copy the script called backup.sh from the root of the flash to be run daily.

 

Peter

 

Thank you for the quick reply....is the script I have up adequate? Should I add anything else to it?

I'm not much of a script person. The rsync should work if it's the correct syntax. I suppose in an ideal world that error handling would be included.

 

Also, make sure that the user shares Photos, Music and Jordan exclude disk3, or else you are just making 2 copies on the same disk.

 

Peter

Thanks I just excluded the shares from disk3. I added the line you gave me to my go scrip it looks like this:

#!/bin/bash

# Start the Management Utility

/usr/local/sbin/emhttp &

/boot/unmenu/uu

 

cd /boot/packages && find . -name '*.auto_install' -type f -print | sort | xargs -n1 sh -c

cp /boot/backup.sh /etc/cron.daily

 

 

Hope this is correct?

Just a general FYI

 

From my understanding of things behind the scenes, multiple files by the same name are only visible once, the file served is the first drive in the array that contains that file. However, UNRAID (at least with v5) does seem to know the files are duplicate and reports it in the syslog.  Things seemed to get tricky when removing a copy of the file from one drive and not others when the file removed was the first drive in the array to contain that file.

Just a general FYI

 

From my understanding of things behind the scenes, multiple files by the same name are only visible once, the file served is the first drive in the array that contains that file. However, UNRAID (at least with v5) does seem to know the files are duplicate and reports it in the syslog.  Things seemed to get tricky when removing a copy of the file from one drive and not others when the file removed was the first drive in the array to contain that file.

 

huh ... might be worth appending a suffix to the file name, maybe "BACKUP-" to avoid that issue.  Certainly easy enough to remove with a script during a restoration [shrug].  I suppose that will defeat the benefits of rsync though

Noted thanks for the heads up. Worst case disaster is to transfer to win and back to a restored array. These items are just critical and I have them backed to an external drive at my parents house.

To get around the duplicate files messages in the syslog, is this.

All you need to do is specify a different subdirectory, an easy way would be to specify the root share of "/important stuff" and put the multiple copies in different subdirectories, if it were me I'd just specify which disk I was saving to, so on disk 1 the full path in the console would be /mnt/disk1/important stuff/disk1/data directories/contents, on disk 2 it would be /mnt/disk2/important stuff/disk2/data directories/contents, etc, for as many disks as you want duplicates. It would show up in the user shares as \\servername\important stuff\disk1 \\servername\important stuff\disk2 etc etc.

 

Or did you not want to see multiple copies in the share, but still have duplication?

The script must also be executable, so in the config/go script,  after

 

cp /boot/backup.sh /etc/cron.daily

put this line

chmod +x /etc/cron.daily/backup.sh

 

Joe L.

The duplicate stuff only applies to files in the same share which are named the same and stored in the exact same directory structure on 2 different disks - which means they are exact duplicates within the same share. It is irrelevent here since the files are being copied to another share.

 

Peter

 

Regarding the duplicate files issue, I parsed this bit,

rsync -av --del /mnt/disk1/ImportantStuff /mnt/disk2

and missed this bit in the later post.

rsync -av --del /mnt/user/Photos /mnt/disk3/BK

 

The first command would need to be modified to avoid the dupe issue, correct?

Thanks joe for the added script line. I also got crash plan up and running. I would most defiantly recommend that to the UNRAID community.

I believe the original poster who posted

 

rsync -av --del /mnt/disk1/ImportantStuff /mnt/disk2

 

Was trying to say the line would copy disk2 to a directory called ImportantStuff on disk1.

 

In any case, this command line does not seem to be correct to me.

 

Peter