I would like the ability to maintain more than a single copy of some data on UnRAID (i.e. pictures). While I don't think this makes sense for everything, having this options for certain shares/folders or even drives would be awesome. I have a picture library that covers 12 years of family photos. I do maintain a copy elsewhere, but it would still be nice to have the ability to have multiple copies on my UnRAID server.

what is the total size.

 

you can setup different shares that maybe to a certain drive (maybe to multiple drives).

eg different shares Family pics 1 & backup Family pics 2.

 

otherwise build a 2nd server as backup. not all your eggs in the same basket

 

It depends on what I end up mirroring. Again, I wouldn't for movies (17-18TB), but I might for TV shows (3TB), Pictures (200GB) and ISOs/Docs (maybe 200 GB). I realize there are things I can do (i.e. I have multiple copies of my pictures library), but it would be nice to manage it all within UnRAID. Again, I could build a 2nd server, but I would rather have a disk dedicated to backups to decrease the likelihood of data loss - it would also be nice to have this automated so I don't have to think about it.

 

I've lost multiple drives before, so tend to be a little more conscious of this possibility.

Setting up two shares that replicate your data [e.g. "Important Stuff 1" and "Important Stuff 2"], with the shares assigned to a different set of drives would do what you want here.  You could simply run a synchronization utility that kept them in sync at whatever interval you choose (daily, weekly, hourly, etc.).

 

But that is NOT a good backup strategy -- it still provides no protection against a catastrophic failure of that system.  Your backups should be on a different system, or perhaps an external drive.

 

With the sizes you indicated, you could back everything you've listed on one 4TB drive.                So I'd just buy an external drive, and use it to backup those folders you consider important.

 

You are right Gary. I know it's not a real backup solution. I am thinking of it as more a first line of defence. As mentioned I do keep external backups of my pictures, but if I look at TV Shows, it's something that is updated multiple times a day as SAB finds new episodes. Some I delete after watching, some I keep the entire series for. With how dynamic this content is I am thinking that something like your sync utility solution would help with. When I ran Windows Media Center I had all my TV shows on the same internal disk, and it died. Trying to figure out all the shows I had saved, and/or watched, and what episodes I needed to get back was a pain.

 

Do you have any suggestions on a sync utility that would help?

 

I like this idea as I am only investing in a single 4TB disk for my system, rather than building a new UnRAID solution fro backup or having to manually update to an external drive.

 

Coincidentally i have been thinking about this. There are quite a few scenarios where you would want the speed of the cache disk but some reliabilty in the form of of rolling mirrors to the array.

 

This would not a backup alternative but it would add a nice local extra level of redundancy.

 

 

Luckily its actually rather easy to do and would just need some fettling. rsync can do differential backups so  storing only the data that has changed. The clever bit is each "backup" looks like a complete backup.

 

There are countless examples of this on the net and one could be adapted to our purposes easily. This is a tried and tested method.

Glad I am not the only one who sees value here. Since I have no idea how to code it I was hoping someone else with better Linux knowledge would be able to jump in with suggestions.

 

I'd use SyncBack running on a Windows box with a synchronization profile scheduled to run once/day (perhaps 3 or 4 in the morning).

 

I'd use SyncBack running on a Windows box with a synchronization profile scheduled to run once/day (perhaps 3 or 4 in the morning).

 

+1

I'd use 'include'  or 'exclude' to define a share for 'Very Important Stuff', thus forcing the share onto known disks.

Then I'd define 'Very Important Backup' as the opposite 'include' or 'exclude'.

Then SyncBack, which I like...rsync would work, too.

If you have something that will work and you know it then I wouold use that.

 

If you dont know any of these tools you might as well go for the native solution of rsync I would say. Wont be hard.

I did suggested this a while back but got the same typical response of its not a backup alternative.  And use rsync.  Not disagreeing with either of those points but my request was a simple plug in to allow less educated users the ability to set up mirrored data,  and making it easier for everyone

 

Sent from my SM-N900W8 using Tapatalk

 

 

I think I will look into rsync. I don't really want to rely on a Windows machine. I'd rather have something just internal to UnRAID without any external dependencies.

 

Thanks for the suggestions.

 

Bnevets27 - I agree a plugin would have been great - however since it looks like we are moving away from plugins I think I will need to learn rsync and learn to automate this.

With the recent addition of perl to the default V6 install this becomes viable:

 

http://www.rsnapshot.org/

 

same premise just alot of the hard work done for you.

 

Just a sanity check. We all know this isnt a backup but it is still an increase in redundancy IF and ONLY if you do incremental and differerntial backups. If you dont then all that will happen is you will end up with 2 copies of the corrupted file or no copies at all of the deleted file

 

Once i get some time I can help you with this.

With the recent addition of perl to the default V6 install this becomes viable:

 

http://www.rsnapshot.org/

 

same premise just alot of the hard work done for you.

 

Just a sanity check. We all know this isnt a backup but it is still an increase in redundancy IF and ONLY if you do incremental and differerntial backups. If you dont then all that will happen is you will end up with 2 copies of the corrupted file or no copies at all of the deleted file

 

Once i get some time I can help you with this.

 

Thanks NAS. I appreciate the information and offer to help.

 

If I understand your solution, this is going to create a backup file (presumably a tar file or something?), which I will need to restore in the event of a data loss (assuming I can't just replace the disk and have UnRAID restore it).

 

Is this a correct understanding?

Actually no

 

So basically rsync is a tool to copy files just like you would copy any file. At the end you have two files the original and the copy in the new location.

 

Where rsync is clever is if you have already copied the file, then edit a bit of the original the next time you run it it will make the copy match the original by only copying the changed bit.

 

This makes it very clever and very efficicent (it is designed to do this over unrelaible slow links so done localy is blistering fast).

 

OK so lets get and example:

 

Original: /mnt/cache/bob

 

Copy /mnt/user/backups/bob

 

So on the first run all files in bob are copied to /backup/bob. Nothing fancy there.

 

Now the clever bit on run 2:

 

Original: /mnt/cache/bob

 

Copy /mnt/user/backups/bob

Copy /mnt/user/backups/bob.1

 

The names are not important but:

 

Copy /mnt/user/backups/bob would contain your first backup unchanged

Copy /mnt/user/backups/bob.1 would contain only the changed files since the first backup.

 

However because of some clever linux magic bob.1 would seem to have a complete backup since it also links to the unchanged files in /backups/bob

 

Long story short you have have backups and even if you dlete stuff as long as you havent expired your backup you wil still have copies of the deleted data

 

Thanks for the clarification NAS. That sounds really good (and better than I had been thinking). It definitely sounds like it will meet my requirements.

 

Why dont you just start with using rsync manually to create a single local redundant file set.

 

This will be very simple and will at least get you away from having nothing.

Can I point you to a user that has done this recently:

 

http://lime-technology.com/forum/index.php?topic=31876.0;topicseen

See this thread.

 

http://lime-technology.com/forum/index.php?topic=31876.msg290541#msg290541

 

Look at my google code page for a shell script called rsync_linked_backup.sh

It's a simple shell example of how to do it using date formatting for the output directory.

 

With the right format, you can have daily, weekly, hourly, monthly.

I did this rather then a rotating number.

What I do not have is an automated purge mechanism yet.

That's pending on some bash loadable tools I've yet to release.

 

It uses similiar concepts as rsnapshot and rlbackup, only I use date formatting for the output directory name (easier for me) and I do it based to run on the system storing the backups. i.e. pulling the backups.

 

The purpose for that is so the backup directory can be inspected, sorted and you can link to the last proper backup.

It was also meant to be as minimal as possible.

I've always thought rdiff-backup was one of the better tools.

http://www.nongnu.org/rdiff-backup/

 

What stopped me from using that was it's dependency on python.  Weebo don't do python. LOL.

 

Building on that was rBackup

http://rbackup.lescigales.org/

 

So you have the following.

http://www.rsnapshot.org/

http://www.math.ualberta.ca/imaging/rlbackup/

http://www.nongnu.org/rdiff-backup/

http://rbackup.lescigales.org/

 

and my simpler rsync_linked_backup.sh

 

and s'more information here.

http://www.mikerubel.org/computers/rsync_snapshots/

 

and yet..

http://www.noah.org/wiki/Rsync_backup

Thanks NAS/Weebo

 

I appreciate all the pointers to great info on this.

 

this is definitely my next UnRAID project. Once I get into it I may have some follow-up questions but you guys have given me a wealth of info to start from.

I would also appreciate this functionality.  It is one feature that I really miss from Windows Home Server.

 

I have ~900GB of photos that I'd like to have mirrored on separate drives on my server, in addition to an offsite backup.

Thanks WeeboTech, rsync_linked_backup.sh is great, I was going to go with rdiff-backup but your solution seems cleaner.

 

I currently have all my machines, phone and tablet syncing to unraid with bittorent sync and once a night run rsync_linked_backup.sh via cron to back everything up to an external HDD so everything is in 3 places, I might add an offsite backup at some point too.

 

I've wrapped it up in a script so I can keep the backup disk unmounted most of the time;

 

diskid="ata-WDC_WD5000AAKS-00TMA0_WD-WCAPW0198105-part1";

# check if disk already mounted
mountpoint /mnt/backup-to-$diskid > /dev/null 2>&1
if [ $? -eq 0 ] ; then
        echo "Already mounted, backup may already be in progress..."
else
        # check if disk exists
        if [ -b /dev/disk/by-id/$diskid ];
        then
                #create mount point
                mkdir -p /mnt/backup-to-$diskid

                # mount disk
                mount /dev/disk/by-id/$diskid /mnt/backup-to-$diskid

                # run backup
                /boot/backup/rsync_linked_backup.sh /boot/backup/UNRAID.conf

                # unmount disk
                umount /mnt/backup-to-$diskid
                wait

                # check its actually unmounted before removing mount point
                mountpoint /mnt/backup-to-$diskid > /dev/null 2>&1
                if [ $? -eq 0 ] ; then
                        echo "Drive mounted. Cannot remove mount point."
                else
                        rm -rf /mnt/backup-to-$diskid
                fi
        else
           echo "Backup drive not connected."
        fi
fi

 

All works very well.

 

The only problem I can see is that I might start running into trouble once the disk starts filling up, did I read on another of your posts that you have a script to prune the backup in order to only keep the last couple of months daily backups + one for each month + one for each year?