Differential backup?

[tucansam]

I now have a large number of old, small hard disks left over from dumping all of my data onto my new unraid server.  I have enough spare parts laying around to build a second unraid server for testing and, more importantly, for backing up my critical data.  I have about 1.1TB of data that I can't lose.  I regularly copy this data to an external hard disk, but I'm looking to bring the second server online to mirror the critical data on the first server.

 

I have a gig network under my desk and can dump data at gig speeds for the initial mirroring, but after that, the second server is going to get plugged into a ethernet-over-AC adapter (very slow) and, ultimately, be on a VPN at my folks' house via two cable modem connections.  So, after the initial mirror, copying data to the second server is going to be slow.

 

Hence, I'd like to find a way to have unraid do differential backups, so only the new/changed data gets copied over to the second server.

 

I don't need to automate this, necessarily.  I can set a day (once a week, twice a month, whatever) to do it by hand (via a shell script I would hope, at the minimum) but, later on, automating it would be awesome.

 

I know rsync exists but I've never used it and haven't started reading up on it.

 

Any suggestions?

[Joe L.]

I now have a large number of old, small hard disks left over from dumping all of my data onto my new unraid server.  I have enough spare parts laying around to build a second unraid server for testing and, more importantly, for backing up my critical data.  I have about 1.1TB of data that I can't lose.  I regularly copy this data to an external hard disk, but I'm looking to bring the second server online to mirror the critical data on the first server.

 

I have a gig network under my desk and can dump data at gig speeds for the initial mirroring, but after that, the second server is going to get plugged into a ethernet-over-AC adapter (very slow) and, ultimately, be on a VPN at my folks' house via two cable modem connections.  So, after the initial mirror, copying data to the second server is going to be slow.

 

Hence, I'd like to find a way to have unraid do differential backups, so only the new/changed data gets copied over to the second server.

 

I don't need to automate this, necessarily.  I can set a day (once a week, twice a month, whatever) to do it by hand (via a shell script I would hope, at the minimum) but, later on, automating it would be awesome.

 

I know rsync exists but I've never used it and haven't started reading up on it.

 

Any suggestions?

yes,

rsync

keep reading.

[BLKMGK]

I too would like to learn more about how to use rsync. One thing that occurs to me - if the source data becomes damaged then copying it will also toast the backups. How best to ensure integrity prior to backup?

[Joe L.]

I use this set of commands on my first server to back up selected user-shares to my second server ( named "tower2" )

cd /mnt

rsync -rlpgoDvrH --timeout=120 --progress --size-only user/Pictures tower2::mnt/user/

rsync -rlpgoDvrH --timeout=120 --progress --size-only user/Movies tower2::mnt/user/

rsync -rlpgoDvrH --timeout=120 --progress --size-only user/Mp3 tower2::mnt/user/

rsync -rlpgoDvrH --timeout=120 --progress --size-only user/TV tower2::mnt/user/

rsync -rlpgoDvrH --timeout=120 --progress --size-only user/data tower2::mnt/user/

 

 

Before running them, on the second server, I had created a configuration file

/boot/config/rsyncd.conf

looking like this:

uid            = nobody

gid            = users

use chroot      = no

max connections = 4

pid file        = /var/run/rsyncd.pid

timeout        = 600

log file        = /var/log/rsyncd.log

 

[mnt]

    path = /mnt

    comment = /mnt files

    read only = FALSE

 

And then started the rsync daemon on the second server running by invoking

rsync --daemon --config=/boot/config/rsyncd.conf

 

Note.  This WILL overwrite the file on the second server with a copy of what is on the first, so if the file on the first server file is corrupted somehow, the second will be as well, once synced.

 

The only protection I know for priceless data from that is dated copies (at the expense of space), or copies to read-only media (writable DVDs)  I know of no way to check and verify individual files for integrity prior to a sync to a backup server.

 

Joe L.

[BLKMGK]

How about incremental backups? I have done this with Acronis. You make a full backup and then additional backups save off what is essentially diffs. You then have a choice of versions to goto depending upon number of backups. Acronis will also collapse diffs into the master when you pass a certain number of them etc.

 

Probably asking for an awful lot but when it worked this was great for backing a desktop. Depending in what you have on your NAS, in my case mostly static, you probably don't need something so complex.

 

Appreciate you posting your configuration! Examples are much easier to learn from :-)

[hilly]

It took me some time to work out but i now use crashplan to do what you want to do. I have 2 servers tower: and backup: and a windows pc. With crashplan installed i copy my tower: shares with the data i need to keep onto my backup server which has a "crashplan backups" share. As a double security i also take the real critical stuff and send it off to crashplan central - the first load takes a very long time but as the server is on 24*7 thats not really an issue for me. You need to think it through as only one can be a kind of maser for backup to crashplan central but from one share to another within your network it all works easy

 

The good thing is once you are set up and running it all happens in the background and you get a status mail of what has been happening on a weekly basis and messages if backup destinations are not available etc

 

You can then recover a specific file or a whole directory back to a given date etc based upon your settings within crashplan. 

[tucansam]

I appreciate the reply, but with all do respect, I will never store my data on a commercial server, where I do not have complete control over it.  Encrypted or not, I don't trust anyone.

 

I'll keep looking into rsync.

[TheDragon]

Perhaps I've misunderstood how Crashplan works - but isn't storing it on their server optional?

 

I thought you could just use their software to only backup to a server within your own LAN, in other words so that you maintain complete control over your data.

 

Maybe someone who uses it can confirm one way or the other.

[WeeboTech]

I use rsync in linked-backup mode. Very similiar to rlbackup and some of the other tools out there.

 

 

The only issue is the backup server needs to "pull" the data from the source server.

 

 

The way it works is the backup server will run a job to rsync the remote server's tree to a local directory.

The shell job I have, looks at the dated directories and finds the most current directory.

This directory is passed to the rsync command with. --link-dest=(directory of most current backup)

 

 

This instructs rsync to link every file/directory from the most current backup directory to the new directory before the actual rsync.

After the links are done. a regular rsync (network or local) is executed to the new directory.

only the newly changed files are actually rsync'ed.

 

 

What I end up with is a directory tree of dated directories, each directory encompassing a full backup with ALL duplicate files being hardlinked to one version of the file.  When I need to prune for space I choose which directories to remove with rm -r YYYYMMDD and go about my merry way.  if a file has changed in some directory I can do a find and see all versions of a file.

 

 

This was the simplest way to do a full backup with daily incrementals of only the changes without tools other then rsync.

YMMV. it works for me. Only caveat is my unRAID server needs to pull files from all my other servers (Which is ok for me).

 

This could also work locally to a USB drive if it were a unix filesystem and writing locally.

 

 

Another plus with this is as long as your directories are named and sort in the order you want, you can do hourly or weekly backups also.

 

 

One of my directory trees is YYYYMMDD Another is YYYYMMDDHH Another is YYYY-WW (where WW is the number of the week).

 

 

Example: 

root@atlas /mnt/disk2/backups/hercules.vmware #ls -l
total 0
drwxr-xr-x 3 root root  72 Aug 20  2011 2011-33/
drwxr-xr-x 3 root root  72 Nov 17  2011 2011-46/
drwxr-xr-x 3 root root  72 Nov 30  2011 2011-47/
drwxr-xr-x 3 root root  72 Jan 22  2012 2012-03/
drwxr-xr-x 3 root root  72 Sep 15 18:48 2012-37/
drwxr-xr-x 2 root root 440 Sep 28 09:05 log/

 

 

Other reading

http://www.mikerubel.org/computers/rsync_snapshots/

http://www.rsnapshot.org/

http://www.nongnu.org/rdiff-backup/

http://www.saltycrane.com/blog/2008/02/backup-on-linux-rsnapshot-vs-rdiff/

http://evasive.ru/rsync-backup.html

 

 

I probably would have used rdiffbackup, however it's written in python and I did not want to add that layer of complexity.

 

 

Here's a link to my example shell.

http://code.google.com/p/unraid-weebotech/downloads/detail?name=rsync_linked_backup

[hilly]

As was said by jack0w in crashplan the sending to the cloud bit is optional. I have my pc doing a backup to another disk in the same pc (c:\ to d:\) and i have a set of folders on my main server doing a backup to my backup server. You can keep everything local if thats how you want it. The retention times, frequency of backup etc are all user adjustable. I guess rsync is more powerful but for those of us who are not command line jedi's once crashplan is set up its easy to use and has a nice user interface. You could even set up a round robin where every machine backs up to the next - or all of the others. You can also set up a backup to an external disk and when you plug it back in the backup will start regardless of how long its been out - i have one of these in 1TB form which is in a safe in someone elses house - cant be too careful

 

Its just an option and for home use its free, maybe not an ideal one for you but maybe for someone else looking at these posts as the backup topic seems to crop up now and again

[WeeboTech]

Crash plan is definitely a nice option for those who are not command line jedi's (gotta love it!)

[vca]

I use http://arcvback.com/arcvback.html to do this sort of thing.  Run it once to do a full backup and then later passes are incrementals.  You can run as many incrementals as you want (I typically run about a years worth before starting a whole new backup set.

 

Currently it needs a slightly older version of Python.

 

Stephen

 

[hilly]

WeeboTech - glad you approve of the sentiment :-)

 

Since starting with UnRaid i have learnt a lot and so much of that has come from people like yourself in these forums. I can do some of my required activities by command line but as you get older learning from scratch is harder, and i am from the point and click on the pretty picture generation i fear

I guess the trick is to work out clearly what you really need and then find the best solution for you to deliver it

[WeeboTech]

WeeboTech - glad you approve of the sentiment :-)

 

Since starting with UnRaid i have learnt a lot and so much of that has come from people like yourself in these forums. I can do some of my required activities by command line but as you get older learning from scratch is harder, and i am from the point and click on the pretty picture generation i fear

I guess the trick is to work out clearly what you really need and then find the best solution for you to deliver it

 

 

I started in CP/M, IBM PCDOS and MSDOS. IBM Mainframe S370 Assembler and DEC VMS... When I discovered UNIX. I knew I was /home !

[tr0910]

I use rsync in linked-backup mode. Very similiar to rlbackup and some of the other tools out there.

 

Here's a link to my example shell.

http://code.google.com/p/unraid-weebotech/downloads/detail?name=rsync_linked_backup

 

Is it possible to have the source and destination on the same server?  I have a personal document store on unRaid that I would like to have a rsnapshot of (or something similar)  and the backups could live on the same server on a different disk.

 

Was it that difficult to get rdiffbackup or rsnapshot working?  I see only one person has set it up.  http://lime-technology.com/forum/index.php?topic=4783.msg45651#msg45651

 

[WeeboTech]

I use rsync in linked-backup mode. Very similiar to rlbackup and some of the other tools out there.

 

Here's a link to my example shell.

http://code.google.com/p/unraid-weebotech/downloads/detail?name=rsync_linked_backup

 

Is it possible to have the source and destination on the same server?  I have a personal document store on unRaid that I would like to have a rsnapshot of (or something similar)  and the backups could live on the same server on a different disk.

 

Was it that difficult to get rdiffbackup or rsnapshot working?  I see only one person has set it up.  http://lime-technology.com/forum/index.php?topic=4783.msg45651#msg45651

 

 

I don't know, I never tried. I tried to keep my backup solution as simple as possible with rsync alone. It will allow you to backup one area to another and having a set of changes backed up. Download the script from my google code page and check it out.