Backup to duplicate offsite UnRAID server.

[CHBMB]

Hi,

 

My UnRAID server has now progressed from being a novelty to an absolute necessity.  Should my house burn down it would be among the items I'd be rushing to save!

 

Hence I have decided I need an offsite backup.

 

My plan is to have a complete clone of my server.

 

For example each disk in the array is an identical image in each server.

 

That way even if I lose 2 disks in one array, and can't rebuild the data I only have to copy the files from my offsite backup.

All new data to be copied to the server will be contained on the cache disk so I can remove that and manually copy the files to my offsite backup.

 

Can anyone help with getting this going?  Ideally I'd like not to use windows as I've found copying large amounts of data through windows sometimes is a little unreliable.

 

Thanks

 

Neil

[WeeboTech]

This is very easy to do with rsync.

The first copy of diskx -> diskx.backupserver is going to take a really long time. So you may want to do local usb rsync's.

 

Once it is remote, you and do rsync's via the net and keep the two disks in sync daily.

 

I would suggest begin reading up on rsync.

Search the forum for subject matter on rsync. I have a few scripts that setup the rsync server in the go script.

 

Then you can play locally and once you are done. put the server remotely.

 

You will need to setup ssh and some form of tunneling in the routers. But that's subject matter for another thread.

 

 

 

[CHBMB]

I'm not planning on doing anything remotely.  Currently they are both attached to the same network hub in my house and that's how I plan to do the initial backup.  After that I will do it manually by taking my cache drive to the offsite backup server (Will be at my parents house) and manually copying the data across there.

 

Does that make things a little easier?

 

Thanks for the reply.

[WeeboTech]

There are plenty of ways of propagating additions to the remote server via some form of external drive.

 

I know rsync has the --remove-sent-files so that files can be moved from the cache drive.

But I do not know how you will deal with deletions.

 

a disk1 to disk1 rsync with rsync -av --delete will synchronize everything including files that should not belong.

However with deletions, that may be a bit tricky if you plan to do the synchronization via the extra drive.

 

 

 

[CHBMB]

I am sorry as I keep adding things to clarify and you're being very patient with me.  I won't be deleting any files, it's all archived stuff. 

 

The first task I want to accomplish is just to create a duplicate server with the same files on each disk.

 

After that I intend to copy new files to the cache drive and leave them there, then transport that drive to the offsite and copy them onto the backup server, then transport the drive back and move the files to my server. 

 

Once again, thank you so much for taking the trouble to help and painstakingly drag the necessary information from me!!

 

[WeeboTech]

I would not do this with the cache drive. Instead I would use an external USB drive.

Mount it on /mnt/backups when needed and umount disconnected it when done.

 

With the cache drive the array will have to be stopped. The cache drive will need to be taken out of the array then restarted.

The opposite on the other side.

 

With an external USB drive it can be mounted/unmounted on demand.

 

You can do an rsync -a /mnt/disk1 /mnt/usbdrive/disk1 on one machine

then do rsync --remove-sent-files -av /mnt/usbdrive/disk1 /mnt/disk1 on the other machine. which will empty each folder respectively.

 

The trick is knowing which files to back up.

In that case you may need to use find command like this.

 

find /mnt/disk1 -newer /mnt/cache/.backups/disk1.lastbackup -type f -print  > /mnt/cache/.backups/disk1.filelist

 

Then use rsync with the files-from=/mnt/cache/.backups/disk1.filelist

then touch /mnt/cache/.backups/disk1.lastbackup to set a new pointer.

[tritron]

I was thinking about using hard drives as tape drive backup type stuff. So if we  take 2TB that $60 so compressed we could get 4TB per each drive. Now trayless drive is like $10 to $15. So I was thinking on puting hd in and using one of linux tape backup programs to backup with compression. The data can be stored for ever that way. 

[WeeboTech]

I was thinking about using hard drives as tape drive backup type stuff. So if we  take 2TB that $60 so compressed we could get 4TB per each drive. Now trayless drive is like $10 to $15. So I was thinking on puting hd in and using one of linux tape backup programs to backup with compression. The data can be stored for ever that way. 

 

Here's an excellent way of doing that externally with USB or eSATA

 

http://www.newegg.com/Product/Product.aspx?Item=N82E16817707169

 

 

[mcs]

2TB compressed to 4TB

 

This compression depends on what you are backing up. This compression ratio is not likely with videos or photos, which are significantly compresses to begin with. It would be unlikely to save more than a few percent in space with video or photos. 

[CHBMB]

I would not do this with the cache drive. Instead I would use an external USB drive.

Mount it on /mnt/backups when needed and umount disconnected it when done.

 

With the cache drive the array will have to be stopped. The cache drive will need to be taken out of the array then restarted.

The opposite on the other side.

 

With an external USB drive it can be mounted/unmounted on demand.

 

You can do an rsync -a /mnt/disk1 /mnt/usbdrive/disk1 on one machine

then do rsync --remove-sent-files -av /mnt/usbdrive/disk1 /mnt/disk1 on the other machine. which will empty each folder respectively.

 

The trick is knowing which files to back up.

In that case you may need to use find command like this.

 

find /mnt/disk1 -newer /mnt/cache/.backups/disk1.lastbackup -type f -print  > /mnt/cache/.backups/disk1.filelist

 

Then use rsync with the files-from=/mnt/cache/.backups/disk1.filelist

then touch /mnt/cache/.backups/disk1.lastbackup to set a new pointer.

 

I think that sounds like a much better way to do things actually.  What I'd like to do is copy everything to the cache drive as per normal, and when it moves it to the array also make a copy and put it on the external USB.  Is that feasible?

[WeeboTech]

I would not do this with the cache drive. Instead I would use an external USB drive.

Mount it on /mnt/backups when needed and umount disconnected it when done.

 

With the cache drive the array will have to be stopped. The cache drive will need to be taken out of the array then restarted.

The opposite on the other side.

 

With an external USB drive it can be mounted/unmounted on demand.

 

You can do an rsync -a /mnt/disk1 /mnt/usbdrive/disk1 on one machine

then do rsync --remove-sent-files -av /mnt/usbdrive/disk1 /mnt/disk1 on the other machine. which will empty each folder respectively.

 

The trick is knowing which files to back up.

In that case you may need to use find command like this.

 

find /mnt/disk1 -newer /mnt/cache/.backups/disk1.lastbackup -type f -print  > /mnt/cache/.backups/disk1.filelist

 

Then use rsync with the files-from=/mnt/cache/.backups/disk1.filelist

then touch /mnt/cache/.backups/disk1.lastbackup to set a new pointer.

 

I think that sounds like a much better way to do things actually.  What I'd like to do is copy everything to the cache drive as per normal, and when it moves it to the array also make a copy and put it on the external USB.  Is that feasible?

 

Yes, you could adjust the mover script to rsync the same files in the same structure to the USB drive. then do the rsync / move to the array.

[dgaschk]

You can use SNAP to mount the USB drive. Search for SNAP in the User Customizations forum.

[teamhood]

Cool topic and if you search for rsync you will probably find a thread or two that I started and within the one that is titled 'rsync between 2 unraid server' you will see a step by step guide that I put together.

 

It's funny you mention this because I've been thinking about leasing a cabinet... So I work for a carrier and I can easily sell myself data center colocation at very minimum margins. The problem is that it's still expensive, but maybe we can all figure out something...

 

I would love nothing more than to have  replication of my home unRAID as I have footage from my hobby job that I would be crushed to loose. Sure I have some Bluray backup, but I would really love to host something in data center Yes, I'm a sales guy that loves technology and people look at me like I'm crazy that I want a cabinet, LOL

[CHBMB]

Cool topic and if you search for rsync you will probably find a thread or two that I started and within the one that is titled 'rsync between 2 unraid server' you will see a step by step guide that I put together.

 

It's funny you mention this because I've been thinking about leasing a cabinet... So I work for a carrier and I can easily sell myself data center colocation at very minimum margins. The problem is that it's still expensive, but maybe we can all figure out something...

 

I would love nothing more than to have  replication of my home unRAID as I have footage from my hobby job that I would be crushed to loose. Sure I have some Bluray backup, but I would really love to host something in data center Yes, I'm a sales guy that loves technology and people look at me like I'm crazy that I want a cabinet, LOL

 

I came to the realisation that it doesn't actually cost much for me to build an UnRAID server and that it was basically an insurance policy.  But with a one off fee and very low yearly fees as the backup will only be switched on when required.  I've lost count of the amount of important data I've lost over the years because I didn't protect it well enough so this was the idea I came up.  Currently cloning disk 3 of 5 

 

Will look into the Rsync method, thanks for letting me know about your post.