Multi unRAID synching

[vexhold]

Was wondering if it is possible to have two unRAID servers in two seperate locations in the same town synched together. There will be different users at both location who will be adding/deleting/changing files on both systems at the same time. But like once every 2 weeks or so I would like them to sync and add/remove/change what the other one has done making them basically back each other up.

 

Something similar to AD replication in a server environment or offline files or exchange. That kind of thing.

[vexhold]

Maybe I can describe my question better.

 

I would like an unRAID server at my home and one at my mothers house.

 

I would be using it with SageTV to store all my Media Files.

 

My mother would be using it for her photography business to store her files in a central location.

 

Currently, I am just using a couple 1TB HD's in a XP machine and she is using the same in a Windows 7 machine.

 

I am always scared of having to reload all my media from scratch and I have had to "Miraculously revive her files from crashed OS's several times in the last 2 years"

 

So my question is...

 

 

If she has an unRAID server at her location, adding/deleting/changing her files and I have an unRAID server at my location adding/deleting/changing media files, can we cross back-up each of our files on the others server across town using unRAID?

 

[BRiT]

Some users utilize 'rsync' to sync between servers. I do not use it, but I think in most situations, one server is setup as the base and the other servers become an exact mirror of that one. In your case, I think you could do it on a share-by-share basis, with your server being the base for 'Vexhold Files', and her system as the base for 'Business Files'.

 

No matter the solution you try out, you want to be careful with blindly synchronizing, to prevent from mirroring mass accidental deletions. I think you will want to use an incremental backup plan for your mother's business files.

[Joe L.]

Maybe I can describe my question better.

 

I would like an unRAID server at my home and one at my mothers house.

 

I would be using it with SageTV to store all my Media Files.

 

My mother would be using it for her photography business to store her files in a central location.

 

Currently, I am just using a couple 1TB HD's in a XP machine and she is using the same in a Windows 7 machine.

 

I am always scared of having to reload all my media from scratch and I have had to "Miraculously revive her files from crashed OS's several times in the last 2 years"

 

So my question is...

 

 

If she has an unRAID server at her location, adding/deleting/changing her files and I have an unRAID server at my location adding/deleting/changing media files, can we cross back-up each of our files on the others server across town using unRAID?

 

Yes, you can use rsync.  It will do incremental backups as needed, but you also need to deal with a VPN and security.. It is NOT enough to just open up the ports on your firewall/router.

[vexhold]

Thanks guys. That answers my question. And Joe, I understand fully about the VPN and Security. We will both be upgrading to business class comcast to be able to tunnel.

 

Thanks again.

[Joe L.]

Thanks guys. That answers my question. And Joe, I understand fully about the VPN and Security. We will both be upgrading to business class comcast to be able to tunnel.

 

Thanks again.

I sincerely doubt you need business class comcast to be able to tunnel using a VPN.  Others on the forum might be able to guide you in what you really need.  (Of course, Comcast will be happy to charge you more for the business class service) 

 

Joe L.

[WeeboTech]

unRAID can do the back and forth rsync as described.

I would probably suggest dedicating spindles or shares for each location.

 

You can use VPN or you an use rsync over ssh.

In doing rsync over ssh, it's prudent to use IP address control in the routers and/or with tcpwrappers so that the ssh port is not visible to anyone on the internet.

Otherwise you will have many automated bots knocking on your door trying to get in.

[neilt0]

I tried to get my head around rsync, but could not find any tutorials on how it works.

 

I'd love a couple of command line examples of syncing a folder between two servers if someone could help with that?

[mcs]

I tried to get my head around rsync, but could not find any tutorials on how it works.

 

I'd love a couple of command line examples of syncing a folder between two servers if someone could help with that?

 

A good set of examples can be found the rsync example site...

http://www.samba.org/rsync/examples.html

[Joe L.]

Locally, this is what I do... Note: it is over my LAN and would not be secure over the internet

 

I created a rsyncd.conf file that looks like this:

cat /boot/config/rsyncd.conf

uid            = root

gid            = root

use chroot      = no

max connections = 4

pid file        = /var/run/rsyncd.pid

timeout        = 600

log file        = /var/log/rsyncd.log

 

[mnt]

    path = /mnt

    comment = /mnt files

    read only = FALSE

 

I then invoked the rsync daemon process with this command:

rsync --daemon --config=/boot/config/rsyncd.conf

 

The /mnt path gives me the ability to write to any user-share, you might want to pick a specific share for your situation.

 

Then, on the other unRAID server, you can do something like this:

cd /mnt

rsync -avrH user/Pictures tower2::mnt/user/

 

It will sync all the files under the "Pictures" share on tower1 to the second server tower2.  It will not remove files on tower2 that were removed on tower1, but there is a "--delete" option to rsync for that if that is your desire.

 

For your situation, you would want to use rsync over ssh, or a VPN.

 

Joe L.

[WeeboTech]

Keep in mind rsync running as a daemon is not secure, so doing that over the internet without some form of VPN would allow anyone to see the traffic.

 

rsync to a remote server over ssh would be secure, but it requires locking down the port to port communication with tcpwrapper or using the router's filtering. (and also requires installation of ssh).

[vexhold]

Thanks guys. That answers my question. And Joe, I understand fully about the VPN and Security. We will both be upgrading to business class comcast to be able to tunnel.

 

Thanks again.

I sincerely doubt you need business class comcast to be able to tunnel using a VPN.  Others on the forum might be able to guide you in what you really need.  (Of course, Comcast will be happy to charge you more for the business class service) 

 

Joe L.

 

Yeah, but when asked, Comcast replied that the DO NOT allow VPN's on a home account as it adversly effects the other users on the network, Comcast also has a data cap on home accounts that would easily be passed with VPN traffic.

[neilt0]

Thanks for your example, Joe -- that's pretty easy to follow.

 

I had seen the samples on the rsync site, but had no idea what was going on there.

I'm guessing you could throw in a wrong command and rsync can delete data. Which would be bad. So, having a real example with unRAID mounts is helpful.

[WeeboTech]

rsync never deletes anything by default.

It can overwrite a file, but it will not remove a destination file unless you use the --delete option.

 

In that case, rsync will delete all files on the destination that do not exist on the source. Thus mirroring the source to the destination exactly.

 

If you want to "move" files from one location to another.

Then use --remove-sent-files (which removes them from the SOURCE).

 

FWIW, rsync has a cool feature called --link-dest.

using it correctly, allows rsync to keep multiple archive directories at the destination, but only copy over the newer or changed files.  Sort of an incremental backup.

 

here's more information

http://www.mikerubel.org/computers/rsync_snapshots/

 

I have example shells which allow you to keep dated copies of one directory to another directory via rsync.

If anyone is interested let me know. I'll post the updated scripts.

 

It uses 1x the size of the source directory plus size of any changes for as long as you keep them online.

It's good for picture and document directories.

If nothing changes, very little space is added(just directory housekeeping).

As files change, the repository grows by the difference of each change once.

[ajeffco]

I have example shells which allow you to keep dated copies of one directory to another directory via rsync.

If anyone is interested let me know. I'll post the updated scripts.

 

yes please

 

I'm interested in doing the very same thing, but it will be from my office (literally a few feet from a hardened datacenter), pulling from my home unraid.  I'm familiar with rsync (I'm an AIX and SAN admin), but always welcome examples.  This will be the first time doing it over ssh.  Can you run it via pre-shared keys I wonder.

[WeeboTech]

This will be the first time doing it over ssh.  Can you run it via pre-shared keys I wonder.

 

rsync runs at a different level then ssh.

If you can connect the ssh channel , then you can set up keys.

The only issue is the account used and where the keys live relative to the home directory.

Remember that unRAID runs from a ramdisk.  So if the home directory is in the root tree somewhere, you have to restore them from flash when you reboot.

 

See my rsync_linked_backup and cached_home.conf file.

http://code.google.com/p/unraid-weebotech/downloads/list

 

The conf looks like this

BACKUPROOT=/mnt/disk2/backups

BACKUPDEST=cached_home

BACKUPSRC[0]="/mnt/cache/.home"

 

The directory looks like this:

 

root@atlas /mnt/disk2/backups #ls -l /mnt/disk2/backups/cached_home

total 0

drwxr-sr-x 3 root 500  72 Jul 11 21:51 20100711/

drwxr-sr-x 3 root 500  72 Jul 18 10:17 20100718/

drwxr-sr-x 2 root 500 136 Jul 18 10:39 log/

 

 

or

 

root@atlas /mnt/disk2/backups #ls -l /mnt/disk2/backups/underlord

total 4

drwxr-xr-x 28 root root 800 May 11 20:46 20100626/

drwxr-xr-x 28 root root 800 May 11 20:46 20100628/

drwxr-xr-x 28 root root 800 Jan 26  2010 20100704/

drwxr-xr-x 28 root root 800 Jan 26  2010 20100705/

drwxr-xr-x 34 root root 960 Jul 10 08:09 20100710/

drwxr-xr-x 35 root root 984 Jul 11 08:26 20100711/

drwxr-sr-x  2 root  500 384 Jul 11 21:07 log/

 

 

 

because backupsrc is an array you can backup more then one directory to BACKUPDEST

 

I.E.

 

BACKUPSRC[0]="/mnt/cache/.home"

BACKUPSRC[1]="/mnt/cache/.local"

BACKUPSRC[2]="/mnt/cache/.boot"

 

 

Note, there is no purge logic here.

That need to be configured by what ever retention calculations you deem necessary.