I have an 'online' unraid server that I'd like to sync with another 'offline/backup' one...

 

Basically my goal is to have the backup system come online perhaps every couple weeks, and perform a sync of any changes from the primary to the backup system.  I'm not doing anything that needs up to the minute backup, and also don't want to find out something gets corrupted and then replicates corrupted data before I realize it, wiping out my backup.  

 

My primary host which is online all the time is a Win 7 64 bit server and I'm dealing with around 14TB of data, but not a ton of changes.. Does anyone have any reccomendations as far as how I might best accomplish this or any good/bad experiences they've had doing a similar config?

 

Part of me is wondering if rsync is the best route, and keep the communication between the unraid servers and somehow do some manual power up/powerdown of the backup unraid box for the rsync process to take place... Another idea I had was to use software on my Win7 host to manage the replication so I could keep an eye on it and setup some email alerts and see some representation of what's going on, as I've never worked with rsync.  

 

Any tips from folks that might have gone down this path before?

 

Pete

Depends on how handy you are with code. The latest version of SMB (not sure about Samba 3) has the ability to output a system event ON_FILE_CHANGE/COPY (or something to that effect). You could take advantage of that to get the file location and sync across to the other server in close to real time. That was designed to somewhat replicate Microsofts distributed filesystem.

 

**  Just read your main server is Windows **

 

Or, you could just do syncs across the units. That would be a much easier way if you don't rely on the changes instantly. It would be relatively easy to have the main server wake the backup server at a specified time and initiate a sync. Just have the backup server on UPS, and asleep when you don't need it. You may as well sync more often than two weeks, especially if you're not needing to access the server at night.

 

Automation and backup, with periodic testing is one of the most important things you should do. Never rely on yourself to make backups  

Thanks for the info - are you referring to sync by using rsync or an addon of somesort?

 

Pete

Whatever you feel comfortable using. If you're initiating a sync on a Windows machine, you're probably better off using a tool you're familiar with. I think they do Rsync for Windows, but I'm not sure of its maturity.

My only issue with the windows solution is that while comfortable, i can't believe it's going to be real quick watching the ping pong of data between two remote servers over samba...

 

This is what I'm looking at trying with rsync...wasn't real clear on what crontab needed.  I was hoping to just get this thing to automate on sun/mon nights, log the rsync progress, and eventually put something in there to shut down the destination after the backup is completed..

 

Again lets say I have two identical servers and want to copy everything from the 5 drives on SERVER1 over to the 5 on SERVER2.  Server1 has about 6 shares, Server2 has no shares setup...I'm trying to copy the whole thing over and don't really care where it lands across the 5 disks on the destination side...will this work?

 

1) copy rsyncd.conf file to SERVER2 (destination server) under flash/config/rsyncd.conf

 

contents of file:

 

uid             = root

gid             = root

use chroot      = no

max connections = 4

pid file        = /var/run/rsyncd.pid

timeout         = 600

log file        = /var/log/rsyncd.log

 

[mnt]

   path = /mnt

   comment = /mnt files

   read only = FALSE

 

 

2) Put code in the end of my go script on SERVER2 (destination server) so it holds after reboot

 

rsync  --daemon  --config=/boot/config/rsyncd.conf

 

 

3) On the source (SERVER1), use commands like this to write to the destination (SERVER2)

 

Manually:

cd /mnt

rsync  -avrtH  user/  server2::mnt/

 

(archive mode, verbose, recursive, preserve timestamp, preserve hard links)

 

or..setup cron job by putting the following code in the go file:

 

crontab -l >/tmp/crontab

echo "# Run a Sunday and Monday 1am Backup" >>/tmp/crontab

echo 0 1 * * 0-1 rsync -avrtH user/  server2::mnt/>>/tmp/crontab

crontab /tmp/crontab

 

 

Hope this is getting closer... just trying to get a replica of the data and chip away at automating this more without having the secondary box online all the time....

Pete

Agreed, SMB is not really designed/optimised for that sort of punishment. If you're comfortable with the command line then Rsynd is extremely easy to deal with (much more so than tunneling through SSH), and handy as it's built into unRAID. I'd use a single user share/mount, as unRAID is already proven to split the files between drives properly. This will be an issue if you want to compress, as you'll have issues with files larger than the largest hard drive available to write to. Again, I'm not too sure of the maturity of Windows Rsync. Basic testing will help here.

 

You'll want to sleep the Backup server after backup completion. This makes it easier to use Windows task manager/Cron clone to initiate backups without human interaction. You could use telnet to tunnel through the sleep commands after successful backup, WOL for wake and initiation. If you prefer shutting down, don't forget the motherboard features should allow startup scheduling etc. That should be configurable within unRAID, or in the BIOS.

 

Good luck with your testing 

Just curious if anyone knew if my syntax was right in the example from my last post?

 

I ran it as a test and my destination (server2) is completely unresponsive now..can ping it but can't telnet into the box, bring up the web interface, browse any shares, etc...

 

Pete

Unfortunately I didn't read all of your syntax. Get physical access to the machine and push the syslog to /boot. Hopefully that'll provide some insight to why it crashed.

 

For testing, try something simple first, and work your way up from there. Try the same syntax from a linux VM. It should "just work" with two Linux boxen, and fail nicely if there's an issue with syntax. I've not much experience with rsyncd, but I've used rsync over SSH, and it has never crashed on me.

 

Happy testing 

When it locks up, it really locks up... the console doesn't work either...the only way to get the box back is to reboot (i commented out the rsyncd command in the go menu on the destination side...otherwise after the reboot it will lock up once the rsync kicks off again).  I don't see drive activity when it locks up either, so I know it's not just 'really busy' heh...  There's no log available and no sign that files got copied over...that seems to get wiped out after the reboot

 

On the source side, I see a list of files that need to be copied scrolling on the screen, but then it stops...

 

It's strange..but when I first send the rsync command from the source, i do see the rsync log appear in /var/logs and it looks like it's on the right track.  Is there perhaps something I need to do outside the steps above?  Do i have to setup a mount or setup the user shares to mimic the source server, or anything along those lines?

 

i'll poke around with it some more...really surprised there's no howto on this yet...there's a lot of 'how do I sync my unraid with third party services, over ssh for individual disks/files, or between windows hosts and third party apps, but I haven't run across a thorough 'this will work' config for Unraid->Unraid for the entire storage device... and whatever i'm interpreting from these similar configs out there, isn't exactly correct...

 

Pete

Try running it from a linux VM, after googling multiple tutorials/manuals on how to initiate a backup via rsyncd. There should be a verbose option to call as well, so run with that. It might be that the windows version of Rsync doesn't support what you're doing?

 

The version of Rsync in unRAID isn't that old, so I presume it should all work as planned. You should be able to copy to user shares in /mnt just like you copy to SMB shares.

Wow IE8 doesn't like these forum editing pages very well, outside of compatibility mode..

 

Anyway - I made the adjust in the line in bold... got things running, but now it fills up disk1 and doesn't move to the other free disks.  I have all my user shares set to level 1, fillup, 20gb min free.  Not sure quite what that means yet, if I should be doing level 0(don't think so), perhaps adjusting it to high water?... A part of me wonders if the share settings are even applicable when i'm copying the entire /user contents over...anyone know?

 

Here's the process i'm following, latest change in bold...

 

1) copy rsyncd.conf file to SERVER2 (destination server) under flash/config/rsyncd.conf

 

contents of file:

 

uid            = root

gid            = root

use chroot      = no

max connections = 4

pid file        = /var/run/rsyncd.pid

timeout        = 600

log file        = /var/log/rsyncd.log

 

[mnt]

    path = /mnt

    comment = /mnt files

    read only = FALSE

 

 

2) Put code in the end of my go script on SERVER2 (destination server) so it holds after reboot

 

rsync  --daemon  --config=/boot/config/rsyncd.conf

 

 

3) On the source (SERVER1), use commands like this to write to the destination (SERVER2)

 

Manually:

cd /mnt

rsync  -avrtH  user/  server2::mnt/user

(archive mode, verbose, recursive, preserve timestamp, preserve hard links)

 

or..setup cron job by putting the following code in the go file:

 

crontab -l >/tmp/crontab

echo "# Run a Sunday and Monday 1am Backup" >>/tmp/crontab

echo 0 1 * * 0-1 rsync -avrtH user/  server2::mnt/user>>/tmp/crontab

crontab /tmp/crontab

 

 

I suggest setting up folders on each disk called "Backup", then create that share in unRAID called "Backup". Then point all the destinations to /mnt/user/Backup .

 

I believe unRAID does the split based on the Folders inside each disk. So /mnt/user/Backup will move to /mnt/disk/dev1/Backup - /mnt/disk/dev2/Backup etc.

Do you know why I would need to create them manually?  Wouldn't it just work like a normal SMB share would..it fulls up one disk, then moves onto the next?  It works when it's a windows host copying to the smb share..is this different since it's rsync...perhaps does it need an export setup instead?

Yeah it doesn't seem to really care about my share settings. I put a folder on disk2 and set the share to use disk2,disk1 and as soon as I start the rsync it keeps filling up disk1 even though it has 3gb free and disk2 has 2tb free.  It's bypassing the whole 'share ruleset' so to speak...

 

I wonder if server1 is copying the share folder over server2's same foldername and somehow losing the config?  hmm..

 

Pete

Did you ever figure this out?

 

Neil

 

Yeah it doesn't seem to really care about my share settings. I put a folder on disk2 and set the share to use disk2,disk1 and as soon as I start the rsync it keeps filling up disk1 even though it has 3gb free and disk2 has 2tb free.  It's bypassing the whole 'share ruleset' so to speak...

 

I wonder if server1 is copying the share folder over server2's same foldername and somehow losing the config?  hmm..

 

Pete

 

Change he path to /mnt/user/share then share ruleset will be used.

 

 

I'd recommend mirroring the disks, i.e, server1/disk1 = server2/disk1. This will make restoration of a subset of failed disks much easier and you don't need to worry about user shares.

Yeah, it was an issue of the split levels being incorrecttly set to 1...

 

The reason I didn't want to sync disk to disk was that my disk sizes may change... or i may want to sync just one of the shares sometime, not go through the whole thing...

 

Pete