I just finished building a second unraid server, to work as a backup of my existing one. I'd like to keep the content between the two server synchronized. I think that rsync is the way to go, but my unix knowledge is very limited. I need help in implementing it.

 

As a test, I opened a PuTTY session into my 1st server (fs1), and typed:

 

rsync -av /mnt/disk1/ fs2:/mnt/disk1

 

This seems to be working, though the copy (~650GB) is still in progress. I have a few questions:

 

- is rsync the right tool in this situation, and is that the right syntax?

- how do I handle something like the "video" share, which spans several disks?

- after the initial copy, do I need to change any switches in rsync

- how can I schedule rsync to run periodically?

- I started the array with the parity drive off-line, thinking that I'd keep it off-line until the copy was completed. Dumb idea?

I just finished building a second unraid server, to work as a backup of my existing one. I'd like to keep the content between the two server synchronized. I think that rsync is the way to go, but my unix knowledge is very limited. I need help in implementing it.

 

As a test, I opened a PuTTY session into my 1st server (fs1), and typed:

 

rsync -av /mnt/disk1/ fs2:/mnt/disk1

 

This seems to be working, though the copy (~650GB) is still in progress. I have a few questions:

 

- is rsync the right tool in this situation, and is that the right syntax?

- how do I handle something like the "video" share, which spans several disks?

- after the initial copy, do I need to change any switches in rsync

- how can I schedule rsync to run periodically?

- I started the array with the parity drive off-line, thinking that I'd keep it off-line until the copy was completed. Dumb idea?

 

- is rsync the right tool in this situation, and is that the right syntax?

Yes, and yes, that is the syntax I use.

 

- how do I handle something like the "video" share, which spans several disks?

I rsync disk shares. I keep my disks the same size across shares when possible.

 

- after the initial copy, do I need to change any switches in rsync

If you are going to run this via cron, then be sure to redirect the rsync command to a log so you can inspect it for errors.

 

rsync -av /mnt/disk1/ fs2:/mnt/disk1 > /mnt/disk1/logs/rsync_disk1.log 2>&1

 

- how can I schedule rsync to run periodically?

Put a shell script in /etc/cron.daily which calls the rsync commands you want to run.

redirect the logs as above.

 

- I started the array with the parity drive off-line, thinking that I'd keep it off-line until the copy was completed. Dumb idea?

allot of people do this during an initial load to speed up the load time.

I think with today's improvements this is not that necessary, but if it's faster, then it is a good idea.

 

 

Thanks sharing this WeeboTech.  So, after the initial copy is done, rsync -av is still the right command to run? Meaning is it going to handle changed/added files only, but not touch files that are already there, right?

 

Thanks sharing this WeeboTech.  So, after the initial copy is done, rsync -av is still the right command to run? Meaning is it going to handle changed/added files only, but not touch files that are already there, right?

 

 

that is correct, if you run it in cron you may or may not want to use the -v parameter.

rsync will only transfer changed or new files.

if you want to delete files that have been deleted, then use the --delete option.

 

If you are willing to have your backup server pull them, then you can have dated/archive directories.

 

see my rsync script here.

http://code.google.com/p/unraid-weebotech/downloads/list

 

It is designed to be run on the backup server and pull the files via rsync either with rsync in daemon mode or over ssh. (that's dependent on how you set it up).

 

In any case it is an advanced example on how to use rsync.

I added the log redirection as you suggested:

 

rsync -av /mnt/disk1/ fs2:/mnt/disk1 > /mnt/disk1/logs/rsync_disk1.log 2>&1

This line seems to run just fine from the console, and this is what gets written in the log file:

 

sending incremental file list

logs/rsync_disk1.log

 

sent 3065878 bytes  received 6635 bytes  361472.12 bytes/sec

total size is 527801062192  speedup is 171781.56

 

If I put the same line into a script and run it, I get this:

 

root@FS1:/boot/custom/bin# ./sync

: ambiguous redirect

 

Any idea?

I added the log redirection as you suggested:

 

rsync -av /mnt/disk1/ fs2:/mnt/disk1 > /mnt/disk1/logs/rsync_disk1.log 2>&1

This line seems to run just fine from the console, and this is what gets written in the log file:

 

sending incremental file list

logs/rsync_disk1.log

 

sent 3065878 bytes  received 6635 bytes  361472.12 bytes/sec

total size is 527801062192  speedup is 171781.56

 

If I put the same line into a script and run it, I get this:

 

root@FS1:/boot/custom/bin# ./sync

: ambiguous redirect

 

Any idea?

It might be the space between the ">" and the log file name.  If that is not it, you might try to re-direct stderr to stdout first before re-directing stdout to the log file.

 

rsync -av /mnt/disk1/ fs2:/mnt/disk1 2>&1 >/mnt/disk1/logs/rsync_disk1.log

 

 

Joe L.

It might be the space between the ">" and the log file name. 

 

Removing the space between the ">" and the log file name still gives the "ambiguous redirect" error.

 

If that is not it, you might try to re-direct stderr to stdout first before re-directing stdout to the log file.

 

rsync -av /mnt/disk1/ fs2:/mnt/disk1 2>&1 >/mnt/disk1/logs/rsync_disk1.log

 

This syntax produces a different error:

 

root@FS1:/boot/custom/bin# ./sync
Unexpected remote arg: fs2:/mnt/disk1
rsync error: syntax or usage error (code 1) at main.c(1202) [sender=3.0.4]
root@FS1:/boot/custom/bin#

 

I was RTFM, and noticed that rsync has a --log-file=FILE  option, should I try that instead?

 

remove the "2>&1 >/mnt/disk1/logs/rsync_disk1.log "  for now..

 

make sure your script has

 

#!/bin/bash

 

as the first line.

 

I would suggest renaming the script to rsync_disk1.sh or rsync_disk1_to_fs2.sh

Make sure the script is executable. (i.e.  chmod 755 rsync_disk1.sh )

 

naming it sync is the same as the /bin/sync command.

naming it rsync could lead to confusion. Although I usually use rsync.sh.

 

Once the script is working without the redirects we can add lines to redirect.

We can also add options for the --log-file.

 

Get it working as simple as possible first.

remove the "2>&1 >/mnt/disk1/logs/rsync_disk1.log "  for now..

 

make sure your script has

 

#!/bin/bash

 

as the first line.

 

I would suggest renaming the script to rsync_disk1.sh or rsync_disk1_to_fs2.sh

Make sure the script is executable. (i.e.   chmod 755 rsync_disk1.sh )

 

naming it sync is the same as the /bin/sync command.

naming it rsync could lead to confusion. Although I usually use rsync.sh.

 

Once the script is working without the redirects we can add lines to redirect.

We can also add options for the --log-file.

 

Get it working as simple as possible first.

 

Thanks Joe L. and WeeboTech, the script is working now, including logging.