I thought I share my rsync config. I still feel like Unraid as a Storage Server should provide a build in method to backup or sync its data to an other location, but configuring rsync gives you a lot of power, and once set up, it's just great.

I based my config and setup on the tutorial found here: https://lime-technology.com/forums/topic/52830-syncronize-servers-using-rsync-over-ssh-next-door-or-across-the-world/
And I made some adjustments, specially the option to watch the sync going on, and the notification that works with the build in unraid notification. So on "errors" you will get a E-Mail notification if your unraid is setup for that.

The above tutorial covers the SSH setup and connect via key instead of login, and the basic rsync setup.

You can continue after that guide. What you need is the NerdPack Plugin, and there install "screen". It is used to run rsync in it own screen, giving you the option to pull the progress up in a console at any time.

The second one is optional but recommended, its the "User Scripts" plugin. It allows you to add own scripts and execute it via cron in your unraid GUI.

 

I created 2 scripts for my rsync:

The first one I called "rsync" and does the rsync job, and depending on its exit code, it gives the notification to the Unraid Notification Service.

#!/bin/bash
rsync -avu --bwlimit=4000 --numeric-ids --log-file=/mnt/user/Backup/rsync-logs/log.`date '+%Y_%m_%d__%H_%M_%S'`.log --progress --exclude 'C Sys' --timeout=600 -e "ssh -p 4222 -i /root/.ssh/DSStoDMS-key  -T  -o Compression=no -x " /mnt/user/Backup/Devices/ [email protected]:/volume2/Backup/Unraid_rsync

if [ $? -eq 0 ]
then
/usr/local/emhttp/webGui/scripts/notify -s "`hostname` to DMS Rsync Backup complete" -d "Sync compled. `date` `tail /mnt/user/Backup/rsync-logs/last.log`"
else
/usr/local/emhttp/webGui/scripts/notify -s "`hostname` to DMS Rsync Backup FAILED" -i "alert" -d "Sync compled. `date` `tail /mnt/user/Backup/rsync-logs/last.log`"
fi

Of course, the rsync command would need some adjustments.
If your Unraid server is setup to send you a E-Mail on errors, it will do that too if rsync fails!


The second one is for calling that script in a "screen" session.
 

#!/bin/bash
export SCREENDIR=/root/.screen
screen -dmS rsyncdms /boot/config/plugins/user.scripts/scripts/rsync/script

You can pull it up as long as it is running with

screen -r

from a console. You can  just use the build in one from the WebGUI.
Attention, if you name the first script different than "rsync", you have to adjust this location.

 

Now just set a schedule for the second script. Watch its progress with "rsync -r"
 

I hope this can help somebody. Im not a rsync expert, so if anyone is around with more experience has some input, its very welcome.

And I have a questions myself:

I have large files to transfer, sometimes a lot of new data comes up at once. So if I have to transfer >400GB, it needs more than 24h, and the rsync backup script (which is scheduled to run daily) is started again. It starts with the files that is currently transfering in the first instance of rsync. It works, but it will transfer that big file twice. Should I check when running the script if rsync is already running? Or does running rsync with --partial do any good or bad here?

Edited July 25, 20187 yr by dedi

Hi

Thanks for your scripts, I think I'm gonna use them

 

Maybe this part of my lftp script can help your for your large file transfer. I check if a file .lock exists in temp folder. If not, the file is created and lftp is launched, else, the script stops.

 

#!/bin/bash

trap "rm -f /tmp/syncphotos.lock" SIGINT SIGTERM
if [ -e /tmp/syncphotos.lock ]
then
  echo "Syncphotos is running already."
  exit 1
else
  touch /tmp/syncphotos.lock
  lftp code
quit
EOF
  rm -f /tmp/syncphotos.lock
  /usr/local/emhttp/webGui/scripts/notify -i normal -s "Synchronisation ends" -e "Synchronisation photos"
  exit 0
fi

 

Thank you, I used your example and it works good.

One problem I have with my script is, when the ssh connection cant get established I dont get a fail warning, it looks like everything went well. How could I fix that?

Hi,

Is it me or the "tail" part in the notification doesn't work? When I launch the script after modifying all the path for my server ofc I get the following error:

 

tail: cannot open '/mnt/user/backup/logs/last.log' for reading: No such file or directory

 

this is my rsync command:

 

rsync -avhP --info=progress2 --no-compress --timeout=600 --log-file=/mnt/user/backup/logs/log.`date '+%Y%m%d-%H%M%S'`.log <SOURCE_PATH> <DEST_PATH>