dedi Posted July 25, 2018 Share Posted July 25, 2018 (edited) I thought I share my rsync config. I still feel like Unraid as a Storage Server should provide a build in method to backup or sync its data to an other location, but configuring rsync gives you a lot of power, and once set up, it's just great. I based my config and setup on the tutorial found here: https://lime-technology.com/forums/topic/52830-syncronize-servers-using-rsync-over-ssh-next-door-or-across-the-world/ And I made some adjustments, specially the option to watch the sync going on, and the notification that works with the build in unraid notification. So on "errors" you will get a E-Mail notification if your unraid is setup for that. The above tutorial covers the SSH setup and connect via key instead of login, and the basic rsync setup. You can continue after that guide. What you need is the NerdPack Plugin, and there install "screen". It is used to run rsync in it own screen, giving you the option to pull the progress up in a console at any time. The second one is optional but recommended, its the "User Scripts" plugin. It allows you to add own scripts and execute it via cron in your unraid GUI. I created 2 scripts for my rsync: The first one I called "rsync" and does the rsync job, and depending on its exit code, it gives the notification to the Unraid Notification Service. #!/bin/bash rsync -avu --bwlimit=4000 --numeric-ids --log-file=/mnt/user/Backup/rsync-logs/log.`date '+%Y_%m_%d__%H_%M_%S'`.log --progress --exclude 'C Sys' --timeout=600 -e "ssh -p 4222 -i /root/.ssh/DSStoDMS-key -T -o Compression=no -x " /mnt/user/Backup/Devices/ [email protected]:/volume2/Backup/Unraid_rsync if [ $? -eq 0 ] then /usr/local/emhttp/webGui/scripts/notify -s "`hostname` to DMS Rsync Backup complete" -d "Sync compled. `date` `tail /mnt/user/Backup/rsync-logs/last.log`" else /usr/local/emhttp/webGui/scripts/notify -s "`hostname` to DMS Rsync Backup FAILED" -i "alert" -d "Sync compled. `date` `tail /mnt/user/Backup/rsync-logs/last.log`" fi Of course, the rsync command would need some adjustments. If your Unraid server is setup to send you a E-Mail on errors, it will do that too if rsync fails! The second one is for calling that script in a "screen" session. #!/bin/bash export SCREENDIR=/root/.screen screen -dmS rsyncdms /boot/config/plugins/user.scripts/scripts/rsync/script You can pull it up as long as it is running with screen -r from a console. You can just use the build in one from the WebGUI. Attention, if you name the first script different than "rsync", you have to adjust this location. Now just set a schedule for the second script. Watch its progress with "rsync -r" I hope this can help somebody. Im not a rsync expert, so if anyone is around with more experience has some input, its very welcome. And I have a questions myself: I have large files to transfer, sometimes a lot of new data comes up at once. So if I have to transfer >400GB, it needs more than 24h, and the rsync backup script (which is scheduled to run daily) is started again. It starts with the files that is currently transfering in the first instance of rsync. It works, but it will transfer that big file twice. Should I check when running the script if rsync is already running? Or does running rsync with --partial do any good or bad here? Edited July 25, 2018 by dedi 1 Quote Link to comment
deadnote Posted July 29, 2018 Share Posted July 29, 2018 Hi Thanks for your scripts, I think I'm gonna use them Maybe this part of my lftp script can help your for your large file transfer. I check if a file .lock exists in temp folder. If not, the file is created and lftp is launched, else, the script stops. #!/bin/bash trap "rm -f /tmp/syncphotos.lock" SIGINT SIGTERM if [ -e /tmp/syncphotos.lock ] then echo "Syncphotos is running already." exit 1 else touch /tmp/syncphotos.lock lftp code quit EOF rm -f /tmp/syncphotos.lock /usr/local/emhttp/webGui/scripts/notify -i normal -s "Synchronisation ends" -e "Synchronisation photos" exit 0 fi Quote Link to comment
dedi Posted October 25, 2018 Author Share Posted October 25, 2018 Thank you, I used your example and it works good. One problem I have with my script is, when the ssh connection cant get established I dont get a fail warning, it looks like everything went well. How could I fix that? Quote Link to comment
ale.mediaserver Posted July 13, 2023 Share Posted July 13, 2023 Hi, Is it me or the "tail" part in the notification doesn't work? When I launch the script after modifying all the path for my server ofc I get the following error: tail: cannot open '/mnt/user/backup/logs/last.log' for reading: No such file or directory this is my rsync command: rsync -avhP --info=progress2 --no-compress --timeout=600 --log-file=/mnt/user/backup/logs/log.`date '+%Y%m%d-%H%M%S'`.log <SOURCE_PATH> <DEST_PATH> Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.