ashman70 Posted March 19, 2016 Share Posted March 19, 2016 If this question has been asked before, my apologies. I am looking for a way to be able to sync two directories on two local UnRaid server. I would prefer a utility with a gui as I am not terribly conversant at the command line, however if someone could even help be put together a script that would work too. In short, I have server A which is the primary source of data, and I'd like to be able to sync it one way to a share with the same name on server B, both are on the same lan. Link to comment
garycase Posted March 19, 2016 Share Posted March 19, 2016 I'd use SyncBack on a Windows machine -- very simple to use and you can schedule the task to run at whatever time you'd like (e.g. in the "wee" hours when nothing else is going on). The free version is all you need: http://www.2brightsparks.com/ Note: Some folks setup a Windows VM just to run this utility -- if you don't have a Windows machine on your network that works very well. Link to comment
danioj Posted March 19, 2016 Share Posted March 19, 2016 If this question has been asked before, my apologies. I am looking for a way to be able to sync two directories on two local UnRaid server. I would prefer a utility with a gui as I am not terribly conversant at the command line, however if someone could even help be put together a script that would work too. In short, I have server A which is the primary source of data, and I'd like to be able to sync it one way to a share with the same name on server B, both are on the same lan. The short answer is there is no way to do this in unRAID. There are Dockers which you CAN use to do this for you such as Crashplan: https://lime-technology.com/forum/index.php?topic=33864.0 There have been efforts by unRAIDers to develop scripts to use Rsync to do what you are asking. See here: http://lime-technology.com/forum/index.php?topic=24581.0 Also, there is an experimental Docker for an application that was written to be a front-end to Rsync (more on this below). See here: http://lime-technology.com/forum/index.php?topic=39646.0 However (assuming that both servers are on a LAN where security is not an issue - like me) what you want to do is VERY simple using Rsync yourself. rsync is a widely-used utility to keep copies of a file on two computer systems the same. It is commonly found on Unix-like systems and functions as both a file synchronization and file transfer program. The rsync algorithm, a type of delta encoding, is used to minimize network usage. I know you said in your post you want to avoid the command line BUT sometimes it is unavoidable. I have learned this the hard way using unRAID. So, I am going to show you how simple using the command line to do this is: First, use the Unassigned Devices Plugin to Mount a SMB share to your unRAID Source machine: https://lime-technology.com/forum/index.php?topic=45807.0 You now have your destination (Backup) server mounted within your source server and should have something like this /mnt/disks/backup_nas Now to Rsync. This is all you need: rsync -a /path/to/source/ /path/to/destination/ e.g. I use the following: rsync -a /mnt/user/nas/ /mnt/disks/backup_nas/nas -a is for archive mode (implies recursive, copy symlinks as symlinks, preserve owner, modification times, group, owner, special and device files) Note: the trailing '/' on the end of the source path is important. With this the contents of the source directory will be duplicated into the destination directory. Without it the whole hierarchy of the source directory will be recreated within the destination directory. optional commands (There are heaps more than this BUT given what you have asked to be able to do I am trying not to overwhelm): rsync -a --progress /path/to/source/ /path/to/destination/ --progress shows progress of the file transfers on the command line rsync -a --progress --dry-run /path/to/source/ /path/to/destination/ --dry-run tells rsync to not do any file transfers, instead it will just report the actions it would have taken. Note: ALWAYS do a --dry-run BEFORE doing ANYTHING with your data. Look at the output. Test scenario's. When you're happy, remove the option from the command. Once you have a command that you are happy with, you can easily add this as a cron job utilising the power of dynamix. You do this on the command line, but once again it is easy: nano /boot/config/plugins/dynamix/dailybackup.cron If the "dailybackup.cron" file exists it will edit it. If not, it will create it. Paste the following into this file: # Perform a daily backup of my main share at 1:30am 30 1 * * * rsync -a /mnt/user/nas/ /mnt/disks/backup_nas/nas |& logger <CTRL> + <X> to exit and Save output. The cron Job is now saved. I have noticed that the dynamix cron files are not loaded just because you create a file BUT you can force this. Just go into the Settings>Scheduler AND make a change (so the Apply Button gets un-greyed out) and change it back (Apply Button remains un-greyed out) and Hit Apply (essentially not changing ANYTHING) then the cron file gets loaded. Voila. Done. As we have added |& logger to the command in the cron file, this will add the output of the command line to the unRAID syslog. Now, after all that I feel bad I haven't presented a solution that uses a GUI. So here goes. If you're willing to utilise the resources of a Windows VM to do this for you. I have used this before BUT have just made the switch to Rsync. SyncBack FREE is a GREAT tool: http://www.2brightsparks.com/welcome/backup/freeware.html?gclid=CN2xoNzRzcsCFQoQvQodcgUJ3A It's as simple as installing. Identifying a source. Identifying a destination. Give it some copy instructions (via tick boxes and switches). Set a schedule. Test it. Start it. Leave it alone. Link to comment
JorgeB Posted March 20, 2016 Share Posted March 20, 2016 I also use rsync, just wanted to add that for this you don’t need the unassigned devices plugin, you can run it like so: rsync -avAXP -e ssh root@source-ip-address:/mnt/user/Source-share/* /mnt/user/Dest-share Link to comment
trurl Posted March 20, 2016 Share Posted March 20, 2016 I also use rsync, just wanted to add that for this you don’t need the unassigned devices plugin, you can run it like so: rsync -avAXP -e ssh root@source-ip-address:/mnt/user/Source-share/* /mnt/user/Dest-share This is what I do. Link to comment
danioj Posted March 20, 2016 Share Posted March 20, 2016 I also use rsync, just wanted to add that for this you don’t need the unassigned devices plugin, you can run it like so: rsync -avAXP -e ssh root@source-ip-address:/mnt/user/Source-share/* /mnt/user/Dest-share You're right of course. I like to use Unassigned Devices to manage my SMB links to my Backup Server as it does just that, manages them for me. Link to comment
skylark Posted March 20, 2016 Share Posted March 20, 2016 Can i just say thanks for this comprehensive writeup as i am just about to build my backup server and was needing a solution to this very scenario as a linux 'non-geek'. Can i ask a couple of supplementary to complete the picture ... how can i wake up my backup server at a specific time to run the job and which side is best to control the cron job ie should it be run from the primary or backup server? I'm thinking the backup server IF it can be woken and powered off automatically. Dont really want it up and running 24x7. I'm running a pair of hp microservers (N54L's) btw. Link to comment
garycase Posted March 21, 2016 Share Posted March 21, 2016 Waking it up is trivial -- just send a magic packet. You can set a scheduled job to do this a minute or two before the backup starts. Shutting it down is a bit more complex, as you can't predict exactly when you'll need to do that. If you're using Windows, it's easy -- just include the shutdown command in the same batch file you use to start the backup, and it will run after the backup is completed. I'm sure there's a similar way to do it in Linux ... hopefully a Linux script expert will provide you the details. Link to comment
roland Posted March 21, 2016 Share Posted March 21, 2016 If you run the rsync from the backup server you could just include the appropriate shutdown command in the script. You could possibly even include the whole script in the go file, but just guessing here as I am not sure where in the startup sequence the go file falls. If this is possible you could wake up the backup server. It would run the rsync script and shut down again. Roland Link to comment
trurl Posted March 21, 2016 Share Posted March 21, 2016 If you run the rsync from the backup server you could just include the appropriate shutdown command in the script. You could possibly even include the whole script in the go file, but just guessing here as I am not sure where in the startup sequence the go file falls. If this is possible you could wake up the backup server. It would run the rsync script and shut down again. Roland Probably better to put an 'at' command in the go file to schedule the backup to give everything a little time to settle after booting up. Link to comment
comfox Posted March 21, 2016 Share Posted March 21, 2016 Wouldn't the BTSync docker add on do this? Runs real-time and has the GUI Link to comment
NotYetRated Posted March 29, 2016 Share Posted March 29, 2016 Should one shut down the VM's prior to backing up? Not really sure how KVM handles this... Link to comment
danioj Posted March 29, 2016 Share Posted March 29, 2016 Should one shut down the VM's prior to backing up? Not really sure how KVM handles this... Copying the image files is ok if the guest virtual machines are stopped. But if they are running its possible that as the copy is taking place changes are made within the guest, leading to an inconsistent copy. AFAIK a better way is to use LVM snapshots. lvm appears to be installed in the unRAID base OS and the commands required seem to be available. See here: http://vitobotta.com/setting-up-kvm-lvm-virtual-machine-host/ I have not tried this yet but I intend to try. If you end up doing this before me I would appreciate you posting your results. Link to comment
garycase Posted March 29, 2016 Share Posted March 29, 2016 Should one shut down the VM's prior to backing up? Not really sure how KVM handles this... If you're going to simply copy the virtual disk files to backup, then yes, the VM needs to be shut down before doing the backup. You can, however, use a "live" imager from within the virtual machine (e.g. Acronis, Ghost, Image for Windows, etc.) and back them up from within the running VM. Link to comment
De1taE1even Posted April 2, 2016 Share Posted April 2, 2016 Wouldn't the BTSync docker add on do this? Runs real-time and has the GUI I've used BTSync between 2 local unRaid servers, and can say it works very well, albeit a bit slow. I never saw more than about 15MBps transfer rate. That was fine for my use, but might not be for others. Link to comment
deionmann Posted April 2, 2016 Share Posted April 2, 2016 If this question has been asked before, my apologies. I am looking for a way to be able to sync two directories on two local UnRaid server. I would prefer a utility with a gui as I am not terribly conversant at the command line, however if someone could even help be put together a script that would work too. In short, I have server A which is the primary source of data, and I'd like to be able to sync it one way to a share with the same name on server B, both are on the same lan. The short answer is there is no way to do this in unRAID. There are Dockers which you CAN use to do this for you such as Crashplan: https://lime-technology.com/forum/index.php?topic=33864.0 There have been efforts by unRAIDers to develop scripts to use Rsync to do what you are asking. See here: http://lime-technology.com/forum/index.php?topic=24581.0 Also, there is an experimental Docker for an application that was written to be a front-end to Rsync (more on this below). See here: http://lime-technology.com/forum/index.php?topic=39646.0 However (assuming that both servers are on a LAN where security is not an issue - like me) what you want to do is VERY simple using Rsync yourself. rsync is a widely-used utility to keep copies of a file on two computer systems the same. It is commonly found on Unix-like systems and functions as both a file synchronization and file transfer program. The rsync algorithm, a type of delta encoding, is used to minimize network usage. I know you said in your post you want to avoid the command line BUT sometimes it is unavoidable. I have learned this the hard way using unRAID. So, I am going to show you how simple using the command line to do this is: First, use the Unassigned Devices Plugin to Mount a SMB share to your unRAID Source machine: https://lime-technology.com/forum/index.php?topic=45807.0 You now have your destination (Backup) server mounted within your source server and should have something like this /mnt/disks/backup_nas Now to Rsync. This is all you need: rsync -a /path/to/source/ /path/to/destination/ e.g. I use the following: rsync -a /mnt/user/nas/ /mnt/disks/backup_nas/nas -a is for archive mode (implies recursive, copy symlinks as symlinks, preserve owner, modification times, group, owner, special and device files) Note: the trailing '/' on the end of the source path is important. With this the contents of the source directory will be duplicated into the destination directory. Without it the whole hierarchy of the source directory will be recreated within the destination directory. optional commands (There are heaps more than this BUT given what you have asked to be able to do I am trying not to overwhelm): rsync -a --progress /path/to/source/ /path/to/destination/ --progress shows progress of the file transfers on the command line rsync -a --progress --dry-run /path/to/source/ /path/to/destination/ --dry-run tells rsync to not do any file transfers, instead it will just report the actions it would have taken. Note: ALWAYS do a --dry-run BEFORE doing ANYTHING with your data. Look at the output. Test scenario's. When you're happy, remove the option from the command. Once you have a command that you are happy with, you can easily add this as a cron job utilising the power of dynamix. You do this on the command line, but once again it is easy: nano /boot/config/plugins/dynamix/dailybackup.cron If the "dailybackup.cron" file exists it will edit it. If not, it will create it. Paste the following into this file: # Perform a daily backup of my main share at 1:30am 30 1 * * * rsync -a /mnt/user/nas/ /mnt/disks/backup_nas/nas |& logger <CTRL> + <X> to exit and Save output. The cron Job is now saved. I have noticed that the dynamix cron files are not loaded just because you create a file BUT you can force this. Just go into the Settings>Scheduler AND make a change (so the Apply Button gets un-greyed out) and change it back (Apply Button remains un-greyed out) and Hit Apply (essentially not changing ANYTHING) then the cron file gets loaded. Voila. Done. As we have added |& logger to the command in the cron file, this will add the output of the command line to the unRAID syslog. Now, after all that I feel bad I haven't presented a solution that uses a GUI. So here goes. If you're willing to utilise the resources of a Windows VM to do this for you. I have used this before BUT have just made the switch to Rsync. SyncBack FREE is a GREAT tool: http://www.2brightsparks.com/welcome/backup/freeware.html?gclid=CN2xoNzRzcsCFQoQvQodcgUJ3A It's as simple as installing. Identifying a source. Identifying a destination. Give it some copy instructions (via tick boxes and switches). Set a schedule. Test it. Start it. Leave it alone. Thats great! thank you for this. Quick question though, if I wanted unraid to send me a notification when this cron job starts and finishes, is there anyway to add that capability to this script? Link to comment
roland Posted April 2, 2016 Share Posted April 2, 2016 You can use a line like this. It will generate a notification that you can manage thru the normal notification settings. /usr/local/sbin/notify -i normal -s "Appdata Backup Completed" -d " Appdata Backup completed at `date`" i: importance s: subject d: description I use this in my backup script. Link to comment
bonienl Posted April 2, 2016 Share Posted April 2, 2016 You can use a line like this. It will generate a notification that you can manage thru the normal notification settings. /usr/local/sbin/notify -i normal -s "Appdata Backup Completed" -d " Appdata Backup completed at `date`" i: importance s: subject d: description I use this in my backup script. From unRAID version 6.1 and higher you need to use: /usr/local/emhttp/webGui/scripts/notify Link to comment
chip Posted April 6, 2016 Share Posted April 6, 2016 Is that like a log file? I would like a log file if possible after the job runs. Also my test rsync job shows the following in the log file failed parsing crontab for user root: rsync -a /mnt/user/files/NetflixMCE_BETA /mnt/disk2/unRAID_Test It is running but what is with the failed message? Link to comment
deionmann Posted April 6, 2016 Share Posted April 6, 2016 Thanks guys, I think I got it working pretty good. I ended up with a cron that points to a .sh file Cron: #Perform daily backup of Pictures share from Godzilla > Mothra at 12:30am 30 00 * * * /boot/config/plugins/dynamix/godzilla_pictures.sh |& logger .sh: #!/bin/bash ##Perform backup of Pictures share from Godzilla > Mothra and notify /usr/local/emhttp/webGui/scripts/notify -e "Rsync Backup" -i "normal" -s "Rsync backup of Godzilla share: Pictures started at `date`" rsync -a --progress /mnt/disks/godzilla_pictures/ /mnt/user/godzilla/pictures |& logger /usr/local/emhttp/webGui/scripts/notify -e "Rsync Backup" -i "normal" -s "Rsync backup of Godzilla share: Pictures complete at `date`" Now let me ask you guys this. Is there anyway that I can have rsync log a synopsis of what it did? Like for example "synced 544 files in 30 minutes with an average of 102MBs" Then loop that info into the notification when rsync finishes? I know, I know, it's overkill.... But it would be cool [emoji41] Link to comment
gundamguy Posted April 6, 2016 Share Posted April 6, 2016 Now let me ask you guys this. Is there anyway that I can have rsync log a synopsis of what it did? Like for example "synced 544 files in 30 minutes with an average of 102MBs" Then loop that info into the notification when rsync finishes? I know, I know, it's overkill.... But it would be cool [emoji41] Yes, not sure exactly how you want it to work, because there are about a thousand different ways to approach this, but if you want a wholly separate log file, you could pass the --log-file=FILE argument to rsync as part of your rsync call. It would then look something like this. rsync -a --progress --log-file=/mnt/cache/apps/RSYNCLOG.log /mnt/disks/godzilla_pictures/ /mnt/user/godzilla/pictures There are ways to improve this too, like you could create a variable and use that to dynamically name the log with the date... or something. But this is the basic approach. More info to be found in the Manpage. You might want to increase verbosity if --progress isn't enough on it's own. Link to comment
deionmann Posted April 6, 2016 Share Posted April 6, 2016 Thanks allot @gundamguy Think I finally got my script down #!/bin/bash ##Perform backup of Documents share from Godzilla > Mothra and notify /usr/local/emhttp/webGui/scripts/notify -s "`hostname` Rsync Backup Started" -i "normal" -d "Backup of Documents share started `date`" rsync -avh --log-file=/mnt/user/godzilla/documents/documents-backup.log /mnt/disks/godzilla_documents/ /mnt/user/godzilla/documents /usr/local/emhttp/webGui/scripts/notify -s "`hostname` Rsync Backup complete" -i "normal" -d "Backup of Documents share complete `date` `rsync -avh --log-file=/mnt/user/godzilla/documents/documents-backup.log /mnt/disks/godzilla_documents/ /mnt/user/godzilla/documents`" ends up giving me this via pushover Link to comment
chip Posted April 6, 2016 Share Posted April 6, 2016 To build off of deionmann question of rsync log files...... Can it output data like robocopy & send it as an email once it is done running? Total Copied Skipped Mismatch FAILED Extras Dirs : 53 53 0 0 0 0 Files : 356 356 0 0 0 0 Bytes : 56.03 m 56.03 m 0 0 0 0 Times : 0:00:04 0:00:03 0:00:00 0:00:00 Speed : 14988240 Bytes/sec. Speed : 857.633 MegaBytes/min. Ended : Wed Apr 06 08:00:50 2016 Link to comment
deionmann Posted April 7, 2016 Share Posted April 7, 2016 To build off of deionmann question of rsync log files...... Can it output data like robocopy & send it as an email once it is done running? Total Copied Skipped Mismatch FAILED Extras Dirs : 53 53 0 0 0 0 Files : 356 356 0 0 0 0 Bytes : 56.03 m 56.03 m 0 0 0 0 Times : 0:00:04 0:00:03 0:00:00 0:00:00 Speed : 14988240 Bytes/sec. Speed : 857.633 MegaBytes/min. Ended : Wed Apr 06 08:00:50 2016 I was playing around last night and was able to get something similar to the robocopy printout 2016/04/06 23:44:53 [14503] Number of files: 17 (reg: 13, dir: 4) 2016/04/06 23:44:53 [14503] Number of created files: 0 2016/04/06 23:44:53 [14503] Number of deleted files: 0 2016/04/06 23:44:53 [14503] Number of regular files transferred: 0 2016/04/06 23:44:53 [14503] Total file size: 966.53M bytes 2016/04/06 23:44:53 [14503] Total transferred file size: 0 bytes 2016/04/06 23:44:53 [14503] Literal data: 0 bytes 2016/04/06 23:44:53 [14503] Matched data: 0 bytes 2016/04/06 23:44:53 [14503] File list size: 0 2016/04/06 23:44:53 [14503] File list generation time: 0.001 seconds 2016/04/06 23:44:53 [14503] File list transfer time: 0.000 seconds 2016/04/06 23:44:53 [14503] Total bytes sent: 390 2016/04/06 23:44:53 [14503] Total bytes received: 15 2016/04/06 23:44:53 [14503] sent 390 bytes received 15 bytes 810.00 bytes/sec 2016/04/06 23:44:53 [14503] total size is 966.53M speedup is 2,386,492.95 adding the --stat and --log-file=[location of logfile] triggers will create that printout in an external log file of your choosing then on your message body use `tail [location of your logfile]` Link to comment
chip Posted April 7, 2016 Share Posted April 7, 2016 So how would you append it to the file listed above - sync -a --progress --log-file=/mnt/cache/apps/RSYNCLOG.log /mnt/disks/godzilla_pictures/ /mnt/user/godzilla/pictures Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.