Sync directories between two UnRaid servers

[ashman70]

If this question has been asked before, my apologies. I am looking for a way to be able to sync two directories on two local UnRaid server. I would prefer a utility with a gui as I am not terribly conversant at the command line, however if someone could even help be put together a script that would work too. In short, I have server A which is the primary source of data, and I'd like to be able to sync it one way to a share with the same name on server B, both are on the same lan.

[garycase]

I'd use SyncBack on a Windows machine -- very simple to use and you can schedule the task to run at whatever time you'd like (e.g. in the "wee" hours when nothing else is going on).

 

The free version is all you need:  http://www.2brightsparks.com/

 

Note: Some folks setup a Windows VM just to run this utility -- if you don't have a Windows machine on your network that works very well.

 

[danioj]

If this question has been asked before, my apologies. I am looking for a way to be able to sync two directories on two local UnRaid server. I would prefer a utility with a gui as I am not terribly conversant at the command line, however if someone could even help be put together a script that would work too. In short, I have server A which is the primary source of data, and I'd like to be able to sync it one way to a share with the same name on server B, both are on the same lan.

 

The short answer is there is no way to do this in unRAID. There are Dockers which you CAN use to do this for you such as Crashplan:

 

https://lime-technology.com/forum/index.php?topic=33864.0

 

There have been efforts by unRAIDers to develop scripts to use Rsync to do what you are asking. See here:

 

http://lime-technology.com/forum/index.php?topic=24581.0

 

Also, there is an experimental Docker for an application that was written to be a front-end to Rsync (more on this below). See here:

 

http://lime-technology.com/forum/index.php?topic=39646.0

 

 

However (assuming that both servers are on a LAN where security is not an issue - like me) what you want to do is VERY simple using Rsync yourself.

 

rsync is a widely-used utility to keep copies of a file on two computer systems the same. It is commonly found on Unix-like systems and functions as both a file synchronization and file transfer program. The rsync algorithm, a type of delta encoding, is used to minimize network usage.

 

I know you said in your post you want to avoid the command line BUT sometimes it is unavoidable. I have learned this the hard way using unRAID. So, I am going to show you how simple using the command line to do this is:

 

First, use the Unassigned Devices Plugin to Mount a SMB share to your unRAID Source machine:

 

https://lime-technology.com/forum/index.php?topic=45807.0

 

You now have your destination (Backup) server mounted within your source server and should have something like this /mnt/disks/backup_nas

 

Now to Rsync. This is all you need:

 

rsync -a /path/to/source/ /path/to/destination/

 

e.g. I use the following:

 

rsync -a /mnt/user/nas/ /mnt/disks/backup_nas/nas

 

-a is for archive mode (implies recursive, copy symlinks as symlinks, preserve owner, modification times, group, owner, special and device files)

 

Note: the trailing '/' on the end of the source path is important. With this the contents of the source directory will be duplicated into the destination directory. Without it the whole hierarchy of the source directory will be recreated within the destination directory.

 

optional commands (There are heaps more than this BUT given what you have asked to be able to do I am trying not to overwhelm):

 

rsync -a --progress /path/to/source/ /path/to/destination/

 

--progress shows progress of the file transfers on the command line

 

rsync -a --progress  --dry-run /path/to/source/ /path/to/destination/

 

--dry-run tells rsync to not do any file transfers, instead it will just report the actions it would have taken.

 

Note: ALWAYS do a --dry-run BEFORE doing ANYTHING with your data. Look at the output. Test scenario's. When you're happy, remove the option from the command.

 

Once you have a command that you are happy with, you can easily add this as a cron job utilising the power of dynamix. You do this on the command line, but once again it is easy:

 

nano /boot/config/plugins/dynamix/dailybackup.cron

 

If the "dailybackup.cron" file exists it will edit it. If not, it will create it. Paste the following into this file:

 

# Perform a daily backup of my main share at 1:30am
30 1 * * * rsync -a /mnt/user/nas/ /mnt/disks/backup_nas/nas |& logger

 

<CTRL> + <X> to exit and Save output. The cron Job is now saved. I have noticed that the dynamix cron files are not loaded just because you create a file BUT you can force this. Just go into the Settings>Scheduler AND make a change (so the Apply Button gets un-greyed out) and change it back (Apply Button remains un-greyed out) and Hit Apply (essentially not changing ANYTHING) then the cron file gets loaded.

 

Voila. Done. As we have added |& logger to the command in the cron file, this will add the output of the command line to the unRAID syslog.

 

Now, after all that I feel bad I haven't presented a solution that uses a GUI. So here goes. If you're willing to utilise the resources of a Windows VM to do this for you. I have used this before BUT have just made the switch to Rsync. SyncBack FREE is a GREAT tool:

 

http://www.2brightsparks.com/welcome/backup/freeware.html?gclid=CN2xoNzRzcsCFQoQvQodcgUJ3A

 

It's as simple as installing. Identifying a source. Identifying a destination. Give it some copy instructions (via tick boxes and switches). Set a schedule. Test it. Start it. Leave it alone.

[JorgeB]

I also use rsync, just wanted to add that for this you don’t need the unassigned devices plugin, you can run it like so:

 

rsync -avAXP -e ssh root@source-ip-address:/mnt/user/Source-share/* /mnt/user/Dest-share

[trurl]

I also use rsync, just wanted to add that for this you don’t need the unassigned devices plugin, you can run it like so:

 

rsync -avAXP -e ssh root@source-ip-address:/mnt/user/Source-share/* /mnt/user/Dest-share

This is what I do.

[danioj]

I also use rsync, just wanted to add that for this you don’t need the unassigned devices plugin, you can run it like so:

 

rsync -avAXP -e ssh root@source-ip-address:/mnt/user/Source-share/* /mnt/user/Dest-share

 

You're right of course.

 

I like to use Unassigned Devices to manage my SMB links to my Backup Server as it does just that, manages them for me.

[skylark]

Can i just say thanks for this comprehensive writeup as i am just about to build my backup server and was needing a solution to this very scenario as a linux 'non-geek'. Can i ask a couple of supplementary to complete the picture ... how can i wake up my backup server at a specific time to run the job and which side is best to control the cron job ie should it be run from the primary or backup server? I'm thinking the backup server IF it can be woken and powered off automatically. Dont really want it up and running 24x7. I'm running a pair of hp microservers (N54L's) btw.

 

[garycase]

Waking it up is trivial -- just send a magic packet.    You can set a scheduled job to do this a minute or two before the backup starts.    Shutting it down is a bit more complex, as you can't predict exactly when you'll need to do that.  If you're using Windows, it's easy -- just include the shutdown command in the same batch file you use to start the backup, and it will run after the backup is completed.    I'm sure there's a similar way to do it in Linux ... hopefully a Linux script expert will provide you the details.

 

[roland]

If you run the rsync from the backup server you could just include the appropriate shutdown command in the script.

 

You could possibly even include the whole script in the go file, but just guessing here as I am not sure where in the startup sequence the go file falls.

If this is possible you could wake up the backup server.

It would run the rsync script and shut down again.

 

Roland

[trurl]

If you run the rsync from the backup server you could just include the appropriate shutdown command in the script.

 

You could possibly even include the whole script in the go file, but just guessing here as I am not sure where in the startup sequence the go file falls.

If this is possible you could wake up the backup server.

It would run the rsync script and shut down again.

 

Roland

Probably better to put an 'at' command in the go file to schedule the backup to give everything a little time to settle after booting up.

[comfox]

Wouldn't the BTSync docker add on do this?  Runs real-time and has the GUI

[NotYetRated]

Should one shut down the VM's prior to backing up? Not really sure how KVM handles this...

[danioj]

Should one shut down the VM's prior to backing up? Not really sure how KVM handles this...

 

Copying the image files is ok if the guest virtual machines are stopped. But if they are running its possible that as the copy is taking place changes are made within the guest, leading to an inconsistent copy.

 

AFAIK a better way is to use LVM snapshots. lvm appears to be installed in the unRAID base OS and the commands required seem to be available. See here:

 

http://vitobotta.com/setting-up-kvm-lvm-virtual-machine-host/

 

I have not tried this yet but I intend to try. If you end up doing this before me I would appreciate you posting your results.

[garycase]

Should one shut down the VM's prior to backing up? Not really sure how KVM handles this...

 

If you're going to simply copy the virtual disk files to backup, then yes, the VM needs to be shut down before doing the backup.    You can, however, use a "live" imager from within the virtual machine (e.g. Acronis, Ghost, Image for Windows, etc.) and back them up from within the running VM.

 

[De1taE1even]

Wouldn't the BTSync docker add on do this?  Runs real-time and has the GUI

 

I've used BTSync between 2 local unRaid servers, and can say it works very well, albeit a bit slow.  I never saw more than about 15MBps transfer rate.  That was fine for my use, but might not be for others.

[deionmann]

If this question has been asked before, my apologies. I am looking for a way to be able to sync two directories on two local UnRaid server. I would prefer a utility with a gui as I am not terribly conversant at the command line, however if someone could even help be put together a script that would work too. In short, I have server A which is the primary source of data, and I'd like to be able to sync it one way to a share with the same name on server B, both are on the same lan.

 

The short answer is there is no way to do this in unRAID. There are Dockers which you CAN use to do this for you such as Crashplan:

 

https://lime-technology.com/forum/index.php?topic=33864.0

 

There have been efforts by unRAIDers to develop scripts to use Rsync to do what you are asking. See here:

 

http://lime-technology.com/forum/index.php?topic=24581.0

 

Also, there is an experimental Docker for an application that was written to be a front-end to Rsync (more on this below). See here:

 

http://lime-technology.com/forum/index.php?topic=39646.0

 

 

However (assuming that both servers are on a LAN where security is not an issue - like me) what you want to do is VERY simple using Rsync yourself.

 

rsync is a widely-used utility to keep copies of a file on two computer systems the same. It is commonly found on Unix-like systems and functions as both a file synchronization and file transfer program. The rsync algorithm, a type of delta encoding, is used to minimize network usage.

 

I know you said in your post you want to avoid the command line BUT sometimes it is unavoidable. I have learned this the hard way using unRAID. So, I am going to show you how simple using the command line to do this is:

 

First, use the Unassigned Devices Plugin to Mount a SMB share to your unRAID Source machine:

 

https://lime-technology.com/forum/index.php?topic=45807.0

 

You now have your destination (Backup) server mounted within your source server and should have something like this /mnt/disks/backup_nas

 

Now to Rsync. This is all you need:

 

rsync -a /path/to/source/ /path/to/destination/

 

e.g. I use the following:

 

rsync -a /mnt/user/nas/ /mnt/disks/backup_nas/nas

 

-a is for archive mode (implies recursive, copy symlinks as symlinks, preserve owner, modification times, group, owner, special and device files)

 

Note: the trailing '/' on the end of the source path is important. With this the contents of the source directory will be duplicated into the destination directory. Without it the whole hierarchy of the source directory will be recreated within the destination directory.

 

optional commands (There are heaps more than this BUT given what you have asked to be able to do I am trying not to overwhelm):

 

rsync -a --progress /path/to/source/ /path/to/destination/

 

--progress shows progress of the file transfers on the command line

 

rsync -a --progress  --dry-run /path/to/source/ /path/to/destination/

 

--dry-run tells rsync to not do any file transfers, instead it will just report the actions it would have taken.

 

Note: ALWAYS do a --dry-run BEFORE doing ANYTHING with your data. Look at the output. Test scenario's. When you're happy, remove the option from the command.

 

Once you have a command that you are happy with, you can easily add this as a cron job utilising the power of dynamix. You do this on the command line, but once again it is easy:

 

nano /boot/config/plugins/dynamix/dailybackup.cron

 

If the "dailybackup.cron" file exists it will edit it. If not, it will create it. Paste the following into this file:

 

# Perform a daily backup of my main share at 1:30am
30 1 * * * rsync -a /mnt/user/nas/ /mnt/disks/backup_nas/nas |& logger

 

<CTRL> + <X> to exit and Save output. The cron Job is now saved. I have noticed that the dynamix cron files are not loaded just because you create a file BUT you can force this. Just go into the Settings>Scheduler AND make a change (so the Apply Button gets un-greyed out) and change it back (Apply Button remains un-greyed out) and Hit Apply (essentially not changing ANYTHING) then the cron file gets loaded.

 

Voila. Done. As we have added |& logger to the command in the cron file, this will add the output of the command line to the unRAID syslog.

 

Now, after all that I feel bad I haven't presented a solution that uses a GUI. So here goes. If you're willing to utilise the resources of a Windows VM to do this for you. I have used this before BUT have just made the switch to Rsync. SyncBack FREE is a GREAT tool:

 

http://www.2brightsparks.com/welcome/backup/freeware.html?gclid=CN2xoNzRzcsCFQoQvQodcgUJ3A

 

It's as simple as installing. Identifying a source. Identifying a destination. Give it some copy instructions (via tick boxes and switches). Set a schedule. Test it. Start it. Leave it alone.

Thats great! thank you for this.

 

Quick question though, if I wanted unraid to send me a notification when this cron job starts and finishes, is there anyway to add that capability to this script?

[roland]

You can use a line like this.  It will generate a notification that you can manage thru the normal notification settings.

/usr/local/sbin/notify -i normal -s "Appdata Backup Completed" -d " Appdata Backup completed at `date`"

 

i: importance

s: subject

d: description

 

I use this in my backup script.

[bonienl]

You can use a line like this.  It will generate a notification that you can manage thru the normal notification settings.

/usr/local/sbin/notify -i normal -s "Appdata Backup Completed" -d " Appdata Backup completed at `date`"

 

i: importance

s: subject

d: description

 

I use this in my backup script.

 

From unRAID version 6.1 and higher you need to use: /usr/local/emhttp/webGui/scripts/notify

 

[chip]

Is that like a log file? I would like a log file if possible after the job runs.

 

Also my test rsync job shows the following in the log file

 

failed parsing crontab for user root: rsync -a /mnt/user/files/NetflixMCE_BETA /mnt/disk2/unRAID_Test

 

It is running but what is with the failed message?

[deionmann]

Thanks guys, I think I got it working pretty good. I ended up with a cron that points to a .sh file

 

Cron:

#Perform daily backup of Pictures share from Godzilla > Mothra at 12:30am

 

30 00 * * * /boot/config/plugins/dynamix/godzilla_pictures.sh |& logger

 

.sh:

#!/bin/bash

##Perform backup of Pictures share from Godzilla > Mothra and notify

 

/usr/local/emhttp/webGui/scripts/notify -e "Rsync Backup" -i "normal" -s "Rsync backup of Godzilla share: Pictures started at `date`"

 

rsync -a --progress /mnt/disks/godzilla_pictures/ /mnt/user/godzilla/pictures |& logger

 

/usr/local/emhttp/webGui/scripts/notify -e "Rsync Backup" -i "normal" -s "Rsync backup of Godzilla share: Pictures complete at `date`"

 

 

Now let me ask you guys this. Is there anyway that I can have rsync log a synopsis of what it did? Like for example "synced 544 files in 30 minutes with an average of 102MBs"

 

Then loop that info into the notification when rsync finishes?

 

I know, I know, it's overkill.... But it would be cool

[gundamguy]

 

Now let me ask you guys this. Is there anyway that I can have rsync log a synopsis of what it did? Like for example "synced 544 files in 30 minutes with an average of 102MBs"

 

Then loop that info into the notification when rsync finishes?

 

I know, I know, it's overkill.... But it would be cool

 

Yes, not sure exactly how you want it to work, because there are about a thousand different ways to approach this, but if you want a wholly separate log file, you could pass the --log-file=FILE argument to rsync as part of your rsync call.

 

It would then look something like this.

 

rsync -a --progress --log-file=/mnt/cache/apps/RSYNCLOG.log /mnt/disks/godzilla_pictures/ /mnt/user/godzilla/pictures

 

There are ways to improve this too, like you could create a variable and use that to dynamically name the log with the date... or something. But this is the basic approach.

 

More info to be found in the Manpage.  You might want to increase verbosity if --progress isn't enough on it's own.

[deionmann]

Thanks allot @gundamguy

 

Think I finally got my script down

 

#!/bin/bash 
##Perform backup of Documents share from Godzilla > Mothra and notify

/usr/local/emhttp/webGui/scripts/notify -s "`hostname` Rsync Backup Started" -i "normal" -d "Backup of Documents share started `date`"

rsync -avh --log-file=/mnt/user/godzilla/documents/documents-backup.log /mnt/disks/godzilla_documents/ /mnt/user/godzilla/documents

/usr/local/emhttp/webGui/scripts/notify -s "`hostname` Rsync Backup complete" -i "normal" -d "Backup of Documents share complete `date` `rsync -avh --log-file=/mnt/user/godzilla/documents/documents-backup.log /mnt/disks/godzilla_documents/ /mnt/user/godzilla/documents`"

 

ends up giving me this via pushover

[chip]

To build off of deionmann question of  rsync log files......

 

Can it output data like robocopy & send it as an email once it is done running?

 

              Total    Copied  Skipped  Mismatch    FAILED    Extras

  Dirs :        53        53        0        0        0        0

  Files :      356      356        0        0        0        0

  Bytes :  56.03 m  56.03 m        0        0        0        0

  Times :  0:00:04  0:00:03                      0:00:00  0:00:00

 

 

  Speed :            14988240 Bytes/sec.

  Speed :            857.633 MegaBytes/min.

 

  Ended : Wed Apr 06 08:00:50 2016

 

[deionmann]

To build off of deionmann question of  rsync log files......

 

Can it output data like robocopy & send it as an email once it is done running?

 

              Total    Copied  Skipped  Mismatch    FAILED    Extras

  Dirs :        53        53        0        0        0        0

  Files :      356      356        0        0        0        0

  Bytes :  56.03 m  56.03 m        0        0        0        0

  Times :  0:00:04  0:00:03                      0:00:00  0:00:00

 

 

  Speed :            14988240 Bytes/sec.

  Speed :            857.633 MegaBytes/min.

 

  Ended : Wed Apr 06 08:00:50 2016

 

I was playing around last night and was able to get something similar to the robocopy printout

 

2016/04/06 23:44:53 [14503] Number of files: 17 (reg: 13, dir: 4)
2016/04/06 23:44:53 [14503] Number of created files: 0
2016/04/06 23:44:53 [14503] Number of deleted files: 0
2016/04/06 23:44:53 [14503] Number of regular files transferred: 0
2016/04/06 23:44:53 [14503] Total file size: 966.53M bytes
2016/04/06 23:44:53 [14503] Total transferred file size: 0 bytes
2016/04/06 23:44:53 [14503] Literal data: 0 bytes
2016/04/06 23:44:53 [14503] Matched data: 0 bytes
2016/04/06 23:44:53 [14503] File list size: 0
2016/04/06 23:44:53 [14503] File list generation time: 0.001 seconds
2016/04/06 23:44:53 [14503] File list transfer time: 0.000 seconds
2016/04/06 23:44:53 [14503] Total bytes sent: 390
2016/04/06 23:44:53 [14503] Total bytes received: 15
2016/04/06 23:44:53 [14503] sent 390 bytes  received 15 bytes  810.00 bytes/sec
2016/04/06 23:44:53 [14503] total size is 966.53M  speedup is 2,386,492.95

 

adding the

--stat

and

--log-file=[location of logfile]

triggers will create that printout in an external log file of your choosing then on your message body use

`tail [location of your logfile]`

[chip]

So how would you append it to the file listed above -

 

sync -a --progress --log-file=/mnt/cache/apps/RSYNCLOG.log /mnt/disks/godzilla_pictures/ /mnt/user/godzilla/pictures