dalben Posted August 25, 2011 Posted August 25, 2011 I'm in the throws of getting my first unRaid server running (thought it would singing and dancing as we speak but I underestimated how loing a pre-clear takes). One of the critical pieces I need to find a solution for is remote replication. Searching the forum I assume that a mix of rsync and cron will have me doing what I want. Basically copying certain shares from my unraid server to other devices, either on my home LAN or tunnelled remotely. I'm pretty sure I can just rsync just changed files. Anyway, being the complete linux noob that I am, I was hiping there there is some sort of GUI or Wizard that I can step through, point and click to destinations, times etc and have it all set up automagically. Am I correct in assuming such a beast doesn't exist ? Failing that, is there anything out there, other than a man page, that can make my life easier when setting and maintaining these scripts ? I've seen Crashplan mentioned so I might look at that, but basically I am looking for replication to a couple of dedicated NAS boxes I have on the home lan and possibly a remote NAS or Server, but that can wait. Will I need to get down and dirty with line editors to get what I want up and running ?
WeeboTech Posted August 25, 2011 Posted August 25, 2011 I think it depends on where you want to run these backups. There are plenty of backup tools on windows platforms. But if you want it run behind the scene in the background, then knowing an editor. rsync, unison or some similiar tool will be the best way. I use rsync from one unRAID server to another. I found a way and devised a script that would do incremental rsync'ed backups. Unison is another tool that's available on windows. There are many other windows like tools that can do this. I thin I use something like pbsync to do my DJ drive to an archive drive on the lan.
sam9s Posted August 25, 2011 Posted August 25, 2011 @WeeboTech would rsync sync files from a remote connection via VPN. I mean I access my Unraid Box from my office via VPN, would rsync sync files between my office pc and my home Unraid. If yes can you point me towards a tutorial that can guide how to set up rsync on Unraid. Appreciate your help. Thank in advance Regards Sammy
WeeboTech Posted August 25, 2011 Posted August 25, 2011 There's a wealth of posts on rsync and setting up rsync as a server with rsyncd.conf. Do a search on the board. Here's an example http://lime-technology.com/forum/index.php?topic=12237.0 As long as the vpn allows passing ssh port or rsync server port, you should be OK. Although I have no experience with using rsync on a windows platform. If you can access the disk shares over the vpn, then you can use a windows program like unison. deltasync, pbsync, robosync or others.
dalben Posted August 25, 2011 Author Posted August 25, 2011 I'm hoping for a backend set and forget setup that won't need any pc client running to work. Ideally with email notification on failure. But if that's too much, maybe a Grsync front end to help set it all up on the unraid server would help non-*nix people. I have sw and tools on a windows platform that can automate and manage all of this for me so I'm actually thinking of installing a virtualbox running XP to handle this task for me. Seems like an overkill but if it doesn't hurt performance it might solve all my issues.
WeeboTech Posted August 26, 2011 Posted August 26, 2011 rsync will do what you want, it just requires a little command line work.
dalben Posted August 26, 2011 Author Posted August 26, 2011 rsync will do what you want, it just requires a little command line work. Yes, I know. That's the point of the whole thread.
sam9s Posted August 26, 2011 Posted August 26, 2011 As long as the vpn allows passing ssh port or rsync server port, you should be OK. Although I have no experience with using rsync on a windows platform. If you can access the disk shares over the vpn, then you can use a windows program like unison. deltasync, pbsync, robosync or others. Yes I can access disk shares over VPN. I would be comfortable with windows programms like what you have suggested, but do they also worklike rsync in the sense that they only copy the updated/new contents and not the whole file. I havw worked with few windows sync applications but they sopy the whole file when ever sync is done. This is one feature I was impressed with rsync. Can you suggest which windows programm above works like rsync in that way. Regards Sammy
dalben Posted August 26, 2011 Author Posted August 26, 2011 I noticed that in the unRAID roadmap and rsync server is planned for 5.1. With that in mind, I might just use a Windows PC client to manage the replication until that feature pops up.
cyrnel Posted August 28, 2011 Posted August 28, 2011 I've scripted many a windows box backup with cwrsync. http://www.itefix.no/i2/node/10650 It has the inevitable cygwinisms but works quite well, assuming you can define your include/exclude paths & patterns so Windows users don't have a coronary event. I've run it over ssh and across openvpn and set up some crazy auto-recovering batch files for when links die. I haven't found or looked for any windows front-end for specifying files or restorations though it's bound to exist somewhere. If users will need to be closely involved with things I'd put in another vote for Crashplan. Quite nice, and it takes care of all the encryption and remote access questions without any work on your part. Also, check out backuppc. http://backuppc.sourceforge.net/ I started using it for a small LAN before running out of time & switching it to Crashplan. Nicely efficient hybrid solution for mixed environments. Provides users a web interface to control their parts.
dalben Posted August 29, 2011 Author Posted August 29, 2011 OK, so I got down and dirty with the CLI and scripting. I think I've managed to set it up so on boot I initiate what I need to initiate, add to the crontab and then watch as it checks for changed files every hour and copies them to a NAS. Wasn't that hard after all. I guess my days of writing batch files in MS-Dos 1.0x came in handy. A slightly related question. My scripts are very basic, they just run the command. In the linux world, what is best practice for error checking etc in scripts ?
sam9s Posted August 29, 2011 Posted August 29, 2011 I use something called Allway Sync Thanks dalben, Allway Sync seems to be a pretty decent application. Its working as I want it to be on LAN. I have not tried it form my office over VPN, but I suppose it should perform in the similar manner as it is over LAN. Thanks again for this recommendation. Regards Sammy
WeeboTech Posted August 29, 2011 Posted August 29, 2011 OK, so I got down and dirty with the CLI and scripting. I think I've managed to set it up so on boot I initiate what I need to initiate, add to the crontab and then watch as it checks for changed files every hour and copies them to a NAS. Wasn't that hard after all. I guess my days of writing batch files in MS-Dos 1.0x came in handy. A slightly related question. My scripts are very basic, they just run the command. In the linux world, what is best practice for error checking etc in scripts ? What's supposed to happen is, cron will email you any output of an automated job. unRAID does not have the support to do that automatically. So you may want to pipe the output of your cron command into syslog so you can review the syslog. add the following to the end of your cron or shell command 2>&1 | logger -tYOURJOBNAMEHERE -puser.info -i on some syslogs you need to use -plocal0.info as user may not exist.
dalben Posted August 30, 2011 Author Posted August 30, 2011 No worries Sammy. I ended up paying for the Pro licence. It's a cool app. Cheers Weeboi, I noticed the errors that it could send email so we'll see if that code you gave me worked when it next runs.
WeeboTech Posted August 30, 2011 Posted August 30, 2011 There is a way to get unRAID to send mail by installing a customized exim and mailx. But that's a bit involved because /var lives on rootfs and exim complains. It requires creating a tmpfs on /var/spool, then a customized configuration of exim. Not all that hard, but I would think it to be a pain to maintain. not sure of ssmtp one of the other smtp simple mailers would work for execution via cron. http://linux.die.net/man/8/ssmtp
dalben Posted September 2, 2011 Author Posted September 2, 2011 Im having a really odd problem. Well, odd to me as I know nothing about linux, so it might be simple. I have added these lines to my crontab: * 0-23/4 * * * /boot/custom/tdm_sync_data 2>&1 | logger -ttdm_sync_data -puser.info -i" >>/tmp/crontab * 2 * * * /boot/custom/tdm_sync_photos 2>&1 | logger -ttdm_sync_photos -puser.info -i" >>/tmp/crontab * 4 * * * /boot/custom/tdm_sync_videos 2>&1 | logger -ttdm_sync_videos -puser.info -i" >>/tmp/crontab * 5 * * * /boot/custom/tdm_sync_music 2>&1 | logger -ttdm_sync_music -puser.info -i" >>/tmp/crontab * 6 * * * /boot/custom/tdm_sync_source 2>&1 | logger -ttdm_sync_source -puser.info -i" >>/tmp/crontab Basically running most of the replication scripts once a day, except for data which replicates every 4 hours. Is there anything wrong there ? Using Data as an example, here is my replication script: # !/bin/bash # rsync data echo Syncronising Data - Start ******************************************** rsync --password-file=/mnt/cache/.apps/rsync/.rsyncpwd -vrHltD --delete --stats /mnt/user/data/ [email protected]::data echo Syncronising Data - Complete ******************************************** Anything wrong there ? The problem is that the script keeps restarting every minute ! I restarted the server and it sat there for a few hours before it ran for the first time. Since then the script runs every minute. Here's a dump of the syslog: Sep 2 16:15:01 tdm tdm_sync_data[26546]: Syncronising Data - Start initconfig mdcmd mkmbr powerdown samba (Drive related) Sep 2 16:15:01 tdm tdm_sync_data[26546]: sending incremental file list Sep 2 16:15:01 tdm tdm_sync_data[26546]: Sep 2 16:15:01 tdm tdm_sync_data[26546]: Number of files: 4659 Sep 2 16:15:01 tdm tdm_sync_data[26546]: Number of files transferred: 0 Sep 2 16:15:01 tdm tdm_sync_data[26546]: Total file size: 4619791505 bytes Sep 2 16:15:01 tdm tdm_sync_data[26546]: Total transferred file size: 0 bytes Sep 2 16:15:01 tdm tdm_sync_data[26546]: Literal data: 0 bytes Sep 2 16:15:01 tdm tdm_sync_data[26546]: Matched data: 0 bytes Sep 2 16:15:01 tdm tdm_sync_data[26546]: File list size: 128387 Sep 2 16:15:01 tdm tdm_sync_data[26546]: File list generation time: 0.001 seconds Sep 2 16:15:01 tdm tdm_sync_data[26546]: File list transfer time: 0.000 seconds Sep 2 16:15:01 tdm tdm_sync_data[26546]: Total bytes sent: 128784 Sep 2 16:15:01 tdm tdm_sync_data[26546]: Total bytes received: 392 Sep 2 16:15:01 tdm tdm_sync_data[26546]: Sep 2 16:15:01 tdm tdm_sync_data[26546]: sent 128784 bytes received 392 bytes 258352.00 bytes/sec Sep 2 16:15:01 tdm tdm_sync_data[26546]: total size is 4619791505 speedup is 35763.54 Sep 2 16:15:01 tdm tdm_sync_data[26546]: Syncronising Data - Complete initconfig mdcmd mkmbr powerdown samba (Drive related) Sep 2 16:16:01 tdm tdm_sync_data[26842]: Syncronising Data - Start initconfig mdcmd mkmbr powerdown samba (Drive related) Sep 2 16:16:02 tdm tdm_sync_data[26842]: sending incremental file list Sep 2 16:16:02 tdm tdm_sync_data[26842]: Sep 2 16:16:02 tdm tdm_sync_data[26842]: Number of files: 4659 Sep 2 16:16:02 tdm tdm_sync_data[26842]: Number of files transferred: 0 Sep 2 16:16:02 tdm tdm_sync_data[26842]: Total file size: 4619791505 bytes Sep 2 16:16:02 tdm tdm_sync_data[26842]: Total transferred file size: 0 bytes Sep 2 16:16:02 tdm tdm_sync_data[26842]: Literal data: 0 bytes Sep 2 16:16:02 tdm tdm_sync_data[26842]: Matched data: 0 bytes Sep 2 16:16:02 tdm tdm_sync_data[26842]: File list size: 128387 Sep 2 16:16:02 tdm tdm_sync_data[26842]: File list generation time: 0.001 seconds Sep 2 16:16:02 tdm tdm_sync_data[26842]: File list transfer time: 0.000 seconds Sep 2 16:16:02 tdm tdm_sync_data[26842]: Total bytes sent: 128784 Sep 2 16:16:02 tdm tdm_sync_data[26842]: Total bytes received: 392 Sep 2 16:16:02 tdm tdm_sync_data[26842]: Sep 2 16:16:02 tdm tdm_sync_data[26842]: sent 128784 bytes received 392 bytes 86117.33 bytes/sec Sep 2 16:16:02 tdm tdm_sync_data[26842]: total size is 4619791505 speedup is 35763.54 Sep 2 16:16:02 tdm tdm_sync_data[26842]: Syncronising Data - Complete initconfig mdcmd mkmbr powerdown samba (Drive related) Sep 2 16:17:01 tdm tdm_sync_data[27138]: Syncronising Data - Start initconfig mdcmd mkmbr powerdown samba (Drive related) Sep 2 16:17:02 tdm tdm_sync_data[27138]: sending incremental file list Sep 2 16:17:02 tdm tdm_sync_data[27138]: Sep 2 16:17:02 tdm tdm_sync_data[27138]: Number of files: 4659 Sep 2 16:17:02 tdm tdm_sync_data[27138]: Number of files transferred: 0 Sep 2 16:17:02 tdm tdm_sync_data[27138]: Total file size: 4619791505 bytes Sep 2 16:17:02 tdm tdm_sync_data[27138]: Total transferred file size: 0 bytes Sep 2 16:17:02 tdm tdm_sync_data[27138]: Literal data: 0 bytes Sep 2 16:17:02 tdm tdm_sync_data[27138]: Matched data: 0 bytes Sep 2 16:17:02 tdm tdm_sync_data[27138]: File list size: 128387 Sep 2 16:17:02 tdm tdm_sync_data[27138]: File list generation time: 0.001 seconds Sep 2 16:17:02 tdm tdm_sync_data[27138]: File list transfer time: 0.000 seconds Sep 2 16:17:02 tdm tdm_sync_data[27138]: Total bytes sent: 128784 Sep 2 16:17:02 tdm tdm_sync_data[27138]: Total bytes received: 392 Sep 2 16:17:02 tdm tdm_sync_data[27138]: Sep 2 16:17:02 tdm tdm_sync_data[27138]: sent 128784 bytes received 392 bytes 86117.33 bytes/sec Sep 2 16:17:02 tdm tdm_sync_data[27138]: total size is 4619791505 speedup is 35763.54 Sep 2 16:17:02 tdm tdm_sync_data[27138]: Syncronising Data - Complete initconfig mdcmd mkmbr powerdown samba (Drive related) This first run in this sequence was at 16:00:01. I've managed to stop it running by renaming the script. The syslog tells me it went looking for it but failed. Sep 2 16:19:01 tdm tdm_sync_data[27754]: /bin/sh: /boot/custom/tdm_sync_data: No such file or directory That seems to point to crontab triggering the every minute script run. Can anyone see why ? Or tell where else I can look ? I need to solve this before all the nightly jobs kick off.
dalben Posted September 2, 2011 Author Posted September 2, 2011 Ah, yes. * doesn't mean ignore that value, it means every one. So * in the minute column had my script trying to run every minute of that hour. Sorry for wasting your time if anyone read that post above. I'll keep it there in case another n00b makes the same silly mistake.
WeeboTech Posted September 2, 2011 Posted September 2, 2011 Also * 0-23/4 * * * /boot/custom/tdm_sync_data 2>&1 | logger -ttdm_sync_data -puser.info -i" >>/tmp/crontab the >> /tmp/crontab is not needed because all data is fed to logger. if you add -s then the output of logger would also go to stdout which would be duplicated to >>/tmp/crontab. What I'm finding odd is the stray " character.
dalben Posted September 3, 2011 Author Posted September 3, 2011 Also * 0-23/4 * * * /boot/custom/tdm_sync_data 2>&1 | logger -ttdm_sync_data -puser.info -i" >>/tmp/crontab the >> /tmp/crontab is not needed because all data is fed to logger. if you add -s then the output of logger would also go to stdout which would be duplicated to >>/tmp/crontab. What I'm finding odd is the stray " character. Thanks for that. I've cleaned it up a bit now and it works as expected.
dalben Posted September 8, 2011 Author Posted September 8, 2011 I now use the --delete switch but having a think about it, I was wondering if there is a way to have the --delete based on timestamps. i.e. --delete any files on the destination older than 30days that are no longer on the source. This would ensure any inadvertent delete wasn't replicated for x days giving me a chance to realise and recover. Also, I am now getting an email after every cron run. Is there any way of suppressing it from command line ?
Recommended Posts
Archived
This topic is now archived and is closed to further replies.