March 9, 201511 yr Ever since I built my UnRAID server a little under 2 years ago I've been wanting to build a backup server for it to ensure my media is safe in case of a catastrophic failure. I've finally gotten around to doing so and am looking for suggestions on how best to go about it. I plan to build a second UnRAID server, use rsync locally to backup the first server to the second, and then I will place the second server at my vacation home. So my real question is, once I have my second server situated in a different location, what is the best way to backup my main server to the server at my second home? Internet speed isn't much of a problem as I have 75Mbps upload from where my main server is located. I'm just not sure what's the simplest way to go about setting up automatic (weekly or monthly) backups between two servers connected via the Internet.
March 9, 201511 yr Ever since I built my UnRAID server a little under 2 years ago I've been wanting to build a backup server for it to ensure my media is safe in case of a catastrophic failure. I've finally gotten around to doing so and am looking for suggestions on how best to go about it. I plan to build a second UnRAID server, use rsync locally to backup the first server to the second, and then I will place the second server at my vacation home. So my real question is, once I have my second server situated in a different location, what is the best way to backup my main server to the server at my second home? Internet speed isn't much of a problem as I have 75Mbps upload from where my main server is located. I'm just not sure what's the simplest way to go about setting up automatic (weekly or monthly) backups between two servers connected via the Internet. Manually backing up your server to a remote location with rsync isn't to difficult. You can do it with something like "rsync -av \source user@address:\Dest" the problem is that this command will prompt you for your password, and as such doesn't really work well for automatic backups. So one way people get around this is using ssl key athentication which uses your key instead of a password to athenticate. I think this guide works for unRAID as well.
March 9, 201511 yr If money isn't an issue a VPN switch or router could make this easier. You can do rsync to an rsync server over the VPN. The other choice is rsync over ssh provided that you've set up key exchanges and set up the routers with very strict tunneling. Exposing the ssh server to the whole internet is fraught with peril, so good filtering is imperative.
March 9, 201511 yr Author If money isn't an issue a VPN switch or router could make this easier. You can do rsync to an rsync server over the VPN. The other choice is rsync over ssh provided that you've set up key exchanges and set up the routers with very strict tunneling. Exposing the ssh server to the whole internet is fraught with peril, so good filtering is imperative. Ideally I'd like to make these backups and secure and private as possible but don't have a lot of experience with setting up VPN's. I do have DD-WRT routers with VPN support at both locations though, just never fooled around with setting up a VPN on them.
March 9, 201511 yr I do have DD-WRT routers with VPN support at both locations though, just never fooled around with setting up a VPN on them. That would work, or there is an OpenVPN plugin you can install on your unRAID server to get the same effect. I personally use my router so I cannot say how the plugin works but it seems like a large amount of users like it.
March 9, 201511 yr If you can set up the VPN between homes it makes it allot easier for remote support of the server. You can then go rsync client to an rsync server with disk -> disk syncing.
March 9, 201511 yr Author I do have DD-WRT routers with VPN support at both locations though, just never fooled around with setting up a VPN on them. That would work, or there is an OpenVPN plugin you can install on your unRAID server to get the same effect. I personally use my router so I cannot say how the plugin works but it seems like a large amount of users like it. Hmmm, I will definitely look into the plugin if configuring my router seems too complicated. Thanks for the info. If you can set up the VPN between homes it makes it allot easier for remote support of the server. You can then go rsync client to an rsync server with disk -> disk syncing. That sounds pretty ideal. Once I get my VPN setup I'll be back to inquire about the best way to set that up. Thanks!
March 10, 201511 yr If you can get it set up on your router, it's set it and forget it. If you do it via plugin, or docker, you may be maintaining it when there is an unRAID upgrade. I'm not sure if the open vpn software on the unRAID software will allow access to the webGui, it may be feasible, I just don't know at this point. Yet I'm pretty sure you'll be able to do it if the router supports the vpn.
March 10, 201511 yr If you use an VPN to route your data (which I agree is a good approach to this problem) do you need turn off / turn on the vpn prior to or after the rsycn daemon runs? For people using openvpn with there unRAID set ups do you only route some traffic or all traffic through your VPN? I have an VPN server running on a raspberry pi model b which is sitting on the same LAN as my unRAID. I mostly use this to have secure connections to the web with mobile devices when I'm traveling or at public hotspots, so I turn on and off my connection to the VPN often. Is there a way to automatically do that in a script, or would I just have to have the secondary unRAID sitting behind my VPN all the time?
March 10, 201511 yr Author Hmmm I'm a little confused as to what may be the best way to setup the VPN connection for my needs. Let me give you a little run down on how my network is currently configured in regards to both my locations. Location #1 is basically my "datacenter." It houses my VMware ESXi server running multiple VM's (Plex Media Server, Torrent server, few Windows test VMs, etc.) and my UnRAID 5.04 Plus storage server. Location #2 is where the backup server will live. Both locations are behind DD-WRT routers running VPN builds and the connection between each location is 75Mbps (both ways). So I guess my question is, which end should I setup as the VPN server to allow for the most seamless secure backups? Also, I have no experience using rsync so are there any specific setup guides I should check out that will apply to my situation?
March 10, 201511 yr Put the vpn server at the location where you live. The other end will be the client. The client should always try to reconnect to the server. If you have any issues, it easier to stop/start the vpn server if you are there. I was never able to get the WRT implementation of openvpn to work on my RT-N16's. If you are going to start doing backups, you may need to implement QoS on your routers. Be aware that on most home routers, when you turn on the Qos code, all routing will be in software. The hardware acceleration will turn off. This requires some of the newer hardware so that it has enough CPU grunt to do the vpn and QoS. I believe the new Netgear routers with Adaptive/automatic QoS have implemented FQ-CoDel. I am not sure that the Asus have implemented it. What router hardware are you running.
March 10, 201511 yr Author Put the vpn server at the location where you live. The other end will be the client. The client should always try to reconnect to the server. If you have any issues, it easier to stop/start the vpn server if you are there. I was never able to get the WRT implementation of openvpn to work on my RT-N16's. If you are going to start doing backups, you may need to implement QoS on your routers. Be aware that on most home routers, when you turn on the Qos code, all routing will be in software. The hardware acceleration will turn off. This requires some of the newer hardware so that it has enough CPU grunt to do the vpn and QoS. I believe the new Netgear routers with Adaptive/automatic QoS have implemented FQ-CoDel. I am not sure that the Asus have implemented it. What router hardware are you running. I live at Location #2. There I've got a Netgear R7000 which is pretty beefy so I imagine it can handle it. What kind of issues did you run into trying to get your VPN setup on DD-WRT?
March 10, 201511 yr I had a Asus RT-N16. DD-WRT seems to drop a few firmware loads supporting a specific router and then it is on to the next. They had not fully implemented open VPN in the load that was compatible with my router. Any later loads that did implement it were unstable. I finally went to Tomato - Shibby Build and ran it for ~2 years with no issues with open vpn and QoS enabled. I just recently upgraded to a pfSence solution due to TWC implementing a faster download speed than my old router could handle.
March 11, 201511 yr Hmmm I'm a little confused as to what may be the best way to setup the VPN connection for my needs. Let me give you a little run down on how my network is currently configured in regards to both my locations. Location #1 is basically my "datacenter." It houses my VMware ESXi server running multiple VM's (Plex Media Server, Torrent server, few Windows test VMs, etc.) and my UnRAID 5.04 Plus storage server. Location #2 is where the backup server will live. Both locations are behind DD-WRT routers running VPN builds and the connection between each location is 75Mbps (both ways). So I guess my question is, which end should I setup as the VPN server to allow for the most seamless secure backups? Also, I have no experience using rsync so are there any specific setup guides I should check out that will apply to my situation? I think in your situation you'll want to run rsync as a daemon. Which is something I'm not as familar with. Perhaps the rsync documentation or another user can help you figure that part out. The options that you want to specify with your rsync will vary based on what you want it to do, and what data you want it to copy. For example if you want Rsync to perserve extended attributes you include the -X option. Users here can help you tailor your rsync but I'd suggest you try to solve the vpn issue first.
March 15, 201511 yr Author Alright, I've got my Site-to-Site VPN setup between my two locations as per this guide http://wadihzaatar.com/?p=11. As you can see it's a pretty simple setup. So now I need to determine how best to setup the rsync between my two servers. My backup server is not setup yet (parts are on the way and it will be up by next weekend. Just looking to get things jumpstarted so that when I'm all setup, I know where to start. Do you guys still think running rsync as a daemon is the way to go?
March 15, 201511 yr I'd think that's the case, since you can fully automate the process, but you'll need a "Linux guy" to help with the details. WeeboTech has done a lot in this area, so hopefully he'll provide a bit of additional guidance (if you need it). Since you're setting up the backup server with a one:one correspondence with the disks on your main server, you can simply set up N rsync commands (where N = # of data disks) and do a disk-disk backup without involving the shares. The only real disadvantage of this approach is you need to do matching upgrades on your two servers ... i.e. if a disk fails on your primary server and you upgrade it to a larger disk, you need to do the same on your backup server. [The reverse isn't true -- there's no problem having larger disks on the backup server.]
March 15, 201511 yr This is what I have on each of my unRAID Servers. in /boot/config/go I put this in place from /boot/local/etc/rsyncd.conf to /etc/rsyncd.conf fromdos < /boot/local/etc/rsyncd.conf > /etc/rsyncd.conf chmod 600 /etc/rsyncd.conf uid = root gid = root use chroot = no max connections = 4 timeout = 600 pid file = /var/run/rsyncd.pid log file = /var/log/rsyncd.log socket options = SO_SNDBUF=524288,SO_RCVBUF=524288 [mnt] path = /mnt comment = /mnt files read only = FALSE list = yes [boot] path = /boot comment = /boot files read only = FALSE list = yes I install rsync as a inet service with this in a shell or in /boot/config/go. if ! grep -q ^rsync /etc/inetd.conf ; then fromdos <<-EOF_INETD >> /etc/inetd.conf rsync stream tcp nowait root /usr/sbin/tcpd /usr/bin/rsync --daemon EOF_INETD kill -1 $(</var/run/inetd.pid) fi I have much more advanced examples, but this is the simplest I could come up with. you can also have passwords on the rsync port as well as using tcp wrapper to lock it down more.
March 15, 201511 yr Author This is what I have on each of my unRAID Servers. in /boot/config/go I put this in place from /boot/local/etc/rsyncd.conf to /etc/rsyncd.conf fromdos < /boot/local/etc/rsyncd.conf > /etc/rsyncd.conf chmod 600 /etc/rsyncd.conf uid = root gid = root use chroot = no max connections = 4 timeout = 600 pid file = /var/run/rsyncd.pid log file = /var/log/rsyncd.log socket options = SO_SNDBUF=524288,SO_RCVBUF=524288 [mnt] path = /mnt comment = /mnt files read only = FALSE list = yes [boot] path = /boot comment = /boot files read only = FALSE list = yes I install rsync as a inet service with this in a shell or in /boot/config/go. if ! grep -q ^rsync /etc/inetd.conf ; then fromdos <<-EOF_INETD >> /etc/inetd.conf rsync stream tcp nowait root /usr/sbin/tcpd /usr/bin/rsync --daemon EOF_INETD kill -1 $(</var/run/inetd.pid) fi I have much more advanced examples, but this is the simplest I could come up with. you can also have passwords on the rsync port as well as using tcp wrapper to lock it down more. Ok I'm a little confused here. I have ZERO experience with Linux so this is all pretty foreign to me. My UnRAID setup is very basic and vanilla. In my "go" file all I have is the following: #!/bin/bash # Start the Management Utility /usr/local/sbin/emhttp & /boot/unmenu/uu So I'm a confused as to what I'm putting where.
March 15, 201511 yr Maybe I'll have to devise a package or plugin for the most basic of all rsync services. Really this should just be part of unRAID itself. Give me a lil time to think about it.
March 15, 201511 yr Author Maybe I'll have to devise a package or plugin for the most basic of all rsync services. Really this should just be part of unRAID itself. Give me a lil time to think about it. Sure thing man, I really appreciate the help. Wish I knew more about linux.
March 15, 201511 yr Author Looks like I'm going to need to upgrade my Linksys E4200 router as it seems that's the bottleneck in regards to the speed of my OpenVPN connection between the two sites. I'm currently only getting 10Mbps. Might need to buy another Netgear R7000 so I have one on both ends.
March 19, 201511 yr Author Should have my server up within the next few days (just waiting on the motherboard). Are there any good wiki's out there for me to read that teach a Linux newb about rsync?
April 27, 201511 yr This conversation bears weight on a similar project I have underway. I have a OpenVPN DD-WRT server running now and intend to rsync select folders like the ones me and my family are using for Plex across unraid systems. The unraid client will use the OpenVPN client plugin and just NFS mount the needed shares across the VPN. If I can NFS mount both ways I may install cron jobs on each side, otherwise I may depend on the unraid client to sync both ways that has the NFS mount to my server. I think I should be able to NFS mount the client to my server however the trouble is I do not know what IP address my DD-WRT server will assign to it, which case the cron jobs will be entirely on the client. I don't think there is any need to run a rysnc damon if I have this set as a NFS mount as well as any SSH certs. Plex should see the changes and update the libraries as needed on both ends so either family member can add movies to the library and be apart of their own library which has a few advantages over just sharing your plex server. I would do a DD-WRT point to point however I do not control the router on the other end and just manager the server for my family. Some of my concerns did include QoS. I was planing to run rysnc out of cron at night and run a rsync kill script out of cron in the morning in off hours to address the bandwidth issues. I'm not sure how rysync will behave being killed off but theoretically the libraries should sync eventually. I can share the rsync cron job and kill script I write if I go that route. I was thinking of running a NFS mount command to try and remount any stale NFS mounts then something like /usr/bin/rsync -avrtzH /user/movies/movieshare /mnt/remotesystemdir/movies. I have the compression switch in there too. I may end up manually transferring the files by an external HDD if the first sync is just that nasty. Any suggestions or problems with this setup, I am all ears. I also hope this helps with your unraid sync ideas. I go over to setup the OpenVPN client and NFS mount to my system this weekend. Looking forward to it.
April 28, 201511 yr Author This conversation bears weight on a similar project I have underway. I have a OpenVPN DD-WRT server running now and intend to rsync select folders like the ones me and my family are using for Plex across unraid systems. The unraid client will use the OpenVPN client plugin and just NFS mount the needed shares across the VPN. If I can NFS mount both ways I may install cron jobs on each side, otherwise I may depend on the unraid client to sync both ways that has the NFS mount to my server. I think I should be able to NFS mount the client to my server however the trouble is I do not know what IP address my DD-WRT server will assign to it, which case the cron jobs will be entirely on the client. I don't think there is any need to run a rysnc damon if I have this set as a NFS mount as well as any SSH certs. Plex should see the changes and update the libraries as needed on both ends so either family member can add movies to the library and be apart of their own library which has a few advantages over just sharing your plex server. I would do a DD-WRT point to point however I do not control the router on the other end and just manager the server for my family. Some of my concerns did include QoS. I was planing to run rysnc out of cron at night and run a rsync kill script out of cron in the morning in off hours to address the bandwidth issues. I'm not sure how rysync will behave being killed off but theoretically the libraries should sync eventually. I can share the rsync cron job and kill script I write if I go that route. I was thinking of running a NFS mount command to try and remount any stale NFS mounts then something like /usr/bin/rsync -avrtzH /user/movies/movieshare /mnt/remotesystemdir/movies. I have the compression switch in there too. I may end up manually transferring the files by an external HDD if the first sync is just that nasty. Any suggestions or problems with this setup, I am all ears. I also hope this helps with your unraid sync ideas. I go over to setup the OpenVPN client and NFS mount to my system this weekend. Looking forward to it. Unfortunately the way I decided to do my setup has little in common with what you're planning. First off I'm using pfSense on both ends for a site-to-site VPN. I am still using DD-WRT as an OpenVPN client on one end right now but that will be changed on Wednesday when I configure pfSense in a VM to replace it. As for backups, since I already have a Windows VM that I use for backup jobs (Veeam for VMware backups), I decided (with the advice of garycase) to go with the program SyncBack to mirror my shares. It's free, very easy to use and has a good amount of options for creating jobs and groups of jobs. And with my backups being initiated on a Windows box, I'm using Windows shares and not NFS.
Archived
This topic is now archived and is closed to further replies.