Help - Backing Up One Unraid Server to Another Remotely

[Cpt. Chaz]

Hi guys,

 

Basically what i'd like to do is backup my unraid office server to my unraid home server. Not to be used on a regular basis, just as an offsite backup in the event of some kind of "act of god" scenario is all.

 

Wondering what would be the best way to do this? i currently have vpn's set up between both locations, but would prefer something a little more automated. Seems like FTP may be a good way to go? Although i have zero experience in that realm, so would need a good guide or someone to help get me started if possible.

 

Also open to any other suggestions, main objective here is to simply backup digital office files not less than once a week, automated, and remotely. 

[Hoopster]

1 hour ago, Cpt. Chaz said:

Also open to any other suggestions,

Completely automated in my case via User Scripts and it happens once a week.  Can be bi-directional although I only back up one server to the other.  Since my backup server also supports IMPI, the script powers up the server before the backup and powers it down when it completes.

[Hoopster]

And here is a thread with other suggestions:

 

 

[Cpt. Chaz]

Thanks for the replies hoopster! I’m going to give your first suggestion a good look tomorrow (work was crazy today) and I may be back with some questions. 
 

thanks again, talk soon

[Cpt. Chaz]

The end result for your first suggestion looks pretty much like what i'm looking for. But the steps to get there may be out of my league.

 

following along in the first step, the first rsync cl prompt didn't work for me 😂 then trying to figure out if 'rsync' and 'ssh' should be in thisuser's path (use "which ssh" and "which rsync"), 'rsync' should be in remoteuser's path, and 'sshd' should be running on remotehost completely went over my head. 

 

Also leads to me to think, any sort of troubleshooting i'd encounter with this configuration, i'd be completely in the dark with. IT unfortunately is not my profession, as much as i enjoy it (especially being self employed). Leads me to think i need something a little more user friendly here. 

 

Wish there was, because i hate the monthly fee of a cloud when i have plenty of my own offsite storage, you know?  

[Hoopster]

3 minutes ago, Cpt. Chaz said:

Also leads to me to think, any sort of troubleshooting i'd encounter with this configuration, i'd be completely in the dark with. IT unfortunately is not my profession, as much as i enjoy it (especially being self employed).

Yes, there is a bit of a learning curve.  The hardest part is generating the appropriate SSH keys and ensuring they survive reboots.  After that, it is all 'go' file modifications and the appropriate scripts for your backup scenario.  I am happy to share both of mine with you via PM if you wish

 

It took me a few days to get it all figured out (you'll see my history documented as part of that thread), but those who were more expert were very generous in their responses.  Once I had it nailed down, everything has worked with zero issues for almost two years.  It was truly just a matter of set it and forget it.  Every week the script runs and 1) powers on the backup server, 2) backs up new/modified files in the indicated shares on the main server to the backup server, 3) prepares a summary of everything and emails it to me and, 4) powers off the backup server.

[Cpt. Chaz]

wow, ok let's give it a shot. can you send me a pm to get me started? 

[Hoopster]

43 minutes ago, Cpt. Chaz said:

wow, ok let's give it a shot. can you send me a pm to get me started? 

PM sent.  Note that my source and destination servers are currently on the same LAN.  The OP is backing up from China to the US.  The concepts are the same, you just need a VPN between the servers over a WAN.  I believe the OP's method for doing that is mentioned in the thread.

[Cpt. Chaz]

quick update, i'm still working on initial setup of this - didn't want you to think i fell off after you pm'd the scripts.

 

I'm trying to get this to work across vpn. So here's what i've got: i'm planning to back up "Server 1" to "Server 2". I've also got a mac (Mac 1) on the same lan as "Server 1". Mac 1 is vpn'd to "Server 2".

 

My naive hope had been that Server 1 would be able to access Server 2 via Mac 1's vpn, since Mac 1 and Server 1 share a lan connection. Couldn't SSH in this way, and pinging Server 2 from Server 1 with Mac 1 vpn returned nada. So tonight i'm going to work on getting openvpn client installed on Server 1, so i can directly connect it to Server 2 and try again.

 

unless... i can ssh directly over to Server 2 using my dns address instead of a local ip, but it looks like this requires port 22 to be open. Is it safe to do that? i'm guessing i'd at least need to change unraid's root user password? sure would be nice to not have to rely on a vpn, even though that's not a deal killer

[Hoopster]

6 minutes ago, Cpt. Chaz said:

I'm trying to get this to work across vpn

No problem, don't feel like you owe me any updates.

 

You might want to take a look at the 6.8.0-rc5 currently available for public testing.  It has WireGuard VPN built into unRAID via a plugin.  I am using it now for remote client to server access to my unRAID servers; however, it also supports a server to server access mode.  See the WireGuard Quickstart Guide for more info.  WireGuard is faster and lighter than OpenVPN.

 

It could be a very good solution for you.

[Cpt. Chaz]

how stable is 6.8 right now? i'm running Server 1 in my office and can't afford any downtime at the moment. That does sound like a perfect solution though

 

[Hoopster]

7 minutes ago, Cpt. Chaz said:

how stable is 6.8 right now? i'm running Server 1 in my office and can't afford any downtime at the moment. That does sound like a perfect solution though

 

I've been running it on both my servers since RC1.  Very stable.  I have had zero issues with any of the RCs.  Fortunately, I never had the SQLite DB corruption issues and the RCs have taken care of the simultaneous array write/read issues that plagued 6.7.x releases.  RC5 seems to have solve the DB corruption issues as well.

 

I am on RC5 (the latest) and actually have a much better experience with my server now than I did with 6.7.2.  I think most running the RCs would say the same.

[Cpt. Chaz]

Looks like I’m gonna be up late tonight. 

[Cpt. Chaz]

Hey hoop - i just wanted to say thanks again for your help with all this. i'm going to table this method for the time being - i think given enough time i could get there, but i'm out of my depth here i'm afraid, without just being spoonfed through the entire process. when things slow down and i can dedicate a little more time to trying, i'll look at giving it another shot. just wanted to say thanks again. 

[Hoopster]

4 minutes ago, Cpt. Chaz said:

when things slow down and i can dedicate a little more time to trying, i'll look at giving it another shot. just wanted to say thanks again.

No problem.  Are you hung up on properly generating and testing the SSH keys, or something else?  For me, it was the keys. 

 

I needed to do things exactly opposite to what the OP did, so I had to generate the keys differently.  He initiated the backup from the backup/destination server to pull from the source.  I needed to initiate from the source (where the script runs) to power on and push to the destination (backup) server.  The OPs method also left out a few important steps for me getting it right and Ken-ji jumped in and helped me sort it all out.  That's all documented later in the thread.

[Cpt. Chaz]

30 minutes ago, Hoopster said:

No problem.  Are you hung up on properly generating and testing the SSH keys, or something else?  For me, it was the keys. 

 

I needed to do things exactly opposite to what the OP did, so I had to generate the keys differently.  He initiated the backup from the backup/destination server to pull from the source.  I needed to initiate from the source (where the script runs) to power on and push to the destination (backup) server.  The OPs method also left out a few important steps for me getting it right and Ken-ji jumped in and helped me sort it all out.  That's all documented later in the thread.

that's exactly what it is. i got the remote ssh part working, but then trying to follow along with op's terminal commands was not resulting in the same output for me at all, which was then resulting in me googling almost every single step trying to figure out the delta between my results and his. 

 

but i don't usually skip ahead too much in tutorials, if anything i tend to get bogged down in the details so as not to miss anything. but if there's clarification down the line, maybe i'll give it one last gander!

[Hoopster]

5 hours ago, Cpt. Chaz said:

but i don't usually skip ahead too much in tutorials, if anything i tend to get bogged down in the details so as not to miss anything. but if there's clarification down the line, maybe i'll give it one last gander!

Start with this post of mine on page two of the thread and read through page three.   By the end of page three, everything was working for me thanks to Ken-ji.  Pages four and five are other users jumping in wanting to do this and Ken-ji and me (mostly Ken-ji) helping them get there as well.

Edited November 22, 2019 by Hoopster

[tr0910]

I can still report this working well from USA to China.  No VPN is used once the system is set up and running.  It is all SSH from one server to another.  An always on raspberry PI is used if you need to start a remote server via IPMI. 

 

I hope we get somebody to curate this topic an make it easier to implement.   Thanks to @ken-ji for helping us all.  Finally, it would be great if something like this could be available via the gui as plug and play someday.

[Cpt. Chaz]

Wanted to give a final update, 10 months later... i've become slightly more familiar with basic scripting and linux commands, enough to where i was able to get this implemented and running without any problems. @Hoopster your scripts and go files were key for me to go back and look at. My end result looks quite a bit different, but it was definitely the foundation i needed. Thanks again for the assistance. 

Edited August 14, 2020 by Cpt. Chaz