Need a simple off-site unraid-unraid incremental backup solution

[IamSpartacus]

I have my main Unraid server backup to an offsite (my parents house) Unraid server across a 1Gbps Site-to-Site VPN connection every night currently using SyncBack in a Windows VM.  It just mirrors any changes I've made on my main Unraid server each night.

 

Every so often I look for alternative options as I don't love having to have a VM on for nightly backups.  I looked into rsync via SSH and different docker containers but none of the options have been appealing as of yet.

[lotetreemedia]

Still maintain you have some options avail.

 

1. Setup a connection between host and target - You can do this with 2 routers and create a VPN. So that they are addressable to eachother use DuckDNS containers on both machines to have DynamicDNS. I can help you set this up using 2 x USGs
2. Mount your fileshare using Unassigned devices
3. Use a backup program like CloudBerry / Duplicacy to backup to the mounted share. They both support encryption. Duplicacy in particular supports snapshots as well. There's a Docker container for it and I'm happy to do a guide for you.
4. ???
5. Profit

Any reasons this won't work?

[tr0910]

On 2/20/2019 at 11:36 PM, yusuflimz said:

Still maintain you have some options avail.

 

1. Setup a connection between host and target - You can do this with 2 routers and create a VPN. So that they are addressable to eachother use DuckDNS containers on both machines to have DynamicDNS. I can help you set this up using 2 x USGs
2. Mount your fileshare using Unassigned devices
3. Use a backup program like CloudBerry / Duplicacy to backup to the mounted share. They both support encryption. Duplicacy in particular supports snapshots as well. There's a Docker container for it and I'm happy to do a guide for you.
4. 
5. Profit

Any reasons this won't work?

Cloudberry is an interesting option and would handle the encryption at destination as desired.  Rsync is great, but the encryption requirement will be difficult to implement.  I wonder how Cloudberry can scale? 

 

You seem interested in this, test it out and tell us how it works? 

[maxse]

So I think @yusuflimz way of just having 2 hardware devices for VPN would  be best as that would require the least amount of tinkering. But it seems pricey, about $150 for each VPN Device so $300 total just for VPN. I looked into a raspberry pi but it seems it would be too slow to process encryption with people getting speed uner 10MBPs. I have 100MBps upload speed so would like to utilize that as much as possible. 

 

Anyone know if there are other less expensive devices to have the VPN just run on a hardware device?

Are there any issues in terms of the ISP, or just best practices in general, with having a VPN always on and always connected to each other?

 

So then the flow would be: VPN, mount remote with unassigned devices, then cloudberry pointing to the unassigned devices share for encrypted incremental backups, then btrfs running with snapshots on the remote unraid to protect against crypto?

[lotetreemedia]

2 hours ago, maxse said:

So I think @yusuflimz way of just having 2 hardware devices for VPN would  be best as that would require the least amount of tinkering. But it seems pricey, about $150 for each VPN Device so $300 total just for VPN. I looked into a raspberry pi but it seems it would be too slow to process encryption with people getting speed uner 10MBPs. I have 100MBps upload speed so would like to utilize that as much as possible. 

2

 

All i'm going to say is that you get for what you pay for  I almost always install Ubiquiti for clients and reliability alone makes it worth it. without seriously testing a solution i can't put my name to it. I've set up lower costs VPNs before and they are a headache to maintain. 

 

Even the cheapest pfSense boxes by netgate are around that price.

 

lowest cost might be putting some 4 port NICs (use Intel only) in your unRAID boxes. Setting up pfSense on both of them using docker containers and treat your unRAID boxes as your routers as well. Gonna be helluva learning curve but gonna be a reliable lower cost project. 

Risks will be that you're putting all your eggs in one basket and if you unRAID box / container goes down your entire network won't work. Troubleshooting as well if you haven't worked with pfSense might be an issue. 

 

In terms of ISP some do block VPN connections. Here in the UK Virgin Media consumer do by default and you have to enable it in the modem settings. 

 

In terms of best practice just don't use PPTP as your VPN method. Stick to L2TP over IPSec. PPTP will send your password as plain text and anyone will be able to capture your traffic and see the contents. 

 

EDIT: creating a VPN in UniFi takes around 3 seconds once your sites are configured. Just needs a name, selecting site-to-site-vpn, choosing the site and clicking save.  

 

 

 

Edited February 22, 2019 by yusuflimz

[maxse]

okay thanks so much. Looks like unifi is the answer. Is it going to be something that's always enabled? Both t he VPNs always connected to each other? I would like the process to be automated, so I doubt there's a way for the VPN to know once the backup is complete... So no issues with it always being connected?

 

Gonna have to save up for this I guess, together with cloudberry it will be about a $450 investment not including building the actual 2nd server... May have to figure out a way if cloudberry can back up incrementally to multiple external hard drives, and just back up what's been updated in the meantime

 

Thank you!

Edited February 23, 2019 by maxse

[lotetreemedia]

2 minutes ago, maxse said:

okay thanks so much. Looks like unifi is the answer. Is it going to be something that's always enabled? Both t he VPNs always connected to each other? I would like the process to be automated, so I doubt there's a way for the VPN to know once the backup is complete... So no issues with it always being connected?

 

Thank you!

 

There's no issue with the tunnel staying open. It will always be on. If there's no traffic being routed through it it's just not in use. Personally, I think having it always on means there's one less thing to be broken. 

 

If you wanted the process to be Create a VPN connection > Do the Backup > Close the VPN connection.

 

There's 2 things there that might cause problems. If there's a problem creating the VPN connection, what should the system do? Retry? How many times? Should the backup be delayed to cater for that?

 

If there's a problem closing the connection will it cause a problem when the process happens the next time when it needs to create the VPN connection? I know I'm being a little hypothetical here but I'd be thinking about these things.

 

If the VPN is on all the time you can mitigate those risks. I don't see it being a problem at all.

 

Also I'd love to here what other members think. Good to get a second opinion. 

 

 

[maxse]

sounds good, thanks! What does everyone else think? BTW, no issues setting up the unifi VPN behind an existing router correct? The unifi could act just as a VPN behind the main router? 

 

Wasn't sure if everyone on here also leave there VPN always connected. What does everyone else think?

[lotetreemedia]

sounds good, thanks! What does everyone else think? BTW, no issues setting up the unifi VPN behind an existing router correct? The unifi could act just as a VPN behind the main router? 
 
Wasn't sure if everyone on here also leave there VPN always connected. What does everyone else think?

Hmm now sure about having it behind another router. I imagine your going to have some double NAT issues and it’s not going to work. Why not just use it as your main router?


Sent from my iPhone using Tapatalk

[maxse]

Oh no. Because I'm gonna place the remote at a friend's place and there's no way they would let me mess with their home networking, nor would I want to in case something stops working on their end...

[maxse]

Wow, I'm just shocker that there's no app that basically presents a remote unraid server like cloud storage. So you could just do the same thing that people do with say rclone where they encrypt and upload to the cloud, except "the cloud" is now your own hardware...

[JonathanM]

11 minutes ago, maxse said:

Wow, I'm just shocker that there's no app that basically presents a remote unraid server like cloud storage. So you could just do the same thing that people do with say rclone where they encrypt and upload to the cloud, except "the cloud" is now your own hardware...

That's exactly what I was suggesting earlier in this thread.

 

[hugenbdd]

20 minutes ago, maxse said:

Wow, I'm just shocker that there's no app that basically presents a remote unraid server like cloud storage. So you could just do the same thing that people do with say rclone where they encrypt and upload to the cloud, except "the cloud" is now your own hardware...

Can your backup location open port 21 for you?

 

rclone accepts an ftp as a target.  rclone would encrypt the files for you and you could send an rclone copy command with the ignore-existing switch so if you got hit with a  virus the nightly updates don't overwrite your old data.

 

This doesn't satisfy your versioning requirement though.

[IamSpartacus]

No matter what option you choose, using a site to site VPN makes everything easier.  Here is a basic diagram of my both my server networks connected via VPN.

 

 

Setting up backups or other automation is super easy when your "directly connected" to your servers.

Edited February 23, 2019 by IamSpartacus

[maxse]

okay how about this? Can I set up openvpn as a docker on the remote server, then is there a vpn client docker that I can run on the main unraid server to connect to the remote via the vpn?

 

I could just manually enable it once a week, and manually initiate the backup with cloudberry, while running btrfs with snapshots on the remote? I would be willing to just manually initiate this process in the interest of making things easier... Pfsense is out, just no time to learn all that and I wouldn't be able to change the network system at a friend's house. They don't have a problem with me forwarding a specific port to the unraid server...

 

@jonathanm I read that thread, seemed a bit complicated to set up with letsencrypt, etc... and then got side-tracked with the hardware vpn solution which seemed simpler, but turns out wouldn't work behind a second router. I saw you switched to nextcloud...

 

I was planning on using nextcloud eventually anyway on my main server. How is it working out for you? There's a nice video tutorial by spaceinvader one on setting up nextcloud. Do you think I could do that, basically creating my own cloud? I feel like I read somewhere it wasn't stable enough for long file transfers, (several terabytes at a time). Is there a way to connect nextcloud to cloudberry? I read something about you using webdav with duplicati, but I've read more than a few posts of people having database crashes with Duplicati, so I would rather use cloudberry. But now sure if it has that option to connect it to minio? That would seem like exactly what I need

 

*EDIT* 

Also, how secure will it be with letsencrypt vs. setting up a vpn assuming the above is possible with the client vpn on one unraid and the "server" vpn on the remote unraid? I imagine a VPN is safer? When I say safer I mean in terms of exposing unraid to the outside world...

 

Lastly, how does Synology seem to do it? Do they use the equivalent of letsencrypt to make a remote synology accessible to the web?

Edited February 23, 2019 by maxse

[maxse]

Guys, what happens if I just want to do external drives and that's it. Just manually connect the 10tb WD easystores and tranfer over encrypted data... How can I do this? So say the server is 45tb that needs to be backed up, and the drives are 10tb each. Is there a way to have back up software span them across the drives, and then when new data is added for it to only backup the new data yet still allow me to restore to a point in time (in case a crypto gets backed up without me knowing so that I could restore to a point before the virus infected?) or is that also going to be too difficult? I could just manually plug the drive in every week, and move them off-site when the drive fills up. 

 

I just don't see how the software would know what was copied already say on the second drive, and how it would even keep track of the time stamps, etc... Oy..

[maxse]

@tr0910 seems like the SSH method with rsync with btrfs snapshots on the backup server is almost ideal except one big issue. The remote is at a friend's place, and I don't want someone in his house to be able to easily snoop around the backup up server and be able to view the files on it. Is there a way to encrypt the files then? I get that you're saying encryption wont work with rsync on the main server side. But is there something maybe I could then run on the remote backup server standalone? Just grasping at straws here lol

 

Any comments about the nextcloud/letsencrypt solution in terms of security vs. vpn?

And guys seems like cloudberry doesn't support webdav as @jonathanm used with duplicati (read too many posts of crashes with duplicati), any way to get cloudberry working with nextcloud?

 

Seems like cloudberry supports Minio by clicking on S3 compatible cloudstorage and entering the Minio details. Found that on google, is that correct? I could then just set up minio like Joananm suggested? Is that right? Any thoughts guys?

Edited February 23, 2019 by maxse

[lotetreemedia]

Cloudberry supports minio natively I mentioned it up above.

What might work is having an Ubuntu VM on your source machine with CloudBerry + OpenVPN client.

On your target machine have another VM with OpenVPN server + Minio.




Sent from my iPhone using Tapatalk

[JonathanM]

10 hours ago, maxse said:

Seems like cloudberry supports Minio by clicking on S3 compatible cloudstorage and entering the Minio details. Found that on google, is that correct?

No clue on if it actually works, but it would be easy enough to try. The target machine would just need to have the Minio docker installed, and either a static IP or some flavour of dynamic dns to allow your machine to find it. No need to have a vpn or run a VM. You will need to set up a forwarding rule in the remote router to allow the single outside port to go through to the minio port on the remote server.

[lotetreemedia]

9 minutes ago, jonathanm said:

No clue on if it actually works, but it would be easy enough to try. The target machine would just need to have the Minio docker installed, and either a static IP or some flavour of dynamic dns to allow your machine to find it. No need to have a vpn or run a VM. You will need to set up a forwarding rule in the remote router to allow the single outside port to go through to the minio port on the remote server.

Ah yes this might work actually

 

[maxse]

Niceeeee, wow you guys are too good, haha with diagrams and all! This is it, will try this.

Will I be able to try this on my own local network first to see if this will work before bring the backup server into its remote location?

 

Ahhhh, I just came across a post that said cloudberry on linux does NOT obfuscate the file names when it encrypts! So basically whoever browses to server will be able to read the filenames and know what's on in it  

Damn it, can anyone confirm this?

[lotetreemedia]

Niceeeee, wow you guys are too good, haha with diagrams and all! This is it, will try this.
Will I be able to try this on my own local network first to see if this will work before bring the backup server into its remote location?
 
Ahhhh, I just came across a post that said cloudberry on linux does NOT obfuscate the file names when it encrypts! So basically whoever browses to server will be able to read the filenames and know what's on in it  
Damn it, can anyone confirm this?

Define browse . GUI ? Via a share? Taking the drive out and plugging it In somewhere?


Sent from my iPhone using Tapatalk

[maxse]

I mean it's going to be connected to the network at a friend's house. So anyone in that home will be able to browse to the server on the market and just see the file names.

 

I know they won't be able to just plug the drive in somewhere else because the drives will be encrypted also, but it's going to be always on and the dirve is unlocked when the array is running, just the cloudberry encryption, which doesn't obfuscate the file name in the linux version of the software?

[lotetreemedia]

1 hour ago, maxse said:

I mean it's going to be connected to the network at a friend's house. So anyone in that home will be able to browse to the server on the market and just see the file names.

 

I know they won't be able to just plug the drive in somewhere else because the drives will be encrypted also, but it's going to be always on and the dirve is unlocked when the array is running, just the cloudberry encryption, which doesn't obfuscate the file name in the linux version of the software?

 

Am i missing something? If you mark the share as private and don't Export it how will they be able to access your files? 

 

Here's me creating a share 

1. Name =  maxse 
2. Export = no 
3. Security = Private

 

 

If they browse to your server and you do have some Exported shares they will see this. Not much they can do with that unless you name one of your shares "XxX HardCore...." then they might have some suspicion that your storing some non PG stuff.

 

 

 

If by some miracle they figure out your share name and try to browse to it .

 

 

 

If you had your share as Export = yes (hidden) They would still get a prompt asking for credentials.

 

 

And besides, you don't have to create a share anyway for Minio. just manually create a folder under /mnt/user and pass that to the container as the location to store the data.

 

 

[maxse]

Oh mannn @yusuflimz you are THE man! I didn't know why I didnt think of that! I've always had unraid running stock, never bothered to create a user or password or anything like that. Completely forgot that I could set it up that way! Thank you soooo much for showing me everything with the pictures, etc... awesome!

 

Would I be able to try this out to see if it works with both of the servers on my own network? I think it should still work if minio is set up right?

 

And also, in terms of security and letsencrypt/ngenx and forwarding one port to the server, is that still secure? 

I set up ombi at one point when I was playing around with it, but actually took it down because people said it's not good to open ports to unraid, etc... Not sure if that also applies to this set up, etc... I'm not exactly sure how the reverse proxies work and if it's considered still secure to do it this way?

 

Thank you soooo much!!!