Syncronize servers using rsync over SSH next door or across the world


Recommended Posts

haha, i think i need a vacation :)  and yes it is a great feeling.  my wife thinks i am an idiot for spending all this time on something like this when i can just drive a hard drive over to my parents house..  i told her that isn't the point :)

 

funny thing is i am watching these youtube videos doing it and they are only like 3 minutes long, i got about 25 hours into it!

 

about all i could add to the book is some coloring :)   i just hope neither system goes down because i would hate to have to do it again.

 

 

Edited by xman111
  • Like 1
  • Upvote 1
Link to comment

hey guys, one last question.   I mentioned this before in this thread but I just did some syncing and i go to delete the files in Windows from the shares and it says something about I don't have the proper permissions.  Is there a way to fix this?  don't want to start syncing files and have all sorts of different permissions attached to different files.

Link to comment

figured this out, was a permission setting on the share on Freenas.

 

I finally moved the Freenas out of the house to my folks house. It worked no problem, just gave the machine a static IP and forwarded the port. The problem is, it is very slow, getting only 1.25 mb/s so a 3.2gb Windows ISO takes about 45 minutes.  Any tips on speeding it up?  oh by the way, i have 20 up and they have 50 down.

 

Thanks for all the help with this!

Edited by xman111
Link to comment
15 minutes ago, xman111 said:

Any tips on speeding it up?

If you can't get good speed with remote SSH, perhaps remote access through ZeroTier is an option with SSH as if it were local.  I have not used ZeroTier myself, but, judging from the responses in the thread users are happy with it.  Not sure how the speed compares to SSH. I have no idea if starting ZeroTier an establishing a connection can be automated.

 

 

Link to comment
26 minutes ago, xman111 said:

figured this out, was a permission setting on the share on Freenas.

 

I finally moved the Freenas out of the house to my folks house. It worked no problem, just gave the machine a static IP and forwarded the port. The problem is, it is very slow, getting only 1.25 mb/s so a 3.2gb Windows ISO takes about 45 minutes.  Any tips on speeding it up?  oh by the way, i have 20 up and they have 50 down.

 

Thanks for all the help with this!

if you have 1.2 MB/s uploading to your folks, you are already using ~12mbps of your 20mbps so it might be normal as internet overheads and other stuff can clog your pipes. It can be better, as I can do about 2.2MB/s given the same situation 20up/50down on the other end.

In my case, The whole connection is wrapped in something similar to Zerotier - a pair of Mikrotik routers running Ethernet over IP with IPSEC to do site to site VPN.

Link to comment
  • 5 weeks later...
1 hour ago, BelgarionNL said:

what do you guys recommend as a good starting point? 

Well, obviously, there is no rsync GUI in unRAID.  The best you can hope to accomplish by following the instructions in this thread is to learn how to write rsync/SSH scripts that automate your backups and, if desired, emails you the results.

 

There is a steep learning curve if your are not already familiar with rsync and SSH, but, once you get this working, it really is a set-it-and-forget it solution.  For me the script has been running unattended for months (automated as a cron via the User Scripts unRAID plugin) and, every week, I receive a summary email that gives me the important stats for each share (some do it by disk) I have designated to be backed up to the "remote" server.

 

If you want a GUI for configuring and monitoring your backups, you should look into something like Syncthing or Resilio Sync.  There are dockers for both available in Community Applications.

 

The advantage to the rsync/SSH method is that the backup server does not have to be running/awake.  My script powers on the backup server via IPMI and shuts it down after the backup completes.  Other just wake or S3 sleep the backup server after completing the backup.  The dockers won't give you this ability, but, you can do just that part of it in a script and once the backup server is online, the docker-based backups should start doing their thing.

Link to comment
  • 5 months later...

If you want to really annoy the NSA, CIA, GCHQ, MI5, AIVD etc. you could change your rsync / ssh solution to use solely https://ed25519.cr.yp.to/

A fine tutorial to achieve that is here: https://stribika.github.io/2015/01/04/secure-secure-shell.html

 

Currently I'm using the /boot/config/go script thusly;

 

#!/bin/bash

cp -af /boot/config/xroot/. /root/
cp -af /boot/config/xssh/. /etc/ssh/
chmod -R 0700 /root
chmod 0600 /root/.ssh/*
chmod 0644 /etc/ssh/*

# Start the Management Utility
/usr/local/sbin/emhttp &

 

Edited by fluisterben
  • Like 1
Link to comment
  • 4 months later...
On 10/14/2018 at 12:03 PM, ken-ji said:

So this is what needs to happen for SSH to work without prompts, or errors after a reboot.

Unraid server:

  • /root/.ssh directory with permissions (700)
  • /root/.ssh/id_rsa file needs to exists with the permissions (600); this is your private key
  • /root/.ssh/known_hosts with permissions (600); this file contains the public key of the servers you've connected to and stops the prompting of the untrusted host/ unknown keys; if the server changes (or a MITM attack occurs) this will prevent SSH from connecting until the server public keys match or is scrubbed from the file
  • /root/.ssh/config with permissions (600); this specifies some config options, like the server aliases, keyfiles, etc  - this is not necessary if you are connecting to the other server as root, using the server IP address (or a name that your Unraid server can resolve into its IP adrress)
  • (optional) /root/.ssh/id_rsa.pub file; this is the public key pair to your private key

 

I noticed unRAID alters /etc/ssh content at random times, not sure why or when, but it's not just at boot time.

Do you know if unRAID also alters the /root/.ssh folder at any time, or is that left alone (except at RAMdisk creation of course) ?

Either way, I could run a cron overwriting them every few hours or so, preceded by updating the (custom ssh) source's known hosts and authorized keys files.

 

Link to comment

Mine does not. I has the time stamp of the last time ssh service is restarted (or when my server booted up)

Unraid (and Linux in general) do not even create /root/.ssh until you've tried to connect to a ssh host from the root account; at which point the ssh host keys need to be saved and the directory and file are created.

Edited by ken-ji
Link to comment
  • 2 months later...

Thanks for the write up @tr0910! Thanks also to @ken-ji and @Hoopster for your trouble shooting and help.

 

It took some head scratching for me to get my head wrapped around your source & destination designations and where you were actually running the rsync command. Once I did, I was able to alter it to push from my main server to my backup server (instead of pulling main from backup).

 

I did the whole thing with the backup server already at the remote location. I'd been running Duplicati with the backup server sitting right next to the main server and finally got the backup shipped off with the son in college (half-way across the state should be a safe distance, right? :)). Once the server was there, Duplicati refused to launch the GUI, so my hand was forced to make a change. I've only got about 2.5TB of data left to backup, so I may have the kid bring the backup server home with him when he returns for Christmas break to make this initial backup see run just a bit faster...

 

I'm running ZeroTier (as noted earlier by Hoopster). The kids are using it to access the Emby server on my main machine, and it's working great for having the 2 servers talk to each other. I'm not sure of the impact on backup speeds yet, but we'll see what happens. It's also handy with the ZT client on my phone - I can use ControlR to check on the servers any time and from anywhere. 

 

I haven't yet attempted a reboot of either machine to see if the ssh keys survive. I'm going to let the first phase of backups finish before doing that. I put the keys I created in the /root/config/ssh directory which was already on the flash drive. I've modified my go file to copy just the 3 specific files created for this process. Are there any (known) issues to either this process or unRAID in general by having the extra files in that directory?

Edited by FreeMan
Link to comment
  • 2 weeks later...

Sadly, my config didn't survive a reboot. The good news, though, is that I got it figured out! (I think)

 

While it makes perfect sense, it isn't explicitly clear (at least not to my reading) that the *-rsync-key* files must be on the server being connected to, not the one being connected from. I had modified the go file on my main server to copy the keys, but not on the Backup server. I've manually copied them and set permissions and all is working fine. I'll make the matching changes in the go file and would expect all will be well in Whoville.

 

I do still have some questions about permissions, though. Initially in the OP it says:

Quote

Edit I didn't have the above permissions correct.  Make sure that the files in the .ssh directory have the exact permissions below, and the .ssh directory has 700 permissions

chmod 700 .ssh/

chmod 600 .ssh/Tower-rsync-key.pub

chmod 644 .ssh/authorized_keys

Yet a bit later on, when talking about modifying the go file, here are the recommended lines:

Quote

# Copy files back to /root/.ssh folder and set permissions for key files and known_hosts and authorized_keys

cd /root/.ssh

cp /boot/config/sshroot/* /root/.ssh/

chmod 600 *

It appears to me that authorized_keys needs to be 644 (that's how I have it right now) on my Backup server, yet the GO file instructions indicate setting it to 600.

 

EDIT: Nope, I did `chmod 600 ~/.ssh/*` and I was able to connect. Still though, those two sets statements are confusing.

 

Thoughts? 

Edited by FreeMan
Link to comment

I’ve looked at a couple of other options for backup and wanted to share one that can be used locally or externally without additional conf.

it requires rclone, letsencrypt/similar and nextcloud.

 

nextcloud set up with external access via linuxserver/letsencrypt after that create a rclone remote using the WebDAV option. That’s it. You can also create a Crypt remote if you want to encrypt your transfer.

 

sounds simpler, so I’m tabling it here to see if I’m missing some obvious deficiency.

 

thank you 

Edited by juan11perez
Link to comment
  • 3 weeks later...

I've been trying to get this work but no matter what I do, every time I ssh it keeps asking for a password.

Here is what I do, perhaps I'm missing something.

 

I have an Unraid server (192.168.1.159) and a Synology DS411j (192.168.1.9).

 

On the Unraid server, I run ssh-keygen -t rsa and give it the name "unraidkey". When it asks for a passphrase I just hit enter. It generates a private and public key file. They are in the following location on the Unraid server. ~/root/.ssh.

 

On the Synology I created the user "fsync", made it a member of the administrators group. I also enabled the Homes service. SSH was already enabled. And I uncommented the required lines from the SSHD config file and restarted SSH.

 

Now I copy the public key from the Unraid server to the Synology with the command: ssh-copy-id -i unraidkey [email protected]

 

Back to the Synology. I made sure the chmod permissions are set as described: chmod 700 on .ssh/ and 644 on .ssh/authorized_keys.

 

According to the description I should be able to connect to the Synology from the Unraid without password. But when I SSH from the Unraid terminal to [email protected] it keeps asking for a password.

 

What could I be doing wrong?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.