Syncronize servers using rsync over SSH next door or across the world


Recommended Posts

  • 2 weeks later...

I've run into an odd situation. After every reboot of my backup server, I have no /root/.ssh directory, so my attempt to copy the keys there fails. Any ideas why? 

 

I think something larger may be at play - this server, though built on some rather old hardware, seems to be taking longer and longer to reboot, and the GUI seems to be taking even longer to become available than it used to. I do have a drive that's got SMART errors, and I've got a replacement drive on the way, but I've excluded it from all shares and I've used unBalance to clear all the contents off of it in preparation for its replacement.

 

Diagnostics are attached in case anyone knows how to read the tea leaves and can see anything obvious...

backup-diagnostics-20200112-1321.zip

Link to comment
On 1/12/2020 at 11:22 AM, FreeMan said:

I've run into an odd situation. After every reboot of my backup server, I have no /root/.ssh directory, so my attempt to copy the keys there fails. Any ideas why? 

Because it is a temporary directory that only exists in RAM.  A reboot wipes it out, that's why my 'go' file recreates it on reboot before copying files to it.

 

# Copy SSH files back to /root/.ssh folder and set permissions for files
mkdir -p /root/.ssh
cp /boot/config/ssh/medianas_key /root/.ssh/id_rsa
cp /boot/config/ssh/known_hosts /root/.ssh/known_hosts
cat /boot/config/ssh/backupnas_key.pub > /root/.ssh/authorized_keys
chmod g-rwx,o-rwx -R /root/.ssh

 

Edited by Hoopster
  • Like 1
Link to comment
19 hours ago, Hoopster said:

Because it is a temporary directory that only exists in RAM.  A reboot wipes it out, that's why my 'go' file recreates it on reboot before copying files to it.

Makes sense, but I don't recall having had to recreate it the first time I got this set up. 

 

I was using ZeroTier to talk between the machines, now I'm using WireGuard since it's built in and doesn't involve using a 3rd party.

 

I'll create the directory and give it the proper rights in my go script - should be fine from there.

Link to comment
  • 1 month later...

Now it seems that something is changing on the servers to invalidate the keys each time the server(s) reboot.

 

The contents of my main server's go file:

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp &

cd /root/
mkdir .ssh/
chmod 700 .ssh/

cp /boot/config/ssh/NAS-rsync-key /root/.ssh/
#cp /boot/config/ssh/NAS-rsync-key.pub /root/.ssh/
cp /boot/config/ssh/known_hosts /root/.ssh/
cd /root/
chmod 700 .ssh/
#chmod 600 .ssh/NAS-rsync-key.pub
chmod 644 .ssh/authorized_keys

and the contents of my backup server's go file:

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp &

cd /root/
mkdir .ssh/
chmod 700 .ssh/
cp /boot/config/ssh/NAS-rsync-key.pub /root/.ssh/
chmod 600 /root/.ssh/NAS-rsync-key.pub
cp /boot/config/ssh/authorized_keys /root/.ssh/
chmod 600 /root/.ssh/authorized_keys

On the main server, I don't copy over the .pub file (I don't need the public key there, do I?). My servers rebooted a couple of days ago after I updated to 6.8.3, and now I get an error

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

When I try any rsync command. I have to redo the keygen procedure to generate a new key pair to get it to work.

 

Thoughts or tips?

 

I have to r

Link to comment

hmm. redoing the keys is not a real solution.

I can only guess something happened on your backup server ssh files to change it.

but do this

main+~# cat /boot/config/ssh/known_hosts | grep bakup_ip
backup_ip ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGiAoa1bxGekGXAP+HA8pzHZ0qerF5Cy2KHuZdQF43zE02m9Op0BylDzrGiq0iek5AvpyUetp6yrmHJGSJNGfzw=

backup:~# cat /boot/config/ssh/ssh_host_ecdsa_key.pub
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGiAoa1bxGekGXAP+HA8pzHZ0qerF5Cy2KHuZdQF43zE02m9Op0BylDzrGiq0iek5AvpyUetp6yrmHJGSJNGfzw= root`backup

if they don't match at any time, something changed on your USB

and you can check on the running system as well

main:/root/.ssh/known_hosts and backup:/etc/ssh/ssh_host_ecdsa_key.pub

 

Not a super paranoid guy, but if the Remote host id changed, i would not have recreated the keys right away and done some investigation, because the remote host id doesn't change unless some files are dropped from the backup USB for some reason. or someone gets in does something. maybe check if the time stamps of the ssh_host_*_key files are correct/expected vs when you upgraded and the other files in that directory.

 

Finally... not all your go lines are needed...

you either don't need the pub file on the backup, or you use the pub file to create the authorized_keys file like Hoopster's

 

Edited by ken-ji
Link to comment
23 hours ago, ken-ji said:

hmm. redoing the keys is not a real solution.

I can only guess something happened on your backup server ssh files to change it.

but do this


main+~# cat /boot/config/ssh/known_hosts | grep bakup_ip
backup_ip ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGiAoa1bxGekGXAP+HA8pzHZ0qerF5Cy2KHuZdQF43zE02m9Op0BylDzrGiq0iek5AvpyUetp6yrmHJGSJNGfzw=

backup:~# cat /boot/config/ssh/ssh_host_ecdsa_key.pub
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGiAoa1bxGekGXAP+HA8pzHZ0qerF5Cy2KHuZdQF43zE02m9Op0BylDzrGiq0iek5AvpyUetp6yrmHJGSJNGfzw= root`backup

if they don't match at any time, something changed on your USB

and you can check on the running system as well

main:/root/.ssh/known_hosts and backup:/etc/ssh/ssh_host_ecdsa_key.pub

 

Not a super paranoid guy, but if the Remote host id changed, i would not have recreated the keys right away and done some investigation, because the remote host id doesn't change unless some files are dropped from the backup USB for some reason. or someone gets in does something. maybe check if the time stamps of the ssh_host_*_key files are correct/expected vs when you upgraded and the other files in that directory.

 

Finally... not all your go lines are needed...

you either don't need the pub file on the backup, or you use the pub file to create the authorized_keys file like Hoopster's

 

 

Thanks, @ken-ji.

 

On my main server, cat known_hosts |grep backup yields "CTNPW0sjfvk=" as the last several characters.

On the backup server, cat ssh_host_ecdsa_key.pub yields "yDxNTBhImms=" as the last several characters.

Obviously something's changed somewhere.

 

The date time stamps on the NAS-rsync-key* files and the authorized_keys files match between the main and backup servers. The math between now and my uptime report for the servers indicates that these were created very shortly after I rebooted the server (I believe it was for the 6.8.3 update), which makes sense since that's when I would have discovered that I couldn't connect again.

 

Both machines are located in my house and on the same network. My son can VPN in and he is the only other person who knows the root pwd, but he's too busy with classes to be doing any messing around with the servers, and, frankly, is disappointingly uninterested in doing so (despite his CS major).

 

I'm not concerned with regenerating the keys because I'm (as) certain (as I can reasonably be) that the server hasn't been messed with from the outside. There are zero port forwards to the backup server. There are a couple to the main server (looking at them again, I believe I can shut almost all down at this point).

 

Any other thoughts on things to check before I create new key files? I will likely change the passwords on both servers to something longer and more complex, just to be on the safe side.

Link to comment
  • 4 weeks later...
On 1/15/2020 at 5:50 AM, Hoopster said:

Because it is a temporary directory that only exists in RAM.  A reboot wipes it out, that's why my 'go' file recreates it on reboot before copying files to it.

I have the problem, that the files dont get copied on boot.

 

This is my go file:

#!/bin/bash

# Copy files back to /root/.ssh folder and set permissions for key files and known_hosts and authorized_keys 

cd /root/.ssh
cp /boot/config/sshroot/* /root/.ssh/
chmod 600 *

# Start the Management Utility
/usr/local/sbin/emhttp &

 

When i use the command from you it works fine. Only the automatic copy isnt working.

Link to comment
8 hours ago, ryperx said:

When i use the command from you it works fine. Only the automatic copy isnt working.

You can't cd or cp to a directory that does not exist.  After every reboot /root/.ssh will be wiped out since it only exists in RAM when it is created.  It must be recreated every time via the 'go' file before you attempt to copy anything to it.

 

See my previous post in this thread for an example of what I am doing in the 'go' file.

Link to comment
2 minutes ago, Hoopster said:

You can't cd or cp to a directory that does not exist.  After every reboot /root/.ssh will be wiped out since it only exists in RAM when it is created.  It must be recreated every time via the 'go' file before you attempt to copy anything to it.

 

See my previous post in this thread for an example of what I am doing in the 'go' file.

Yep was my mistake, needed nearly the full day for enabling the headless ssh session and some other things with scripting. Thanks for your time.

So much information in this forum but really hard to find.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.