Syncronize servers using rsync over SSH next door or across the world


Recommended Posts

once i got to this step:

# Copy files back to /root/.ssh folder and set permissions for key files and known_hosts and authorized_keys cd /root/.ssh cp /boot/config/sshroot/* /root/.ssh/ chmod 600 *

 

I rebooted my Unraid and got this..   Everything was working until i rebooted Unraid.  I think the Freenas side is ok still.  I was reading and was able to fix the 500 server error by commenting those lines in my go file and ssh into the box and typing chmod +x /sbin/init.  Not sure what is going on, thought i had it beat.

 

 

servererror.png

Edited by xman111
Link to comment

cd /root/.ssh
cp /boot/config/sshroot/* /root/.ssh/
chmod 600 *

Odd that it caused you issues... but you can change these to

cp /boot/config/sshroot/* /root/.ssh/
chmod 600 /root/.ssh

you might want to read the entire thread for minor corrections and suggestions (I kinda know this stuff already so I might misread a few things here and there)

Link to comment

thanks ken-ji.  i got most of it working, just didn't survive a re-boot.. thanks for helping me, i want to get this Freenas out of my house and over to my parents as soon as possible in case of a fire or anything else.   Don't want to move it until I got this figured out.  I have been working on computers for 30 years but this stuff is fairly complicated.  Even getting the key file from one computer to the other was a pain in the ass :)  Going to read through the thread again to see if i missed anything. 

 

One other thing i was thinking of, if I want to sync from my unraid server to the Freenas, should i generate all the keys and such on the freenas server?  i will only be doing one way syncs from my unraid server over to the Freenas.

Edited by xman111
Link to comment
3 hours ago, xman111 said:

thanks ken-ji.  i got most of it working, just didn't survive a re-boot.

I had the same issue.  Below is my post back on page three about what I did to resolve this.  It was the last step in getting it all working and I have had zero issues since then.  The script runs unattended every week and just works.

 

 

Link to comment

ya, i can't get this thing working.   do both servers need to have the same keys in their directories?  or are there different keys for each machine?  does it matter what machine generates the keys?   I will always just be going from my unraid server (host) to the freenas server (remote), not the other way. I seem to have some permissions or something werid going on, wasted another 2 hours last night and am no further ahead.

 

I was going to setup OPENvpn client on Freenas but everyone said this is easier, so I switched my game plan.  I don't know how a normal person does it, i see how it would be just too much.  I am the goto computer guy for all my friends and family and just can't figure it out. 

 

Would be easier just setting up a pfsense box at my parents house with VPN :)

Edited by xman111
Link to comment

A thorough understanding of ssh keys is required to fully understand what is wrong. Trouble is all of us just hacked ours together. We all had the same troubles you are having. Then suddenly it just worked. And we are not really sure why.

 

@ken-ji is the most knowledgeable and helped most of us get going. Be patient, there really is gold at the end of this rainbow

 

 

Link to comment

So this is what needs to happen for SSH to work without prompts, or errors after a reboot.

Unraid server:

  • /root/.ssh directory with permissions (700)
  • /root/.ssh/id_rsa file needs to exists with the permissions (600); this is your private key
  • /root/.ssh/known_hosts with permissions (600); this file contains the public key of the servers you've connected to and stops the prompting of the untrusted host/ unknown keys; if the server changes (or a MITM attack occurs) this will prevent SSH from connecting until the server public keys match or is scrubbed from the file
  • /root/.ssh/config with permissions (600); this specifies some config options, like the server aliases, keyfiles, etc  - this is not necessary if you are connecting to the other server as root, using the server IP address (or a name that your Unraid server can resolve into its IP adrress)
  • (optional) /root/.ssh/id_rsa.pub file; this is the public key pair to your private key

Target server:

  • /root/.ssh directory with permissions (700)
  • /root/.ssh/authorized_keys with permissions (600); this contains the public key part of your private key (1 pub key per line of the file; can contain multiple keys)

since Unraid is a RAM-disk OS, you just need to make sure that the above directory and 2+ files are created/restored upon reboot.

There are a bunch of scripts/go file modifications above to this extent.

 

Since you mentioned your Target is FreeNAS and IIRC, the root partition is on flash or HDD, so you just need to create the 2 directory and file just once.

You only need one key-pair. Specially since you are doing one way transfers (Unraid connects to FreeNAS, FreeNAS doesn't connect back)

 

Take your time to get this right, as SSH is a very strict protocol and clients will often just fail the connection if something feels off.

Edited by ken-ji
Link to comment

thanks for that Ken-ji.  I have it working with being able to reboot the Freenas server.  this is the command I am working with right now moving some files over.

rsync -avz --no-perms -e "ssh -i /root/.ssh/Server-rsync-key" /mnt/user/misc/Music/ [email protected]:/mnt/Volume1/Media/Music

 

was having a few issues where sometimes i would do a test transfer and i would try to delete the files through windows explorer and it would say access denied, user 'nobody' needs to delete this.  I would have to ssh in and run midnight commander as root to delete the files.

 

going to leave it for the week and start on the renaming the key and rebooting unraid for next weekend.

 

thanks a lot guys for the help.  this is the hardest, stupidest thing i have ever had to do in my life :)

Edited by xman111
Link to comment
3 hours ago, xman111 said:

was having a few issues where sometimes i would do a test transfer and i would try to delete the files through windows explorer and it would say access denied, user 'nobody' needs to delete this.  I would have to ssh in and run midnight commander as root to delete the files.

This is normal since your are mixing and not controlling the way you access all the files.

I have my server mitigate this by making all the dockers and all the SMB accesses are done as the nobody user; but I still get tripped up by some of my directories being owned by root. This happens because I ssh in and perform manipulation of files as the root user - can't do it as nobody - since nobody can't quite login... meh...

Link to comment
15 hours ago, ken-ji said:

Do you think it would sell very well? :D

 

Well, you have already written it with your advice to me and xman. 

 

Just take several of your posts in this thread, put it in book form with a few pictures of kittens, babies and dancing bears (to appeal to the non-technical crowd) and it will be an Internet sensation! 😀

  • Like 1
  • Upvote 1
Link to comment

guys, finally got this working.. i watched a couple more videos on Youtube on the subject.

 

I am only going one way, from Unraid to Freenas.  So i only had to go onto the Unraid server, generate the keys, then only copy the public one over to Freenas.  I only needed one set of keys.

 

this is my go file.  I tried using Hoopster's but had some permission errors on my system.

 

# Copy SSH files back to /root/.ssh folder and set permissions for files
mkdir -p /root/.ssh
cp /boot/config/sshroot/Server-rsync-key /root/.ssh/id_rsa
cp /boot/config/sshroot/known_hosts /root/.ssh/known_hosts
cat /boot/config/sshroot/Server-rsync-key.pub > /root/.ssh/authorized_keys
chmod 700 /root/.ssh
chmod 600 /root/.ssh/id_rsa
chmod 600 /root/.ssd/known_hosts
# chmod g-rwx,o-rwx -R /root/.ssh

 

I have two IP addresses on my home internet account and a spare modem.  I was able to plug my Freenas into a port on the back of that router, so i can Rsync between server and freenas over the internet in my room to test.

 

I still am going to be going through the scripts and change the port for SSH, etc.  Now i can work on something else, that took way too much of my time the last couple weekends. 

 

Thanks guys for all the help and support, sorry for being a pain in the ASS!  

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.