Jump to content
tr0910

Syncronize servers using rsync over SSH next door or across the world

110 posts in this topic Last Reply

Recommended Posts

thanks man, must have missed that.. had family over this weekend. :)

Share this post


Link to post
Posted (edited)

once i got to this step:

# Copy files back to /root/.ssh folder and set permissions for key files and known_hosts and authorized_keys cd /root/.ssh cp /boot/config/sshroot/* /root/.ssh/ chmod 600 *

 

I rebooted my Unraid and got this..   Everything was working until i rebooted Unraid.  I think the Freenas side is ok still.  I was reading and was able to fix the 500 server error by commenting those lines in my go file and ssh into the box and typing chmod +x /sbin/init.  Not sure what is going on, thought i had it beat.

 

 

servererror.png

Edited by xman111

Share this post


Link to post

cd /root/.ssh
cp /boot/config/sshroot/* /root/.ssh/
chmod 600 *

Odd that it caused you issues... but you can change these to

cp /boot/config/sshroot/* /root/.ssh/
chmod 600 /root/.ssh

you might want to read the entire thread for minor corrections and suggestions (I kinda know this stuff already so I might misread a few things here and there)

Share this post


Link to post

thanks ken-ji.  i got most of it working, just didn't survive a re-boot.. thanks for helping me, i want to get this Freenas out of my house and over to my parents as soon as possible in case of a fire or anything else.   Don't want to move it until I got this figured out.  I have been working on computers for 30 years but this stuff is fairly complicated.  Even getting the key file from one computer to the other was a pain in the ass :)  Going to read through the thread again to see if i missed anything. 

 

One other thing i was thinking of, if I want to sync from my unraid server to the Freenas, should i generate all the keys and such on the freenas server?  i will only be doing one way syncs from my unraid server over to the Freenas.

Edited by xman111

Share this post


Link to post
3 hours ago, xman111 said:

thanks ken-ji.  i got most of it working, just didn't survive a re-boot.

I had the same issue.  Below is my post back on page three about what I did to resolve this.  It was the last step in getting it all working and I have had zero issues since then.  The script runs unattended every week and just works.

 

 

Share this post


Link to post

thanks Hoopster, i will follow your advice.. i am going to try again this weekend as I have kids activities all week.  It's a little confusing jumping around pages trying to get it to work.  You and Ken-ji are heros :)

Share this post


Link to post
11 minutes ago, xman111 said:

thanks Hoopster, i will follow your advice.

Good luck with it all.  I don't know what issue you experienced post reboot, but, if it was an unknown host, what I did should help.

 

Family first as it should be always. 👍

  • Upvote 1

Share this post


Link to post

Yeah, there is a bit of black magic here that secures the rsync over ssh. Once properly applied, it just works.

This thread contains detours that frustrate folks before getting things working completely.

Sent from my ONEPLUS A3010 using Tapatalk

  • Like 1

Share this post


Link to post

for sure it does. I appreciate you guys taking the time to teach people.  Family first for sure Hoopster!  

Edited by xman111

Share this post


Link to post

Sorry tr0910, i forgot to mention the original hero that started it all, thanks man!!  I will get this working. 

Share this post


Link to post

ya, i can't get this thing working.   do both servers need to have the same keys in their directories?  or are there different keys for each machine?  does it matter what machine generates the keys?   I will always just be going from my unraid server (host) to the freenas server (remote), not the other way. I seem to have some permissions or something werid going on, wasted another 2 hours last night and am no further ahead.

 

I was going to setup OPENvpn client on Freenas but everyone said this is easier, so I switched my game plan.  I don't know how a normal person does it, i see how it would be just too much.  I am the goto computer guy for all my friends and family and just can't figure it out. 

 

Would be easier just setting up a pfsense box at my parents house with VPN :)

Edited by xman111

Share this post


Link to post

hey guys, playing with this again, tried to run :

 

rsync -avz --e ssh root@192.168.10.100:/mnt/user/misc/Test /mnt/Volume1/DataSet1/Newtest   from server to freenas and i get this error, i think i may have screwed up permissions on unraid. any way to reset them or is my syntax wrong?

 

root@192.168.10.100's password:
Permission denied (publickey,password,keyboard-interactive)

Share this post


Link to post

A thorough understanding of ssh keys is required to fully understand what is wrong. Trouble is all of us just hacked ours together. We all had the same troubles you are having. Then suddenly it just worked. And we are not really sure why.

 

@ken-ji is the most knowledgeable and helped most of us get going. Be patient, there really is gold at the end of this rainbow

 

 

Share this post


Link to post

Your welcome. @ken-ji to the rescue

Sent from my chisel, carved into granite

Share this post


Link to post

So this is what needs to happen for SSH to work without prompts, or errors after a reboot.

Unraid server:

  • /root/.ssh directory with permissions (700)
  • /root/.ssh/id_rsa file needs to exists with the permissions (600); this is your private key
  • /root/.ssh/known_hosts with permissions (600); this file contains the public key of the servers you've connected to and stops the prompting of the untrusted host/ unknown keys; if the server changes (or a MITM attack occurs) this will prevent SSH from connecting until the server public keys match or is scrubbed from the file
  • /root/.ssh/config with permissions (600); this specifies some config options, like the server aliases, keyfiles, etc  - this is not necessary if you are connecting to the other server as root, using the server IP address (or a name that your Unraid server can resolve into its IP adrress)
  • (optional) /root/.ssh/id_rsa.pub file; this is the public key pair to your private key

Target server:

  • /root/.ssh directory with permissions (700)
  • /root/.ssh/authorized_keys with permissions (600); this contains the public key part of your private key (1 pub key per line of the file; can contain multiple keys)

since Unraid is a RAM-disk OS, you just need to make sure that the above directory and 2+ files are created/restored upon reboot.

There are a bunch of scripts/go file modifications above to this extent.

 

Since you mentioned your Target is FreeNAS and IIRC, the root partition is on flash or HDD, so you just need to create the 2 directory and file just once.

You only need one key-pair. Specially since you are doing one way transfers (Unraid connects to FreeNAS, FreeNAS doesn't connect back)

 

Take your time to get this right, as SSH is a very strict protocol and clients will often just fail the connection if something feels off.

Edited by ken-ji

Share this post


Link to post

thanks for that Ken-ji.  I have it working with being able to reboot the Freenas server.  this is the command I am working with right now moving some files over.

rsync -avz --no-perms -e "ssh -i /root/.ssh/Server-rsync-key" /mnt/user/misc/Music/ root@192.168.10.105:/mnt/Volume1/Media/Music

 

was having a few issues where sometimes i would do a test transfer and i would try to delete the files through windows explorer and it would say access denied, user 'nobody' needs to delete this.  I would have to ssh in and run midnight commander as root to delete the files.

 

going to leave it for the week and start on the renaming the key and rebooting unraid for next weekend.

 

thanks a lot guys for the help.  this is the hardest, stupidest thing i have ever had to do in my life :)

Edited by xman111

Share this post


Link to post




 
thanks a lot guys for the help.  this is the hardest, stupidest thing i have ever had to do in my life


I warn you, there is no "SSH for Dummies..."

Sent from my chisel, carved into granite

  • Like 1

Share this post


Link to post
3 hours ago, xman111 said:

was having a few issues where sometimes i would do a test transfer and i would try to delete the files through windows explorer and it would say access denied, user 'nobody' needs to delete this.  I would have to ssh in and run midnight commander as root to delete the files.

This is normal since your are mixing and not controlling the way you access all the files.

I have my server mitigate this by making all the dockers and all the SMB accesses are done as the nobody user; but I still get tripped up by some of my directories being owned by root. This happens because I ssh in and perform manipulation of files as the root user - can't do it as nobody - since nobody can't quite login... meh...

Share this post


Link to post
7 minutes ago, tr0910 said:

I warn you, there is no "SSH for Dummies..."

Do you think it would sell very well? :D

Share this post


Link to post
15 hours ago, ken-ji said:

Do you think it would sell very well? :D

 

Well, you have already written it with your advice to me and xman. 

 

Just take several of your posts in this thread, put it in book form with a few pictures of kittens, babies and dancing bears (to appeal to the non-technical crowd) and it will be an Internet sensation! 😀

  • Like 1
  • Upvote 1

Share this post


Link to post

guys, finally got this working.. i watched a couple more videos on Youtube on the subject.

 

I am only going one way, from Unraid to Freenas.  So i only had to go onto the Unraid server, generate the keys, then only copy the public one over to Freenas.  I only needed one set of keys.

 

this is my go file.  I tried using Hoopster's but had some permission errors on my system.

 

# Copy SSH files back to /root/.ssh folder and set permissions for files
mkdir -p /root/.ssh
cp /boot/config/sshroot/Server-rsync-key /root/.ssh/id_rsa
cp /boot/config/sshroot/known_hosts /root/.ssh/known_hosts
cat /boot/config/sshroot/Server-rsync-key.pub > /root/.ssh/authorized_keys
chmod 700 /root/.ssh
chmod 600 /root/.ssh/id_rsa
chmod 600 /root/.ssd/known_hosts
# chmod g-rwx,o-rwx -R /root/.ssh

 

I have two IP addresses on my home internet account and a spare modem.  I was able to plug my Freenas into a port on the back of that router, so i can Rsync between server and freenas over the internet in my room to test.

 

I still am going to be going through the scripts and change the port for SSH, etc.  Now i can work on something else, that took way too much of my time the last couple weekends. 

 

Thanks guys for all the help and support, sorry for being a pain in the ASS!  

 

 

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now