Jump to content
tr0910

Syncronize servers using rsync over SSH next door or across the world

119 posts in this topic Last Reply

Recommended Posts

7 minutes ago, xman111 said:

guys, finally got this working

 

Congrats!  It's a great feeling when it all works.  Now that you are an SSH expert, perhaps @ken-ji will allow you to guest author a chapter in his book and let you in on the royalties. 😀

Share this post


Link to post

haha, i think i need a vacation :)  and yes it is a great feeling.  my wife thinks i am an idiot for spending all this time on something like this when i can just drive a hard drive over to my parents house..  i told her that isn't the point :)

 

funny thing is i am watching these youtube videos doing it and they are only like 3 minutes long, i got about 25 hours into it!

 

about all i could add to the book is some coloring :)   i just hope neither system goes down because i would hate to have to do it again.

 

 

Edited by xman111

Share this post


Link to post

reinstalled Freenas because i was swapping around hard drives.  Decided to delete all files and redo this from scratch.  Had it up and running in 5 minutes.  Took good notes last time in case this would happen.  :)

Share this post


Link to post

hey guys, one last question.   I mentioned this before in this thread but I just did some syncing and i go to delete the files in Windows from the shares and it says something about I don't have the proper permissions.  Is there a way to fix this?  don't want to start syncing files and have all sorts of different permissions attached to different files.

Share this post


Link to post

figured this out, was a permission setting on the share on Freenas.

 

I finally moved the Freenas out of the house to my folks house. It worked no problem, just gave the machine a static IP and forwarded the port. The problem is, it is very slow, getting only 1.25 mb/s so a 3.2gb Windows ISO takes about 45 minutes.  Any tips on speeding it up?  oh by the way, i have 20 up and they have 50 down.

 

Thanks for all the help with this!

Edited by xman111

Share this post


Link to post
15 minutes ago, xman111 said:

Any tips on speeding it up?

If you can't get good speed with remote SSH, perhaps remote access through ZeroTier is an option with SSH as if it were local.  I have not used ZeroTier myself, but, judging from the responses in the thread users are happy with it.  Not sure how the speed compares to SSH. I have no idea if starting ZeroTier an establishing a connection can be automated.

 

 

Share this post


Link to post
26 minutes ago, xman111 said:

figured this out, was a permission setting on the share on Freenas.

 

I finally moved the Freenas out of the house to my folks house. It worked no problem, just gave the machine a static IP and forwarded the port. The problem is, it is very slow, getting only 1.25 mb/s so a 3.2gb Windows ISO takes about 45 minutes.  Any tips on speeding it up?  oh by the way, i have 20 up and they have 50 down.

 

Thanks for all the help with this!

if you have 1.2 MB/s uploading to your folks, you are already using ~12mbps of your 20mbps so it might be normal as internet overheads and other stuff can clog your pipes. It can be better, as I can do about 2.2MB/s given the same situation 20up/50down on the other end.

In my case, The whole connection is wrapped in something similar to Zerotier - a pair of Mikrotik routers running Ethernet over IP with IPSEC to do site to site VPN.

Share this post


Link to post
1 minute ago, xman111 said:

Are my speeds normal? 

Hard for me to say as my backup server is still in the basement.  I keep threatening to move it to my son's house, but, other projects have taken priority. @tr0910 could certainly chime in on this as he is backing up between China and the US.

Share this post


Link to post

thanks man, will have to play around a bit.  this was the output.

 

sent 3,158,320,869 bytes  received 87 bytes  1,526,865.34 bytes/s 

Share this post


Link to post

well I need to do the same thing. god I wish unraid had a default gui option. I dont mind setting it up but I prefer a more visual way of monitoring it with GUI once it works.

 

what do you guys recommend as a good starting point?

Share this post


Link to post
1 hour ago, BelgarionNL said:

what do you guys recommend as a good starting point? 

Well, obviously, there is no rsync GUI in unRAID.  The best you can hope to accomplish by following the instructions in this thread is to learn how to write rsync/SSH scripts that automate your backups and, if desired, emails you the results.

 

There is a steep learning curve if your are not already familiar with rsync and SSH, but, once you get this working, it really is a set-it-and-forget it solution.  For me the script has been running unattended for months (automated as a cron via the User Scripts unRAID plugin) and, every week, I receive a summary email that gives me the important stats for each share (some do it by disk) I have designated to be backed up to the "remote" server.

 

If you want a GUI for configuring and monitoring your backups, you should look into something like Syncthing or Resilio Sync.  There are dockers for both available in Community Applications.

 

The advantage to the rsync/SSH method is that the backup server does not have to be running/awake.  My script powers on the backup server via IPMI and shuts it down after the backup completes.  Other just wake or S3 sleep the backup server after completing the backup.  The dockers won't give you this ability, but, you can do just that part of it in a script and once the backup server is online, the docker-based backups should start doing their thing.

Share this post


Link to post

I only want to sync some tv shows and all my movies to my parents WD mybook live.

 

and preferably backup their pictures to my unraid server. (but I will use onedrive for that because its just simpler)

 

ps where do I find rsync? is it resillio sync?

Edited by BelgarionNL

Share this post


Link to post
4 hours ago, BelgarionNL said:

 

ps where do I find rsync? is it resillio sync?

No, rsync is a command line utility built into unRAID.  There is a Resilio Sync docker available for unRAID, but, it is not related to rsync.

Share this post


Link to post
Posted (edited)

If you want to really annoy the NSA, CIA, GCHQ, MI5, AIVD etc. you could change your rsync / ssh solution to use solely https://ed25519.cr.yp.to/

A fine tutorial to achieve that is here: https://stribika.github.io/2015/01/04/secure-secure-shell.html

 

Currently I'm using the /boot/config/go script thusly;

 

#!/bin/bash

cp -af /boot/config/xroot/. /root/
cp -af /boot/config/xssh/. /etc/ssh/
chmod -R 0700 /root
chmod 0600 /root/.ssh/*
chmod 0644 /etc/ssh/*

# Start the Management Utility
/usr/local/sbin/emhttp &

 

Edited by fluisterben

Share this post


Link to post
On 10/14/2018 at 12:03 PM, ken-ji said:

So this is what needs to happen for SSH to work without prompts, or errors after a reboot.

Unraid server:

  • /root/.ssh directory with permissions (700)
  • /root/.ssh/id_rsa file needs to exists with the permissions (600); this is your private key
  • /root/.ssh/known_hosts with permissions (600); this file contains the public key of the servers you've connected to and stops the prompting of the untrusted host/ unknown keys; if the server changes (or a MITM attack occurs) this will prevent SSH from connecting until the server public keys match or is scrubbed from the file
  • /root/.ssh/config with permissions (600); this specifies some config options, like the server aliases, keyfiles, etc  - this is not necessary if you are connecting to the other server as root, using the server IP address (or a name that your Unraid server can resolve into its IP adrress)
  • (optional) /root/.ssh/id_rsa.pub file; this is the public key pair to your private key

 

I noticed unRAID alters /etc/ssh content at random times, not sure why or when, but it's not just at boot time.

Do you know if unRAID also alters the /root/.ssh folder at any time, or is that left alone (except at RAMdisk creation of course) ?

Either way, I could run a cron overwriting them every few hours or so, preceded by updating the (custom ssh) source's known hosts and authorized keys files.

 

Share this post


Link to post

I've never suspected this but do you have any proof of it happening?

Sent from my chisel, carved into granite

Share this post


Link to post

Mine does not. I has the time stamp of the last time ssh service is restarted (or when my server booted up)

Unraid (and Linux in general) do not even create /root/.ssh until you've tried to connect to a ssh host from the root account; at which point the ssh host keys need to be saved and the directory and file are created.

Edited by ken-ji

Share this post


Link to post

I've never seen any problems with ssh folders or files being randomly altered.  I've been running the backup script with ssh and go file changes for over 18 months with no problems.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.