Jump to content
tr0910

Syncronize servers using rsync over SSH next door or across the world

120 posts in this topic Last Reply

Recommended Posts

Yes it is the public IP

 

I wanted to share this, but realized I needed to remove personal info. I may have gone so far that the script won't run. Let me know if anything that should be adjusted for future users

 

 

via mobile

 

 

 

 

 

 

Share this post


Link to post

Errors to this tutorial were corrected in post 1.

 

The permissions of the .ssh folder are very strict.  Make sure you have them correct. 

 

I can now state that this process allows incredible rsync speeds even behind the GFW of China.  I had  a server there on a 100mbit download connection actually max the line when downloading from a google fiber gbit line in central USA.  This was with large files.

 

The best thing about this is how efficient it is on subsequent runs.  On a folder with 100,000 files, it can sift through the unchanged files in seconds, even halfway around the world.  Totally awesome.

Share this post


Link to post
15 hours ago, tr0910 said:

I can now state that this process allows incredible rsync speeds even behind the GFW of China.  I had  a server there on a 100mbit download connection actually max the line when downloading from a google fiber gbit line in central USA.  This was with large files.

 

The best thing about this is how efficient it is on subsequent runs.  On a folder with 100,000 files, it can sift through the unchanged files in seconds, even halfway around the world.  Totally awesome.

 

Here is an example of how efficient it is with updating an existing group of files.  Even though the Chinese internet was congested during peak hours it figured out the files that needed to be updated from USA and did it in very fast order.
 

Number of files: 9,643 (reg: 9,431, dir: 212)
Number of created files: 24 (reg: 24)
Number of deleted files: 0
Number of regular files transferred: 28
Total file size: 84,993,365,336 bytes
Total transferred file size: 3,323,141 bytes
Literal data: 3,239,841 bytes
Matched data: 83,300 bytes
File list size: 73,105
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 5,230
Total bytes received: 3,505,563

sent 5,230 bytes  received 3,505,563 bytes  34,933.26 bytes/sec
total size is 84,993,365,336  speedup is 24,209.16

 

Share this post


Link to post

Here is the other extreme.  4 large iso's going halfway around the world.  Over 6mb/sec on a 100mbit line.  There were times it was almost running at line speed.  However at other times of the day, say evening in China, it is nowhere near so impressive.

Number of files: 5 (reg: 4, dir: 1)
Number of created files: 5 (reg: 4, dir: 1)
Number of deleted files: 0
Number of regular files transferred: 4
Total file size: 16,443,052,032 bytes
Total transferred file size: 16,443,052,032 bytes
Literal data: 16,443,052,032 bytes
Matched data: 0 bytes
File list size: 221
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 103
Total bytes received: 16,447,066,831

sent 103 bytes  received 16,447,066,831 bytes  6,142,695.40 bytes/sec
total size is 16,443,052,032  speedup is 1.00

 

Share this post


Link to post

Here is the other extreme.  4 large iso's going halfway around the world.  Over 6mb/sec on a 100mbit line.  There were times it was almost running at line speed.  However at other times of the day, say evening in China, it is nowhere near so impressive.

Number of files: 5 (reg: 4, dir: 1)
Number of created files: 5 (reg: 4, dir: 1)
Number of deleted files: 0
Number of regular files transferred: 4
Total file size: 16,443,052,032 bytes
Total transferred file size: 16,443,052,032 bytes
Literal data: 16,443,052,032 bytes
Matched data: 0 bytes
File list size: 221
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 103
Total bytes received: 16,447,066,831

sent 103 bytes  received 16,447,066,831 bytes  6,142,695.40 bytes/sec
total size is 16,443,052,032  speedup is 1.00

 

Edited by tr0910

Share this post


Link to post

Reporting back to say that this process is so stable, that I may get rid of most of the places where I use the official Limetech Bittorrent Sync.  The only downside to this rsync process, is that it doesn't support 2 way sync.

Share this post


Link to post

Thanks for the rsync tutorial.  I just began yesterday to set up something similar between my two servers. In the future, the backup server will likely be moved to a different location across town, but, for now, both servers are on the same LAN.  Things seem to be working well in my limited tests, but, I have not yet rebooted either server to see if the SSH files survive the reboot.

 

A couple of comments/questions:

 

1 - in your example, you copy from disk to disk.  My systems are not identical as far as the number of disks/size in the array; however, the share definitions are identical.  I am copying files from user share to user share (e.g. /mnt/user/pictures/ to mnt/user/pictures).  In my limited testing so far, I have not noted any problems.  Any issues you can foresee?

 

2 - One of the test files I am copying is a 96GB Acronis True Image full backup of my desktop WIndows PC.  I have Acronis set to backup to my unRAID array.  I am seeing between 4-18 MB/s transfer rates on this file (watching it now), but, the average seems to be around 9-10 MB/s with occasional drops to 4-5 and occasional spikes to 17-18.  I used the rsync command you last noted as "optimal" in your initial post; i.e. rsync -avu --numeric-ids --progress -e "ssh -i /root/.ssh/XXXX-rsync.key -T -o Compression=no -x /mnt/user/Backups/ root@192.168.1.15:/mnt/user/Backups/ 

 

The backup array is parity protected, so, it is writing parity.  On a folder with a bunch of 4-7 MB pictures, it reported a speed of 48 MB/s

 

Am I using the best rsync parameters or have you done further refinements?  10 MB/s locally on a Gigabit LAN seems a little slow, but, maybe because it is a very large single compressed file writing to a parity-protected array, that is a normal speed given the SSH encryption overhead?

 

3 - My backup server S3 sleeps (never fully powered down) most of the time and I only want to wake it up for backup.  Locally, waking it in a script before starting the backup seems to be no problem.  Will that be an issue over the Internet? I have never tried that, but, it looks doable with several PC apps I have seen; however, I want to wake it automatically in the backup script.  Any suggestion for how best to do that?  As with your example, I suppose that depends a lot on the router on the other end.  The backup server does not have IPMI and I don't have a Raspberry Pi (could get one if necessary) to put on the other end.  I already have a DDNS name for the main server through No-IP and I can add the backup server as well.

 

Thanks again for the tutorial and associated links.  I learned a lot.

Share this post


Link to post

Hmm. just found this:

 

"To use WOL over the internet, you will have to forward UDP port 9 to the broadcast address of your local subnet. Because any requests incoming on port 9 are broadcasted, each client in the network (including the server) will receive the UDP packet and inspect the ARP frame it is wrapping. The MAC in the ARP packet then only matches that of the server, and the server's NIC can interpret the WOL request.
 

Oh and, of course you need a tool other than etherwake, because etherwake cannot send the WOL magic packet as a UDP packet to a designated host. It's ethernet-only. There are programs, scripts and even websites that do this however."

 

If I understand this correctly, I would need to send a magic packet with the MAC address of the backup server to the public IP of the router which forwards UDP port 9 to the broadcast address (255.255.255.XXX probably) of my local subnet.  If the router forwards UDP port 9 even if it doesn't see a service listening on that port, then the backup server will respond to the WOL request.

 

Now to figure out how the WOL magic packet can be scripted.

Share this post


Link to post

Sorry I can't help with the WOL. All my servers are IPMI. The ability to control the server via IPMI even on old eBay hardware is too valuable for me. I'm sure you can get WOL working, sort of, but getting it to be dead reliable will be the challenge

Sent from my Nexus 6 using Tapatalk

Share this post


Link to post

This was awesome thread I'd never seen before. I'm glad Hoopster brought it to the top of the list.

 

Hoopster, I've been down the road your thinking of traveling. I solved the WOL problem, but that only opened up the floodgates of more problem to solve, the biggest being able to power-off the remote server in the event it froze or became unresponsive. That lead to me buying a PDU to control the electrical outlet. In hindsight, I probably spend more money on a stupid quest ot make it work, when simply buying a board as tr0910 suggests was the FAR better solution.

Share this post


Link to post
On 8/19/2017 at 9:47 PM, Lev said:

In hindsight, I probably spend more money on a stupid quest ot make it work, when simply buying a board as tr0910 suggests was the FAR better solution.

My options are very limited when it comes to a Mini-ITX socket 1150 board with 6 SATA ports that has IPMI.  In fact, there is exactly one option; the ASRock Rack E3C226D21.  I have been reading up on it and it had lots of IPMI issues with the early BIOS/BMC releases; lots of RMA boards that would not boot and messed up IMPI NIC configurations that had to be reset with a special tool from ASRock.  I hope those issue are long gone.

 

The board does support the CPU I already have (Haswell-R i5 4590) from BIOS revision 2.00 but, I am not certain it supports the Kingston RAM (2x8 GB non-ECC 1333 MHz DDR3) I already have in the server even though, in general, it supports RAM of those specs.  I have contacted ASRock with questions about the IPMI implementation and RAM compatibility for future consideration.

 

EDIT:  On reading further, it looks like the issues were with the E3C224 full-size ATX board and not so much with the E3C226D21 Mini-ITX (although it had some minor issues as well).

Edited by Hoopster
Added info

Share this post


Link to post
On 8/19/2017 at 11:52 AM, Hoopster said:

2 - One of the test files I am copying is a 96GB Acronis True Image full backup of my desktop WIndows PC.  I have Acronis set to backup to my unRAID array.  I am seeing between 4-18 MB/s transfer rates on this file (watching it now), but, the average seems to be around 9-10 MB/s with occasional drops to 4-5 and occasional spikes to 17-18.  I used the rsync command you last noted as "optimal" in your initial post; i.e. rsync -avu --numeric-ids --progress -e "ssh -i /root/.ssh/XXXX-rsync.key -T -o Compression=no -x /mnt/user/Backups/ root@192.168.1.15:/mnt/user/Backups/

 

Answering my own question, yes, the slow speeds do appear to have been related to a very large *.tib file.  With videos. movies, pictures, TV shows, etc the files all transfer much faster (rsync is reporting between 40-100 MB/s depending on the size and number of files). Of course, I realize that is factoring in already existing files. 

 

The bottom line is that I am quite pleased with this rsync backup implementation and appreciate the work tr0910 did to document this.

Share this post


Link to post

I'm glad there are others picking this up and taking it further. unRAID is good for more than just ripped media files. With VPN and rsync and IPMI it can scale

Sent from my Nexus 6 using Tapatalk

Share this post


Link to post
23 hours ago, Hoopster said:

My options are very limited when it comes to a Mini-ITX socket 1150 board with 6 SATA ports that has IPMI. 

 

Just curious, why is Mini-ITX a hard requirement? Is there a physical space constraint where this server will be? Or your preference, if so why?

Share this post


Link to post
2 minutes ago, Lev said:

 

Just curious, why is Mini-ITX a hard requirement? Is there a physical space constraint where this server will be? Or your preference, if so why?

 

Existing case (Fractal Node 304) and space requirements .  I only have six disks in this backup server and do not anticipate needing more. Everything I need fits nicely in the case I already have and I am trying to keep the footprint to a minimum.  I am cheap and am trying to use what I already have as much as possible :)

 

ASRock responded to my concerns on the board and said my existing memory and CPU "should work without issue."

 

Mini-ITX is not an absolute requirement, it's just my lowest-cost option right now even though Mini-ITX server/workstation boards are not generally inexpensive and often cost more than their mATX or ATX equivalents.

Share this post


Link to post

Thanks, got it. I've been down that road before, so I can understand where you're coming from.

Share this post


Link to post

Resurrecting this topic since I never completely automated my rsync project between servers and now wish to do so.  Everything I did manually worked and I assumed there would be no issues; bad assumption!

 

I have changed motherboards in my "remote" server and it now has IPMI capabilities, so, my script now closely resembles the script developed by the OP.  Both servers are still on the same local network as I am not moving the backup server off site until I can get the process fully automated locally.

 

My problem is the same stated by Skulker.  After rebooting the "remote" server, ssh will not work to the remote server without a password. 

 

- I am not using the SSH Config plugin - may do so when everything is working before adding another variable

- both unRAID servers are running 6.4.0 stable release.

- Everything works as expected until the remote server is rebooted.  I do not want this server running constantly.  I power it on once a week via IPMI to receive backup files from the main server and then power it off via IPMI when done.

 

I have verified that the permissions are correct as per the updates in the original post .

 

I had to add a "mkdir /root/.ssh" line to my go file before copying the files from /boot/config/sshroot to /root/.ssh.  Without this, the server was inaccessible.  I could ping it but that was it; no GUI, no PuTTY. I had to modify the go file on the flash drive with my laptop and reboot the remote server via IPMI

 

My go file on the "remote" server contains the following and the server reboots properly:

 

# Copy files back to /root/.ssh folder and set permissions for files
mkdir /root/.ssh
cd /root/.ssh
cp /boot/config/sshroot/* /root/.ssh/
chmod 600 MediaNAS-rsync-key.pub
chmod 644 authorized_keys

 

After reboot, the .ssh folder is recreated, the files are copied correctly and with the correct permissions; however, no ssh operations from "local" to "remote" server work without prompting for root password.  If I delete the .ssh folders on both servers and redo all steps as outlined in the original post, ssh without password works again, but, after a reboot, I am back to square one. Since the automated script will involve powering the remote server on and off (reboot)  via IPMI this is a huge problem.

 

The files survive the reboot, the ability to ssh without a password does not.

 

Any ideas?

Edited by Hoopster

Share this post


Link to post

Here's what you need to do in your go file on each server:

mkdir /root/.ssh
cp /boot/config/ssh/authorized_keys /boot/config/ssh/id_rsa /root/.ssh
chmod g-rwx,o-rwx -R /root/.ssh

authorized_keys contains all the public keys that you will using to be able to login to this server. So for backup server, its the public key of id_rsa as saved on the main server.

id_rsa is the default name of the private key - you can name it anything you want, but you'll need to create/maintain an /root/.ssh/config file to specify this alternate name.

 

It seems you are not maintaining the directory permission for /root/.ssh properly.

Share this post


Link to post
49 minutes ago, ken-ji said:

Here's what you need to do in your go file on each server:


mkdir /root/.ssh
cp /boot/config/ssh/authorized_keys /boot/config/ssh/id_rsa /root/.ssh
chmod g-rwx,o-rwx -R /root/.ssh

authorized_keys contains all the public keys that you will using to be able to login to this server. So for backup server, its the public key of id_rsa as saved on the main server.

id_rsa is the default name of the private key - you can name it anything you want, but you'll need to create/maintain an /root/.ssh/config file to specify this alternate name.

 

It seems you are not maintaining the directory permission for /root/.ssh properly.

 

Thanks I will give this a try later tonight when I return home.  I appreciate the suggestion and will report the results.

Share this post


Link to post

>>

cat Tower-rsync-key.pub >> authorized_keys 

 

This part from the original doesn't seem to be getting saved back to your destination flash drive.  Later when you reboot, you are bringing in an old copy of authorized keys that doesn't have this new information in it.  However Ken-Ji is far more qualified to get you going than I am.  I only hacked this together, and the initial writeup is ugly but it works.  We look forward to your success and hope to see a smoother writeup being proposed by the adopters of this process.

 

I'll only say that once you figure it out, it will be super easy from then on.  It is working for me with 6.4.0 -> 6.3.5 without issue.  The raspberry pi at the destination location is the key to making my destination location dead silent and power bills low until it needs to wakeup to do something.

Share this post


Link to post
18 hours ago, ken-ji said:

Here's what you need to do in your go file on each server:


mkdir /root/.ssh
cp /boot/config/ssh/authorized_keys /boot/config/ssh/id_rsa /root/.ssh
chmod g-rwx,o-rwx -R /root/.ssh

authorized_keys contains all the public keys that you will using to be able to login to this server. So for backup server, its the public key of id_rsa as saved on the main server.

id_rsa is the default name of the private key - you can name it anything you want, but you'll need to create/maintain an /root/.ssh/config file to specify this alternate name.

 

It seems you are not maintaining the directory permission for /root/.ssh properly.

 

Obviously, I am still trying to wrap my head around SSH and public/private keys and how they work.  Thanks for your patience as I learn.

 

There is no file named id_rsa in the /boot/config/ssh folder on either server. 

 

These files exist in /boot/config/ssh folder on both servers:

ssh_host_rsa_key

ssh_host_rsa_key.pub

 

is it one (or both) of these files that need to be copied to /root/.ssh by a go file entry as in your example?

 

EDIT:  OK, reading this again, is "id_rsa" a placeholder for "whatever file name I created" in the ssh-keygen step in the OP?  If so, I generated a file called "MediaNAS-rsync-key.  Is it this file that should be copied to /root/.ssh?

 

Are you saying that I need to maintain a copy of "Name of my id_rsa file" in the /root/.ssh/config folder (and that his folder must be created on every reboot?)

Edited by Hoopster

Share this post


Link to post

Sorry about that: Let me be a bit more concise

On Serever1:

# ssh-keygen -t rsa -b 2048 -f /boot/config/ssh/server1_key
Generating public/private rsa key pair. 
Enter passphrase (empty for no passphrase): [press enter here] 
Enter same passphrase again: [press enter here]
# scp /boot/config/ssh/server1_key.pub root@server2:/boot/config/ssh

Insert this into your /boot/config/go file

mkdir -p /root/.ssh
cp /boot/config/ssh/server1_key /root/.ssh/id_rsa
cat /boot/config/ssh/server2_key.pub > /root/.ssh/authorized_keys
chmod g-rwx,o-rwx -R /root/.ssh

Repeat for Server2

 

Then on each server, run the same lines you inserted into the go file

# mkdir -p /root/.ssh
# cp /boot/config/ssh/server1_key /root/.ssh/id_rsa
# cat /boot/config/ssh/server2_key.pub > /root/.ssh/authorized_keys
# chmod g-rwx,o-rwx -R /root/.ssh

at this point you can do password-less ssh between servers. If you have more servers, just add the keys to the cat command which dumps it all into the authorized_keys file

I forgot that you shouldn't use the wildcard *.pub here as it will include the unwanted ssh_host keys too.

Share this post


Link to post
4 hours ago, Hoopster said:

 

Obviously, I am still trying to wrap my head around SSH and public/private keys and how they work.  Thanks for your patience as I learn.

 

There is no file named id_rsa in the /boot/config/ssh folder on either server. 

 

These files exist in /boot/config/ssh folder on both servers:

ssh_host_rsa_key

ssh_host_rsa_key.pub

 

is it one (or both) of these files that need to be copied to /root/.ssh by a go file entry as in your example?

 

EDIT:  OK, reading this again, is "id_rsa" a placeholder for "whatever file name I created" in the ssh-keygen step in the OP?  If so, I generated a file called "MediaNAS-rsync-key.  Is it this file that should be copied to /root/.ssh?

 

Are you saying that I need to maintain a copy of "Name of my id_rsa file" in the /root/.ssh/config folder (and that his folder must be created on every reboot?)

/root/.ssh/config is a file and like .ssh needs to be created everytime

and if your key file (generated with ssh-keygen) is not named id_rsa, you'll need to insert something like this in the /root/.ssh/config file

Host server2
    IdentityFile /root/.ssh/server1

 

Share this post


Link to post
20 hours ago, ken-ji said:

Sorry about that: Let me be a bit more concise

On Serever1:


# ssh-keygen -t rsa -b 2048 -f /boot/config/ssh/server1_key
Generating public/private rsa key pair. 
Enter passphrase (empty for no passphrase): [press enter here] 
Enter same passphrase again: [press enter here]
# scp /boot/config/ssh/server1_key.pub root@server2:/boot/config/ssh

Insert this into your /boot/config/go file


mkdir -p /root/.ssh
cp /boot/config/ssh/server1_key /root/.ssh/id_rsa
cat /boot/config/ssh/server2_key.pub > /root/.ssh/authorized_keys
chmod g-rwx,o-rwx -R /root/.ssh

Repeat for Server2

 

Then on each server, run the same lines you inserted into the go file


# mkdir -p /root/.ssh
# cp /boot/config/ssh/server1_key /root/.ssh/id_rsa
# cat /boot/config/ssh/server2_key.pub > /root/.ssh/authorized_keys
# chmod g-rwx,o-rwx -R /root/.ssh

at this point you can do password-less ssh between servers. If you have more servers, just add the keys to the cat command which dumps it all into the authorized_keys file

I forgot that you shouldn't use the wildcard *.pub here as it will include the unwanted ssh_host keys too.

 

Thanks for this.  These are very clear instructions.  I followed them on both servers and I am able to rsync in either direction between servers without a password prompt.  Of course, when repeating the instructions on server two "server1" info is replaced with "server2" info and vice versa.  I think most people will figure that out if I did :)

 

These instructions work perfectly!

 

I have not yet rebooted either server as I have an additional question related to your follow-up post below.

 

20 hours ago, ken-ji said:

/root/.ssh/config is a file and like .ssh needs to be created everytime

and if your key file (generated with ssh-keygen) is not named id_rsa, you'll need to insert something like this in the /root/.ssh/config file


Host server2
    IdentityFile /root/.ssh/server1

 

 

"Server1" in my case is named "medianas" and "Server2" is called "backupnas."  When I generated the ssh keys on each server I named them "medianas_key" and "backupnas_key." 

 

Even though these keys are being copied to the file id_rsa in the go file (as per your instructions) and "id_rsa" is the key file referenced in my rsync commands, are you saying that since the keys generated by ssh-keygen are not named "id_rsa" that I need to take the additional steps you mentioned, or, is everything OK since "id_rsa" is the file being created in /root/.ssh by the go file commands?

 

I am assuming that since the generated keys (even though they are not called id_rsa) are being copied to "id_rsa" in /root/.ssh that this will work without creating /root/.ssh/config and specifying the identity files.  Is this correct?

Share this post


Link to post
On 12/9/2016 at 12:30 PM, tr0910 said:

Edit I didn't have the above permissions correct.  Make sure that the files in the .ssh directory have the exact permissions below, and the .ssh directory has 700 permissions


chmod 700 .ssh/
chmod 600 .ssh/Tower-rsync-key.pub
chmod 644 .ssh/authorized_keys

 

@ken-ji The above was added to the original post based on a comment you made regarding permissions.  I see that after following your instructions, there are three files in .ssh.  authorized_keys has 600 permission, yet you indicated it should have 644 permissions.; id_rsa has 700 permissions, yet you indicated the key file should have 600 permissions; known_hosts is also in .ssh with 600 permissions.

 

This is what is see in .ssh:

-rw------- authorized_keys

-rwx------ id_rsa

-rw------- known_hosts

 

rsync in both directions without password is working now (I have yet to reboot either server) with the above permissions.

 

The inclusion of

chmod g-rwx,o-rwx -R /root/.ssh

in the go file would seem to indicate that all files in .ssh should have rwx permissions for group and other users, but, it appears that is not being set. Only owner permissions are set.

 

If specific file permissions are important, I just want to make sure the proper permissions are being set.

 

I apologize for all the replies/questions; however, in case others want to go down the rsync backup path, I want to make sure that the correct information is in this thread.  I thank you and @tr0910 for all your great work to get this properly documented.

 

Edited by Hoopster

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.