Jump to content
tr0910

Syncronize servers using rsync over SSH next door or across the world

115 posts in this topic Last Reply

Recommended Posts

7 minutes ago, xman111 said:

guys, finally got this working

 

Congrats!  It's a great feeling when it all works.  Now that you are an SSH expert, perhaps @ken-ji will allow you to guest author a chapter in his book and let you in on the royalties. 😀

Share this post


Link to post

haha, i think i need a vacation :)  and yes it is a great feeling.  my wife thinks i am an idiot for spending all this time on something like this when i can just drive a hard drive over to my parents house..  i told her that isn't the point :)

 

funny thing is i am watching these youtube videos doing it and they are only like 3 minutes long, i got about 25 hours into it!

 

about all i could add to the book is some coloring :)   i just hope neither system goes down because i would hate to have to do it again.

 

 

Edited by xman111

Share this post


Link to post

reinstalled Freenas because i was swapping around hard drives.  Decided to delete all files and redo this from scratch.  Had it up and running in 5 minutes.  Took good notes last time in case this would happen.  :)

Share this post


Link to post

hey guys, one last question.   I mentioned this before in this thread but I just did some syncing and i go to delete the files in Windows from the shares and it says something about I don't have the proper permissions.  Is there a way to fix this?  don't want to start syncing files and have all sorts of different permissions attached to different files.

Share this post


Link to post

figured this out, was a permission setting on the share on Freenas.

 

I finally moved the Freenas out of the house to my folks house. It worked no problem, just gave the machine a static IP and forwarded the port. The problem is, it is very slow, getting only 1.25 mb/s so a 3.2gb Windows ISO takes about 45 minutes.  Any tips on speeding it up?  oh by the way, i have 20 up and they have 50 down.

 

Thanks for all the help with this!

Edited by xman111

Share this post


Link to post
15 minutes ago, xman111 said:

Any tips on speeding it up?

If you can't get good speed with remote SSH, perhaps remote access through ZeroTier is an option with SSH as if it were local.  I have not used ZeroTier myself, but, judging from the responses in the thread users are happy with it.  Not sure how the speed compares to SSH. I have no idea if starting ZeroTier an establishing a connection can be automated.

 

 

Share this post


Link to post
26 minutes ago, xman111 said:

figured this out, was a permission setting on the share on Freenas.

 

I finally moved the Freenas out of the house to my folks house. It worked no problem, just gave the machine a static IP and forwarded the port. The problem is, it is very slow, getting only 1.25 mb/s so a 3.2gb Windows ISO takes about 45 minutes.  Any tips on speeding it up?  oh by the way, i have 20 up and they have 50 down.

 

Thanks for all the help with this!

if you have 1.2 MB/s uploading to your folks, you are already using ~12mbps of your 20mbps so it might be normal as internet overheads and other stuff can clog your pipes. It can be better, as I can do about 2.2MB/s given the same situation 20up/50down on the other end.

In my case, The whole connection is wrapped in something similar to Zerotier - a pair of Mikrotik routers running Ethernet over IP with IPSEC to do site to site VPN.

Share this post


Link to post
1 minute ago, xman111 said:

Are my speeds normal? 

Hard for me to say as my backup server is still in the basement.  I keep threatening to move it to my son's house, but, other projects have taken priority. @tr0910 could certainly chime in on this as he is backing up between China and the US.

Share this post


Link to post

thanks man, will have to play around a bit.  this was the output.

 

sent 3,158,320,869 bytes  received 87 bytes  1,526,865.34 bytes/s 

Share this post


Link to post

well I need to do the same thing. god I wish unraid had a default gui option. I dont mind setting it up but I prefer a more visual way of monitoring it with GUI once it works.

 

what do you guys recommend as a good starting point?

Share this post


Link to post
1 hour ago, BelgarionNL said:

what do you guys recommend as a good starting point? 

Well, obviously, there is no rsync GUI in unRAID.  The best you can hope to accomplish by following the instructions in this thread is to learn how to write rsync/SSH scripts that automate your backups and, if desired, emails you the results.

 

There is a steep learning curve if your are not already familiar with rsync and SSH, but, once you get this working, it really is a set-it-and-forget it solution.  For me the script has been running unattended for months (automated as a cron via the User Scripts unRAID plugin) and, every week, I receive a summary email that gives me the important stats for each share (some do it by disk) I have designated to be backed up to the "remote" server.

 

If you want a GUI for configuring and monitoring your backups, you should look into something like Syncthing or Resilio Sync.  There are dockers for both available in Community Applications.

 

The advantage to the rsync/SSH method is that the backup server does not have to be running/awake.  My script powers on the backup server via IPMI and shuts it down after the backup completes.  Other just wake or S3 sleep the backup server after completing the backup.  The dockers won't give you this ability, but, you can do just that part of it in a script and once the backup server is online, the docker-based backups should start doing their thing.

Share this post


Link to post

I only want to sync some tv shows and all my movies to my parents WD mybook live.

 

and preferably backup their pictures to my unraid server. (but I will use onedrive for that because its just simpler)

 

ps where do I find rsync? is it resillio sync?

Edited by BelgarionNL

Share this post


Link to post
4 hours ago, BelgarionNL said:

 

ps where do I find rsync? is it resillio sync?

No, rsync is a command line utility built into unRAID.  There is a Resilio Sync docker available for unRAID, but, it is not related to rsync.

Share this post


Link to post
Posted (edited)

If you want to really annoy the NSA, CIA, GCHQ, MI5, AIVD etc. you could change your rsync / ssh solution to use solely https://ed25519.cr.yp.to/

A fine tutorial to achieve that is here: https://stribika.github.io/2015/01/04/secure-secure-shell.html

 

Currently I'm using the /boot/config/go script thusly;

 

#!/bin/bash

cp -af /boot/config/xroot/. /root/
cp -af /boot/config/xssh/. /etc/ssh/
chmod -R 0700 /root
chmod 0600 /root/.ssh/*
chmod 0644 /etc/ssh/*

# Start the Management Utility
/usr/local/sbin/emhttp &

 

Edited by fluisterben

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.