[SOLVED] Rsync between 2 unraid @ 10MB/s?

[dgaschk]

WeeboTechs script maks sure that there is no existing rsync line in /etc/inetd.conf before adding a new one. I'm not sure if it matters if there are 2 and in this case there are not but it is more future proof.

 

So replace

echo "rsync   stream  tcp     nowait  root    /usr/sbin/tcpd  /usr/bin/rsync --daemon" >> /etc/inetd.conf
cp /boot/custom/rsyncd.conf /etc
killall -HUP inetd

with

if ! grep ^rsync /etc/inetd.conf > /dev/null ; then 
cat <<-EOF >> /etc/inetd.conf
rsync   stream  tcp     nowait  root    /usr/sbin/tcpd  /usr/bin/rsync --daemon 
EOF
killall -HUP inetd
fi
cp /boot/custom/rsyncd.conf /etc

in the go file.

 

This

killall -HUP inetd

and this

 read PID < /var/run/inetd.pid
kill -1 ${PID}

are equivalent so I would keep the shorter version.

 

 

 

[whiteatom]

This is a great solution and all your scripts work great.. I'm actually using this to replicate my media library for backups.

 

I have run into one small issue.. I'm putting data into a user share (because i have way too much data for one disk) and it shows up properly on the disks and the shares, but if unRaid doesn't see the data. It still shows the disk as empty. I have restarted and everything.. I'm just in the procecss of doing a parity check.. but is this data being properly stored?

 

I'm using 5.0b11, so I'm aware it could be a bug.. but it appears that unRAID only monitors data that comes through the network shares.... Thoughts?

 

whiteatom

[dgaschk]

It sounds like the files are hidden. You can adjust the rsync flags to set the permissions correctly. See this for a fix to current files: http://lime-technology.com/forum/index.php?topic=3172.0

[whiteatom]

I don't think they are hidden.. but they could have a permissions issue... reading your post made me question the user I have the rsycn daemon running under. As I said in response to another post of yours on this topic.. I'm going to try again tonight with the newest beta in.

 

whiteatom

 

[Joe L.]

I don't think they are hidden.. but they could have a permissions issue... reading your post made me question the user I have the rsycn daemon running under. As I said in response to another post of yours on this topic.. I'm going to try again tonight with the newest beta in.

 

whiteatom

 

I seem to remember reading about a bug in some of the latest versions where the free space read by windows through the user-shares is not being updated properly.

 

Do not know if it has been fixed.

 

Don't know if it is an issue for rsync.... probably is if rsync target destination is also a user-share.

[molesza]

This is an awesome thread! Thanks for the tutorial! Going to use this to replicate my unraid server to another. Once everything is setup how would you run an rsync command say every hour?

[dgaschk]

google "crontab"

[aptalca]

I followed this thread to run an rsync server on unraid and perform successful syncs using root

 

However, then all the files on the server are owned by root only and not accessible through smb by other users

 

I know I can change permissions later on, but I thought, why not perform rsync as another user that belongs to the "users" gid so I don't have to worry about that extra step. That's where I ran into an issue and I hope you can help me as I am not that well versed in linux

 

First, I set up the openssh plugin on 5.0 stable. I created the folders for "root" and "rsync" and copied the public key to both.

 

Then I entered the line "sync stream tcp nowait root /usr/sbin/tcpd /usr/bin/rsync --daemon" in inetd and restarted it.

 

I also created the rsyncd.conf with the following:

 

uid             = root
gid             = root
use chroot      = no
max connections = 4
pid file        = /var/run/rsyncd.pid
timeout         = 600

[mnt]
    path = /mnt/user/Misc/deneme
    comment = nocomment
    read only = FALSE

 

If I use the "root" user in my rsync command on the client, I can copy whatever file into the "deneme" folder under the user share "Misc" while using ssh with the private key

 

However, if I use the user "rsync", I get an error message that says the host key accepted unconditionally, but "rsync: connection unexpectedly closed. . . rsync error: error in rsync protocol data stream (code 12)"

 

For the user rsync, I also changed the rsyncd.conf so the uid is rsync and the gid is users, still no go. Do I have to change the line in inetd as well?

 

Any ideas?

 

Thanks

[dgaschk]

change guid from root to users.

gid             = users

[aptalca]

You, Sir, are a gentleman and a scholar 

[eroz]

I figure I will help out the other non-linux gurus and create a step by step on how to make this work as I've often struggled with not knowing exactly how to do some of this.

 

Warning: I'm still a  linux newb. Guru's, please correct me where I'm wrong

Warning2: If anything breaks all blame goes to WeeboTech and if it works just right all prop goes to WeeboTech 

 

1. Install vim using unMenu's Pkg Manager (it's near the bottom)

2. Telent into your unRAID machine that you want to receive the data (in my case that is 'unRAID' ip: 192.168.0.189

3. Type this command:

 vim /etc/inetd.conf 

-This will open the inetd.conf file in Vi editor

4. Press the letter 'i' on your keyboard

5. Now you need to insert the following:

 rsync   stream  tcp     nowait  root    /usr/sbin/tcpd  /usr/bin/rsync --daemon 

-I placed it in the section below FTP

6. Once completed press 'ESC' then ':' followed with 'wq'

- Pressing escape will remove you from insert mode. Pressing ':' will give you an input line at the bottom of your screen and 'wq' will write and quit.

7. Issue the killall command to restart the inetd.conf

killall -HUP inetd 

8. Check that inetd is loaded

 ps -ef | grep inetd 

It should look like this:

root      1754     1  0 May21 ?        00:00:00 /usr/sbin/inetd
root     10250 10240  0 09:07 pts/0    00:00:00 grep inetd

9. Setup a rsyncd.conf file by doing the following:

 vim /etc/rsyncd.conf 

10. Press the letter 'i' to go into insert mode

11. Enter the following into the conf file

 
uid             = root
gid             = root
use chroot      = no
max connections = 4
pid file        = /var/run/rsyncd.pid
timeout         = 600

[mnt]
    path = /mnt
    comment = /mnt files
    read only = FALSE

12. Press ESC, ':' and 'wq'

13. Now open up another telnet window on the array that has the data you want to copy to the machine we just setup as the Rsync server (for me that is 'Tower' or 192.168.0.199)

14. Type in the following command to being rsync

rsync -av --stats --progress /mnt/disk1/ rsync://192.168.0.189/mnt/disk11/

 

Note that the above will copy disk1 (on tower) to disk11 (on unraid)

 

** I highly suggest that you install Screen from unMenu and add screen to the beginning of the above command

screen  rsync -av --stats --progress /mnt/disk1/ rsync://192.168.0.189/mnt/disk11/

 

This will allow you to close your telnet window and also protect you if you shut down your local PC and still have the rsync script running. To check on the progress you will simply log back into your array and type

screen -x or screen -r

Awesome.  I will be setting up a new unraid tower to be my backup.  I was looking for a way to get it started.  I hope this How-To is still the way to do it.

[dgaschk]

Yes. And here: http://lime-technology.com/forum/index.php?topic=13432.msg127684#msg127684

[eroz]

Yes. And here: http://lime-technology.com/forum/index.php?topic=13432.msg127684#msg127684

Great....let me see if I understand it. Your method is all done on the receiving unraid and then on the sending unraid I would type something similiar to

screen  rsync -av --stats --progress /mnt/Movies/ rsync://192.168.1.6/mnt/Movies/

 

with the -a means to sync it in archive mode

and the  v  means to be verbose

and the --progress means to give live status.

 

but what does the --stats  mean?

[Fireball3]

[pre]--stats                give some file-transfer stats

[/pre]

[EdgarWallace]

This is the best rsync thread this I have found. Thanks to the contributors. Everything was working well in my case.

 

I was now trying to run a backup every hour. I created a script that is going to be copied into /etc/cron.daily/ via the go script on every reboot. This is what I have:

#!/bin/bash
#=======================================================================================
#  Name:        rsync_backup.sh
#=======================================================================================
#  Description:
#
#  A simple script to copy directory and files from tower ---> tower2 as remote backup
#
#  How to invoke in your "go" script (copy to /boot/scripts):
#  chmod +x /boot/scripts/rsync_backup.sh
#=======================================================================================
#
rsync -avH –-stats –-progress --delete-after --ignore-existing –log-file=/boot/logs/Movies_backup.txt /mnt/user/movies/       rsync://tower2/mnt/user/backup/movies/
rsync -avH –-stats –-progress --delete-after --ignore-existing –log-file=/boot/logs/HDX_backup.txt /mnt/user/HDX_Backup/      rsync://tower2/mnt/user/backup/HDX_backup/
rsync -avH –-stats –-progress --delete-after --ignore-existing –log-file=/boot/logs/iTunes_backup.txt /mnt/user/iTunes/       rsync://tower2/mnt/user/backup/iTunes/
rsync -avH –-stats –-progress --delete-after --ignore-existing –log-file=/boot/logs/ownCloud_backup.txt/mnt/user/ownCloud/    rsync://tower2/mnt/user/backup/ownCloud/
rsync -avH –-stats –-progress --delete-after --ignore-existing –log-file=/boot/logs/user1_backup.txt --exclude=iMacBackup.sparseimage  /mnt/user/user1/ rsync://tower2/mnt/user/backup/user1/

 

I have tested the iTunes backup first:

rsync: link_stat "/boot/scripts/–-stats" failed: No such file or directory (2)
rsync: link_stat "/boot/scripts/–-progress" failed: No such file or directory (2)
rsync: change_dir "/boot/scripts//–log-file=/boot/logs" failed: No such file or directory (2)

 

Here are my questions:

1. [*]what is the issue with the rsync options that I took from here:

http://rsync.samba.org/ftp/rsync/rsync.html?

[*]is the approach to copy the script into /etc/cron.daily/ the right one (survive boot)?

Thank you very much.

[dgaschk]

File should start with:

#!/bin/bash

[EdgarWallace]

Thanks dgaschk, I have made a copy&past error - corrected now....

[wolferl99]

great thread, thanks everyone. One slight addition: it might make sense to lock rsync down a bit for security reasons. If you want/need to, simply add a hosts allow statement to your rsync.conf file, e.g

 

[mnt]
    path = /mnt
    comment = /mnt files
    read only = FALSE
    hosts allow = 192.168.0.100

 

only allowing 192.168.0.100 to rsync to this box.

[teamhood]

Fantastic that this thread is still alive. Using it again and I forgot how I did this

 

Thanks to the update by @wolferI99

[gundamguy]

great thread, thanks everyone. One slight addition: it might make sense to lock rsync down a bit for security reasons. If you want/need to, simply add a hosts allow statement to your rsync.conf file, e.g

 

[mnt]
    path = /mnt
    comment = /mnt files
    read only = FALSE
    hosts allow = 192.168.0.100

 

only allowing 192.168.0.100 to rsync to this box.

 

Older thread is old, but still worthwhile.

 

I want to add that this is a great method of handling traffic you want to send within your trusted LAN. The Host allow option isn't so great if you’re planning to use the internet to transfer between two (or more) remote unRAID boxes. Just be aware that rsync as a deamon doesn't use ssh and send all information unencrypted. This is fine if you are on a trusted lan, but not fine for internet applications.

 

[rx8freehk]

Its 2015... isn't there a better option?  I need to move 12TB of data to a new Unraid server and this is maddening.. My Linux skills are admittedly horrindious

[trurl]

Its 2015... isn't there a better option?  I need to move 12TB of data to a new Unraid server and this is maddening.. My Linux skills are admittedly horrindious

If I just wanted to do it once, I would use Unassigned Devices plugin to SMB mount the other server shares, then use mc (Midnight Commander) to do the move.

[aptalca]

Its 2015... isn't there a better option?  I need to move 12TB of data to a new Unraid server and this is maddening.. My Linux skills are admittedly horrindious

If you don't feel comfortable with rsync, you can use mc in screen to copy it all

[Fireball3]

When doing so, make sure you don't miss hardlinks (if you have some).

I'm not sure if MC can handle that.

[RussellinSacto]

Wait a second...  Can I use Midnight Commander to move files from one UnRaid to another UnRaid?

 

That would be amazing!

 

Russell