Rsync- kindly point me in the right direction

[gmahn72]

Hello,

 

Ive been trying to do my research on finding the solution to my problem but cant seem to find the right thread.  Pretty simple thing I want to do - hopefully someone can point me in the right direction. 

 

I basically just want to sync a remote directory (within my LAN) with my unraid box.  The remote folder is where my completed torrents reside and once complete, I want them to be transferred to my unraid box.  Ive been doing so with a permanent mount, but feel that going the rsync way is much more efficient.  This thread http://lime-technology.com/forum/index.php?topic=6350.0 basically highlights what I want to do, however it goes a little over my head since I almost need a walkthrough from inception  - as I have some pretty basic q's:

 

1. Based on my research, something like this might be what I need:

    rsync -a --progress --size-only /home/newfiles/* server.example.com:/unraidbox/destination

 

(I presumed that this would run from my remote box, but not sure if that is the case.    Is there a difference?  If not, I dont really care - as long as the directories are sync'd and once complete, the orig files on the source directory can be erased.

 

2.  If the script is to be run from the unraid box, how do I make it so that it survives a reboot?

 

 

So in summary, I just would like a remote directory's files to be syncd with a share on my unraid box/  Probably run this once or twice a day, or even better - will run automatically once a file is added. 

 

can someone please shed some light on this? Id really appreciate it.  Thnks.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

f

f

f

[starcat]

Look here: http://lime-technology.com/forum/index.php?topic=3471.msg64096#msg64096

and here http://lime-technology.com/forum/index.php?topic=6479.msg63046#msg63046

and if you want here http://lime-technology.com/forum/index.php?topic=3159.0

 

and have fun!

 

[gmahn72]

Ok I think I got it to work, after much reading.  I do have a follow up question regarding syntax of rsync.  What would the command be if I just wanted the source directory to check the contents of the dest drive, and anything that is missing, it will add it on.  If it is there (size), then to pass, and if there are items in the destination directory that are not in the source, to NOT delete them.

 

Lastly, is there a followup command that can be issued that once the files are copied, that they can be deleted from the source? 

 

So, in sum.....Drive A has apples, bears and coconuts.  Drive B has Coconuts and Dogs.  After rsync is initiated on Drive A, Drive B will now have apples, bears coconuts and dogs.  Drive A will be empty.

 

Would this do it?

 

#!/bin/bash

rsync -ra /drivea/folder/* tower.lan.ip:/mnt/user/NewStuff/

 

What I cant get is how to script the deletion of all copied files from Drive A, if that is even possible. 

 

Much appreciated.

 

 

 

[BRiT]

To delete everything inside the Drive A folder:

 

rm -rf /drivea/folder/*

[starcat]

Take extreme care to not put a space before "*", otherwise you will delete all that is in /drive/folder and also all from the current directory!

[gmahn72]

Thanks for the heads up/tip !  In any regard, the directory and all subfolders in drive a are empty (or would have been transferred to the unraid box) so it shouldnt affect anything....however, good to know, and point well taken.  Much appreciated.

[gmahn72]

Sorry - just a clarification  - this is what the script would be.....correct?  i just wanted to be sure that the files were transferred first, prior to deletion. 

 

 

#!/bin/bash

rsync -ra /drivea/folder/* tower.lan.ip:/mnt/user/NewStuff/

 

rm -rf /drivea/folder/*

 

[gmahn72]

Got my answer - it worked perfect.Thanks again

[BRiT]

Now that I'm more awake, there's an option to rsync to do this in line, which I think you should use instead of the 2 command syntax. I'd only use the 'rm' after manually verifying the rsync process finished without issue. The way it's setup there, the rsync could fail and yet the delete would still run.

 

It's the "--remove-source-files" option.

 

#!/bin/bash

rsync -ra --remove-source-files /drivea/folder/* tower.lan.ip:/mnt/user/NewStuff/

 

You may still need to deal with empty directories on the source side. You may also need to deal with files in use on the source side too. This is quickly becoming more complicated, so you may want to start off with the built-in unRAID 'mover' script and modify from there to suit your needs. I think it's located in /usr/local/sbin/mover.

 

You'll want to change the /mnt/user0 references to be your Destination info and /mnt/cache to be your Source infor. You'll likewise need to remove the destination directory check before it kicks off too.

 

I think it'll end up looking something like this -- mymover.sh :

 

#!/bin/bash

# If a previous invokation of this script is already running, exit
if [ -f /var/run/mymover.pid ]; then
  if ps h `cat /var/run/mymover.pid` | grep mymover ; then
      echo "mymover already running"
      exit 0
  fi
fi
echo $$ >/var/run/mymover.pid

echo "mymover started"

(cd /drivea/folder/ ; find -depth -print \
\( -type f -regex '[.]/.*/.*' ! -exec fuser -s {} \; \
    -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace --remove-source-files {} tower.lan.ip:/mnt/user/NewStuff/ \; \) -o \
\( -type d -regex '[.]/.*' \
    -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} tower.lan.ip:/mnt/user/NewStuff/ \; -empty -delete \) \
)

sync

rm /var/run/mymover.pid

echo "mymover finished"

 

 

[gmahn72]

Funny you should mention that.....woke up this morning to find that that exact scenario happened and a few files were deleted before they were copied.  I got a syslog message saying something to that effect.  No biggie at all though.

 

Appreciate the help.  I will try what you wrote since it is now becoming a little over my head.  This is what the script looks like  - will report back shortly.

 

#!/bin/bash

 

# If a previous invokation of this script is already running, exit

if [ -f /var/run/mymover.pid ]; then

  if ps h `cat /var/run/mymover.pid` | grep mymover ; then

      echo "mymover already running"

      exit 0

  fi

fi

echo $$ >/var/run/mymover.pid

 

echo "mymover started"

 

(cd /drivea/folder/ ; find -depth -print \

\( -type f -regex '[.]/.*/.*' ! -exec fuser -s {} \; \

    -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace --remove-source-files {} tower.lan.ip:/mnt/user/NewStuff/ \; \) -o \

\( -type d -regex '[.]/.*' \

    -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} tower.lan.ip:/mnt/user/NewStuff/ \; -empty -delete \) \

)

 

sync

 

rm /var/run/mymover.pid

 

echo "mymover finished"