RSYNC assitance

[dalben]

As a relative newbie when it comes to rsync, I'm looking for some help to craft a script to copy files based on a time stamp.

 

Scenario:

 

Unraid cache drive is where Transmission downloads files.

The files need to remain on the Cache for seeding.

I also want to copy the files to the array so they can easily play them on the various XBMC boxes around the house.

XBMC boxes will not have access to the Cache drive

 

I currently have a script that rsyncs between my cache drive and a directory on the array.  This is a fairly basic file the file isn't on the array, copy it.

It need to change so that it detects if the file on the cache drive is *newer* than the last time the rsync script ran. 

If the file is newer, copy it across.

If the file is older than the last run, ignore it.

 

I am trying to achieve 2 things:

 

1) the array doesn't spin up to check if a file exists everytime the script runs

2) If I move/rename a file from the destination, the script doesn't copy it over again.

 

I tried playing with touch to give a marker file a new timestamp but getting the logic to only copy files newer than the touched file is something I can't get my head aroun.

 

Any pointers ?

 

 

 

[Fireball3]

I have just been looking for a solution to the same problem.

Found this thread where weebotech points to a script in his google code.

I can't test it, as I don't have my backup server ready yet.

 

 

[WeeboTech]

Here's the basic principle.

I've done this with success.

 

use touch to create a file marker to start the process at some point in time.

 

touch a new (temp) file marker

use find (directory) -newer (touched file marker[not the temp one]) > filelist

use this filelist as input to rsync with --files-from=filelist

 

if the rsync is succesful, touch the initial file marker using -r (new (temp) file maker) to update the old file marker.

 

EDIT: I thought about this s'more.

 

You need to create an initial file marker for your starting point of reference.

These are the options to use.

 

-r, --reference=FILE  use this file's times instead of current time

-t STAMP              use [[CC]YY]MMDDhhmm[.ss] instead of current time

 

So create the initial marker with -t STAMP.

when you are going to actually do the work

touch a new temporary file marker to start the point of reference of when the find starts

do the find with -newer and the 'initial touched file marker' (not the temp one).  > filelist

do the rsync with --files-from=filelist

if the rsync ends without errors

touch -r 'new temporary file marker' initial file marker

remove 'new tempory file marker'

 

The reason to touch a file before the find is, depending on how large your directory is, find could take a long time.  So you need to capture a point of reference just before the find.

 

At the end of the rsync, if successful, update the -newer file marker with the filemarker from just before the find.

 

it all depends on how fast the find runs, if it's really fast you can touch the newer file marker using the mtime of the filelist as a point of reference.

 

-r filelist 'initial file marker'

 

Point is a file could be changed from the start of the find until the finish of the find, so you want to be sure and capture all the relevant files that are modified.  There's the possibility of processing a file twice this way, but since rsync only copies files that differ between source and destination, the worst possible problem is that a drive spins up and the file does not get copied because it already exists.

[dalben]

Thanks.  I'll try and conjure something  up.  My unix scripting skills only go as far as hacking other peoples scripts to change what I want so I might take a while to get this going.

[dalben]

trouble I am having is that the paths are doubling up.

 

the cron job runs from a different directory.  filelist contains a string like:

 

/mnt/cache/apps/downloads/torrents/complete/test.txt

 

The rysnc command is as follows:

rsync -avih --delete --stats --files-from=synclist --log-file=/tmp/torrlog /mnt/cache/apps/downloads/torrents/complete/ /mnt/user/videos/zz-newfiles

 

The errors I get are:

rsync start
building file list ... rsync: link_stat "/mnt/cache/apps/downloads/torrents/complete/mnt/cache/apps/downloads/torrents/complete" failed: No such file or directory (2)
rsync: link_stat "/mnt/cache/apps/downloads/torrents/complete/mnt/cache/apps/downloads/torrents/complete/test.txt" failed: No such file or directory (2)
done

 

 

Any ideas how I can stop that ?  Like I said, I am complete noob and can only hack away at others work.  Thinking for myself is tough....

 

[WeeboTech]

trouble I am having is that the paths are doubling up.

 

the cron job runs from a different directory.  filelist contains a string like:

 

/mnt/cache/apps/downloads/torrents/complete/test.txt

 

The rysnc command is as follows:

rsync -avih --delete --stats --files-from=synclist --log-file=/tmp/torrlog /mnt/cache/apps/downloads/torrents/complete/ /mnt/user/videos/zz-newfiles

 

The errors I get are:

rsync start
building file list ... rsync: link_stat "/mnt/cache/apps/downloads/torrents/complete/mnt/cache/apps/downloads/torrents/complete" failed: No such file or directory (2)
rsync: link_stat "/mnt/cache/apps/downloads/torrents/complete/mnt/cache/apps/downloads/torrents/complete/test.txt" failed: No such file or directory (2)
done

 

 

Any ideas how I can stop that ?  Like I said, I am complete noob and can only hack away at others work.  Thinking for myself is tough....

 

Why are there two paths on the rsync command line if you have -files-from= which should contain the full path of the files.

if you are using files from, I would expect only the destination directory on the command line.

 

or you may need to use relative paths for the filesfrom synclist.

 

I.E. relative from the first directory path so that they can be appended correctly.

[dalben]

Not having the source path there gave me errors.

 

I got it working in the end by changing the directory to the source directory and then running everything from in there.  Works fine for now though very crude.  I'll start adding bits of error trapping down the track.

 

Eventually I'll use this logic for all my rsync/backup scripts as it stops the array spinning up drives just to check whats at the destination even if it doesn't copy everything.

 

Thanks for the assistance.

[dalben]

Quic k question.  It all works fine except that when it detects a new file to copy, it also adds the directory root to filelist.txt.  Here arfe the contents oif an example filelist.txt

 

.
./test.txt

 

The script then copies over every file in thew directory regardless of the time stamp.  Is there a way to exclude the . from  the find command ?

[WeeboTech]

Quic k question.  It all works fine except that when it detects a new file to copy, it also adds the directory root to filelist.txt.  Here arfe the contents oif an example filelist.txt

 

.
./test.txt

 

The script then copies over every file in thew directory regardless of the time stamp.  Is there a way to exclude the . from  the find command ?

 

there is a ! parameter.

i.e. something like.

 

! -name '.'

[dalben]

Thanks.  That worked.,

 

Here's the script if anyone else is trying to do similar.  No laughing at how crude it is......

 

#!/bin/bash
#
# script to check new torrents and copy to array
# 
  
echo "rsync of the torrents begins"
# change to the download directory because I couldn't get it to work any other way
echo "we change the directory"
cd /mnt/cache/apps/downloads/torrents/complete/
# find files that are newer than the last script run and add them to the include file list
# added the ! to exclude the directory from the filelist
echo "we find the new files"
find ! -name '.' -newer  /boot/config/tdmrsync/torrsync/ttouch >/boot/config/tdmrsync/torrsync/synclist
# copy the bastards across
echo "we rsync the files"
rsync -avih --stats --files-from=/boot/config/tdmrsync/torrsync/synclist /mnt/cache/apps/downloads/torrents/complete/ /mnt/user/videos/zz-newfiles
# reset the touch file timestamp
echo "we touch the touchee"
touch /boot/config/tdmrsync/torrsync/ttouch
echo "rsync of the torrents end"