I can't get my head around a way to do this:

 

I want to backup my most critical unRAID data (photos & HD videos of my kids & documents) to an offsite location (NAS box) using a USB key as the transport device...

 

But here's the catch: I don't want my old data left on the USB key indefinitely, just new/changed files due to:

  A) security reasons - what if I lose the key? Those keys are pretty small.

  B) filesize - I'd eventually run out of space on the key.

 

So, to summarize, I really want the USB key to act as a temporal transport system where the changed files are deleted from it once successfully moved to the destination NAS box.

 

I've had some fun getting incremental backups working with rsync using the 'link-dest' parameter, but rsync requires access to the destination structure when determining which files have changed. 

Unless I'm missing something?

 

 

Assuming you know unix/linux well enough to help yourself with details and implementation, here are some ideas.

 

I've used it on production servers at my job.

 

You can use the find command with the -newer option to find all files newer(mtime) > then a reference file.

You will need to touch this reference file and leave it somewhere.

 

Then you do something like

 

find /mnt/diskx -newer /boot/reference.file -print >> /flash/filelist

 

Now you have a list of files that are newer then the reference.

At this point I would touch the /boot/reference file.

You can also do some fancy schmancy stuff where once you've used the /flash/filelist you cp -p that file to the /boot/reference file so you know what the prior copied files are.

 

You can use this /flash/filelist with rsync, tar, cpio to create some archive you work with.

rsync can use an external filelist as can tar.

I.E. rsync to a flash drive, tar the list of files with gzip to the flash drive, etc, etc.

 

 

That's all I could come up with.

 

root@atlas ~ #touch --help
Usage: touch [OPTION]... FILE...
Update the access and modification times of each FILE to the current time.

Mandatory arguments to long options are mandatory for short options too.
  -a                     change only the access time
  -c, --no-create        do not create any files
  -d, --date=STRING      parse STRING and use it instead of current time
  -f                     (ignored)
  -m                     change only the modification time
  -r, --reference=FILE   use this file's times instead of current time
  -t STAMP               use [[CC]YY]MMDDhhmm[.ss] instead of current time
  --time=WORD            change the specified time:
                           WORD is access, atime, or use: equivalent to -a
                           WORD is modify or mtime: equivalent to -m
      --help     display this help and exit
      --version  output version information and exit

Note that the -d and -t options accept different time-date formats.

If a FILE is -, touch standard output.



root@atlas ~ #find --help
Usage: find [path...] [expression]

default path is the current directory; default expression is -print
expression may consist of: operators, options, tests, and actions:

operators (decreasing precedence; -and is implicit where no others are given):
      ( EXPR )   ! EXPR   -not EXPR   EXPR1 -a EXPR2   EXPR1 -and EXPR2
      EXPR1 -o EXPR2   EXPR1 -or EXPR2   EXPR1 , EXPR2

positional options (always true): -daystart -follow -regextype

normal options (always true, specified before other expressions):
      -depth --help -maxdepth LEVELS -mindepth LEVELS -mount -noleaf
      --version -xdev -ignore_readdir_race -noignore_readdir_race

tests (N can be +N or -N or N): -amin N -anewer FILE -atime N -cmin N
      -cnewer FILE -ctime N -empty -false -fstype TYPE -gid N -group NAME
      -ilname PATTERN -iname PATTERN -inum N -iwholename PATTERN -iregex PATTERN
      -links N -lname PATTERN -mmin N -mtime N -name PATTERN [b]-newer FILE[/b]
      -nouser -nogroup -path PATTERN -perm [+-]MODE -regex PATTERN
      -wholename PATTERN -size N[bcwkMG] -true -type [bcdpflsD] -uid N
      -used N -user NAME -xtype [bcdpfls]

actions: -delete -print0 -printf FORMAT -fprintf FILE FORMAT -print 
      -fprint0 FILE -fprint FILE -ls -fls FILE -prune -quit
      -exec COMMAND ; -exec COMMAND {} + -ok COMMAND ;
      -execdir COMMAND ; -execdir COMMAND {} + -okdir COMMAND ;

 

 

 

 

Assuming you know unix/linux well enough to help yourself with details and implementation, here are some ideas.

 

I've used it on production servers at my job.

 

You can use the find command with the -newer option to find all files newer(mtime) > then a reference file.

You will need to touch this reference file and leave it somewhere.

 

Then you do something like

 

find /mnt/diskx -newer /boot/reference.file -print >> /flash/filelist

 

Now you have a list of files that are newer then the reference.

At this point I would touch the /boot/reference file.

You can also do some fancy schmancy stuff where once you've used the /flash/filelist you cp -p that file to the /boot/reference file so you know what the prior copied files are.

 

You can use this /flash/filelist with rsync, tar, cpio to create some archive you work with.

rsync can use an external filelist as can tar.

I.E. rsync to a flash drive, tar the list of files with gzip to the flash drive, etc, etc.

 

That's all I could come up with.

Thanks that really helped me.

Think I've got it working now...

 

#!/bin/bash
# filelist_allBackups.txt contains log of all backups
# filelist_lastBackup.txt contains log of most recent backup
# note: to reset the backup & recopy all files in SOURCE locations irrespective of when they were updated, use: touch -t 197001010000 $HOME/filelist_lastBackup.txt

date=`date "+%Y-%m-%d_%H-%M-%S"`
LOG=/var/log/rsyncBackup.log
HOME=/boot/rsyncBackup

SOURCEROOT="/mnt/user/"
SOURCE1="Media/Pictures"
SOURCE2="Media/Videos"
SOURCE3="Time Machine"
SOURCE4="Backup"

EXTERNALDISK=/mnt/disk/ExternalBackupDisk
EXTERNALDISKPATH=$EXTERNALDISK/BACKUPS/Location1
BACKUPDESTINATION=$EXTERNALDISKPATH/rsyncBackup

# check that the expected path exists...
if [ -d $EXTERNALDISKPATH ]
then
echo "$date STARTING" >> $LOG
echo "$date Looking for changed files since last sync..." >> $LOG

# look for changed files since last sync (i.e since the modification time of filelist_lastBackup.txt)
cd "$SOURCEROOT"
find "$SOURCE1" \
  "$SOURCE2" \
  "$SOURCE3" \
  "$SOURCE4" \
  -newer $HOME/filelist_lastBackup.txt -print > $EXTERNALDISKPATH/filelist_lastBackup.txt

# create local log of filelist
echo "" >> $HOME/filelist_allBackups.txt
echo $date >> $HOME/filelist_allBackups.txt

date=`date "+%Y-%m-%d_%H-%M-%S"`
echo "$date   done" >> $LOG
echo "$date Copying files..." >> $LOG

# copy changed files across...
CMD=`rsync -aP \
  --exclude-from=$HOME/exclude.txt \
  --files-from=$EXTERNALDISKPATH/filelist_lastBackup.txt \
  "$SOURCEROOT" $BACKUPDESTINATION 2>&1`

# error capture & report logging...
if [ $? != 0 ]
then
    echo "BACKUP ERROR: problem syncing $SOURCEROOT to $BACKUPDESTINATION"
    echo "BACKUP ERROR: problem syncing $SOURCEROOT to $BACKUPDESTINATION" >> $LOG
    echo $CMD >> $LOG
	echo "" >> $LOG
    exit 1
else
	 # on successful copy update reference file...
	echo "SUCCESS: syncing $SOURCEROOT to $BACKUPDESTINATION"
	echo "SUCCESS: syncing $SOURCEROOT to $BACKUPDESTINATION" >> $LOG
	cp -p $EXTERNALDISKPATH/filelist_lastBackup.txt $HOME/filelist_lastBackup.txt
	cat $HOME/filelist_lastBackup.txt >> $HOME/filelist_allBackups.txt
	echo $CMD
fi

# place filelist in correct folder (one level deeper)...
mv $EXTERNALDISKPATH/filelist_lastBackup.txt $BACKUPDESTINATION/filelist_lastBackup.txt

date=`date "+%Y-%m-%d_%H-%M-%S"`
echo "$date FINISHED" >> $LOG
echo "" >> $LOG
else
echo "$date Destination Path does not exist... skipping backup" >> $LOG
echo "" >> $LOG
fi

Ok, all seems to be going hunky dory with this approach, aside from one problem I'm experiencing:

 

If I rename a file (after it has been backed up) the change does not get reflected on the backup - because this method only backs up new/changed files, and renaming a file does not modify the file's modification date.

 

Any ideas how to ensure my backup filenames truly reflect the source?

 

If you are using the find command method. you could possibly try adding the -cnewer option also.

 

-cnewer file
File's status was last changed more recently than file was modified. -cnewer is affected by -follow only if -follow comes before -cnewer on the command line.

 

I do not know the exact syntax but possibly

 

-newer $HOME/filelist_lastBackup.txt -o -cnewer $HOME/filelist_lastBackup.txt

 

I don't know if this works, it's just an idea.

 

Additional review:

http://linuxmanpages.com/man1/find.1.php

 

an issue with this is deletes never get translated.

 

Thanks WeeboTech - that captures the renamed files.

Shame about deletes never getting translated, but that's not such a big concern for me - at least all the data is backed up in case of emergency.

 

 

Deletes would get translated if you did an rsync drive to drive copy,

But you're doing an incremental backup with adds/changes only, so the deletes never get handled.

 

I suppose they could if you did some additional fancy programming.

 

you would need to do a find down the whole source disk.

Transport that filelist.disk? to the remote system.

 

Then do a find down the matching disk? on the remote system.

 

Now you have two files.

 

With a diff you can see what is on the backup disk that is not on the master and delete it.

You can also see what is on the master disk that is not on the backup disk and act appropriately.

 

Actually with some foresight, you could change the mechanism and use the backup system's filelist as a catalog of sort, to create the new filelist on the source system.

 

You may want to look at unison, I think that keeps catalogs and may provide functionality that you could use.

 

 

-newer $HOME/filelist_lastBackup.txt -o -cnewer $HOME/filelist_lastBackup.txt

 

also, just realised that this works better:

 

-newer $HOME/filelist_lastBackup.txt , -cnewer $HOME/filelist_lastBackup.txt