Like the topic says, I want to clone my existing unraid box, then move it offsite.  Then whenever my unraid changes have it duplicate the change somehow to the offsite box.  The offsite box will not have network access. 

 

My incremental changes will be at most a couple hundred gigabytes.  I'm thinking plug an external hdd into the unraid box, and have any changes made to the server copied to the external hdd.  Then once a week or so, I take the external hdd to the remote location and that box then copies the changes over. 

 

Does anybody know a way of doing this without my having to kludge together a bunch of scripts and cron jobs?  I've been searching the forums with a little bit of googling thrown in, but it seems like most backup solutions are based around network access. 

 

Thanks.

Like the topic says, I want to clone my existing unraid box, then move it offsite.  Then whenever my unraid changes have it duplicate the change somehow to the offsite box.  The offsite box will not have network access. 

 

My incremental changes will be at most a couple hundred gigabytes.  I'm thinking plug an external hdd into the unraid box, and have any changes made to the server copied to the external hdd.  Then once a week or so, I take the external hdd to the remote location and that box then copies the changes over. 

 

Does anybody know a way of doing this without my having to kludge together a bunch of scripts and cron jobs?  I've been searching the forums with a little bit of googling thrown in, but it seems like most backup solutions are based around network access. 

 

Thanks.

Just a thought, but if you use cache drive one can modify the mover script to copy to a mounted external hdd before moving to array.  Keep modified mover script on flash and use go script to replace on boot.  That will cover only new stuff, but not files that are only changed.  For changed stuff talk with squid about his checksum suit.  Might not be too difficult to piggyback or add feature to copy on detected changes.

Thats actually a pretty good idea and would cover 99% of my use case.  How would I get it to then copy off of the external and onto the off site system?  I guess I could manually copy the external to the cache drive. 

How would I get it to then copy off of the external and onto the off site system?

 

There's bound to be some syncing software which can do "New device detected, copying everything!".

Thats actually a pretty good idea and would cover 99% of my use case.  How would I get it to then copy off of the external and onto the off site system?  I guess I could manually copy the external to the cache drive.

 

Can be done manually like you mentioned, or modify the mover script on the offsite server to move from mounted external hdd to array since the folder/share structure would be identical.  It's rather crude/low-tech, but pretty simple to implement.

Ok, so I've finally gotten around to starting this project.  basically I have two portable USB HDDs that I rotate out to the remote site.  Carry and Carry2. 

 

Here is my modified mover script:

 

#!/bin/bash

# This is the 'mover' script used for moving files from the cache disk to the
# main array.  It is typically invoked via cron.

# After checking if it's valid for this script to run, we check each of the top-level
# directories (shares) on the cache disk.  If, and only if, the 'Use Cache' setting for the
# share is set to "yes", we use 'find' to process the objects (files and directories) of
# that directory, moving them to the array.

# The script is set up so that hidden directories (i.e., directory names beginning with a '.'
# character) at the topmost level of the cache drive are also not moved.  This behavior can be
# turned off by uncommenting the following line:
# shopt -s dotglob

# Files at the top level of the cache disk are never moved to the array.

# The 'find' command generates a list of all files and directories on the cache disk.
# For each file, if the file is not "in use" by any process (as detected by 'fuser' command),
# then the file is copied to the array, and upon success, deleted from the cache disk.
# For each directory, if the directory is empty, then the directory is created on the array,
# and upon success, deleted from the cache disk.

# For each file or directory, we use 'rsync' to copy the file or directory to the array.
# We specify the proper options to rsync so that files and directories get copied to the
# array while preserving ownership, permissions, access times, and extended attributes (this
# is why we use rsync: a simple mv command will not preserve all metadata properly).

# If an error occurs in copying (or overwriting) a file from the cache disk to the array, the
# file on the array, if present, is deleted and the operation continues on to the next file.

# Only run script if cache disk enabled and in use
if [ ! -d /mnt/cache -o ! -d /mnt/user0 ]; then
  exit 0
fi

# If a previous invokation of this script is already running, exit
if [ -f /var/run/mover.pid ]; then
  if ps h `cat /var/run/mover.pid` | grep mover ; then
      echo "mover already running"
      exit 0
  fi
fi
echo $$ >/var/run/mover.pid
echo "mover started"

cd /mnt/cache
shopt -s nullglob
for Share in */ ; do
  if grep -qs 'shareUseCache="yes"' "/boot/config/shares/${Share%/}.cfg" ; then

    echo "moving \"${Share%/}\""
    	# Only run script if carry disk enabled and in use
if [ ! -d /mnt/disks/Carry ] && [ ! -d /mnt/disks/Carry2 ]
then
	echo "Carry is not here"  		
	exit 0
else
	if [ -d /mnt/disks/Carry ] ; then 
		echo "Carry is here"
		find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \
      			\( -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} /mnt/disks/Carry/ \; -delete \)
	fi
	if [ -d /mnt/disks/Carry2 ] ; then
		echo "Carry2 is here"
		find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \
      			\( -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} /mnt/disks/Carry2/ \; -delete \)
	fi
fi

    find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \
      \( -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} /mnt/user0/ \; -delete \) -o \( -type f -exec rm -f /mnt/user0/{} \; \)
  else
    echo "skipping \"${Share%/}\""
  fi
done

rm /var/run/mover.pid
echo "mover finished"

 

This works, however I get a chown permissions when running the script.  The exact message is:

 

./Eric/krusaderEt4029.tmp
rsync: chown "/mnt/disks/Carry/." failed: Operation not permitted (1)
.d..t.og... ./
rsync: chown "/mnt/disks/Carry/Eric" failed: Operation not permitted (1)
.d..t.og... Eric/
>f+++++++++ Eric/krusaderEt4029.tmp
rsync: chown "/mnt/disks/Carry/Eric/krusaderEt4029.tmp" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.0]
./Eric/krusaderEt4029.tmp
.d..t...... ./
.d..t...... Eric/
>f+++++++++ Eric/krusaderEt4029.tmp

 

I dont know why I am gettting that error.  I think it is something about not being able to change the user of the files as it copies them? I also get a similar error using the daily backup script posted  in the Unassigned Devices forum. https://lime-technology.com/forum/index.php?topic=45807.0

 

I also added this line to my go script:

 

cp -p /boot/config/mover /usr/local/sbin/mover

 

and of course I copied my modified mover script to the flash drive. 

 

So far, everything appears to be working in test except for the above error.  Can anybody help me resolve that?  I will make some test changes to the array and report back in the morning how things are working. 

 

Does anybody have any suggestions to make this cleaner?

 

Thanks for the help unevent.  You gave me just enough that I was able to get things moving.

 

(edited to fix a bug in the mover script for anybody else that wants to use this)

Hello again all. 

 

So I have been using the scripts above for a couple of weeks now, and I am having a few problems.  I am hoping someone here would be able to help me out. 

 

When I unmount Carry or Carry2 via the Unassigned Devices UI, I get a phantom folder at /mnt/disks/Carry[2].  I can browse the folders and look at the file names but if I try to do anything to them, such as copy or delete, I get "Input/Output error (5)"  I am assuming that is because the disk is not physically mounted, but that the file system is still retaining the mount points.  If that makes any sense. 

 

Does anybody know why it is doing this?  And how I can prevent it?

 

Second issue I am gettting is a chown permissions error as detailed above.  I think it has something to do with the tags on the rsync command, but in theory shouldn't a script running as root be able to do anything it wants to the files?  Sorry for the questions, I am still a novice when it comes to Linux file permissions.