rsync Directory Creation Question

[solidus28]

Good day!  I have a question about the order that rsync does things.  I can share my rsync commands if needed.  I'm hoping this is a simple question :)

 

When I run an rsync between my main and backup Unraid servers, I noticed as I got lower on space that if I was doing a large sync, it would fill up one drive on the backup server rather than respecting my split levels.  What I think I observed is that rsync starts off by creating all of the empty directories, so technically, Unraid is respecting my split level settings, but since all of the directories got configured before the files inside them, they aren't given a chance to split to other drives.

 

Is there a way to have rsync create one directory and it's files/sub directories at a time?  I've resorted to performing a copy/paste from my Windows workstation between each Unraid share as this appears to one directory at a time, therefore actually moving to a new disk as it hits the split level settings.  Make sense?

 

If I'm doing small backups, no big deal, but if I end up with a TB of data, it can cause issues if rsync sticks it all on one disk instead of allowing unraid to divvy files up.

 

Thanks in advance!

[solidus28]

Bumping to see if anyone has any input.  Thanks!

[JorgeB]

On 7/30/2018 at 4:38 PM, solidus28 said:

Is there a way to have rsync create one directory and it's files/sub directories at a time?

Not AFAIK, I searched for a way to do it a while back and found none.

[solidus28]

Ok, thanks for your feedback   

[airbillion]

I have had the same issue and was wondering if there was a way to do this also...

Sent from my SM-N950U using Tapatalk

[apandey]

I ran into the exact same issue. My solution was to use rsync to only itemize the changes but then apply them myself in a script. While applying, I skip directory creating items until I have a file to be written. I used user scripts plugin to manage the sync

 

here is an explanation of rsync itemize - http://www.staroceans.org/e-book/understanding-the-output-of-rsync-itemize-changes.html

 

my example script

function texe() {
  if [ "${run_type}" == "dry-run" ]; then
    echo "dry-run: $@"
  elif [ "${run_type}" == "real-run" ]; then
    "$@"
  fi
}

rsync --dry-run --recursive --itemize-changes --delete --delete-excluded --iconv=utf-8 \
	--exclude '@eaDir' --exclude 'Thumbs.db' $src $dst | while read -r line ; do
    echo "$line"
    read -r op file <<< "$line"

    if [ "x$op" == "x*deleting" ]; then
      log "removing $dst/$file"
      texe rm -rf "$dst/$file"
    else
      op1=$(echo $op | cut -b 1-2)
      sizeTsState=$(echo $op | cut -b 4-5)

      if [ "x$op1" == "xcd" ]; then
	    echo "not eagerly creating $dst/$file"
	    #mkdir -p "$dst/$file"
      elif [ "x$op1" == "x>f" ]; then
        fpath="$dst/$file"
        dpath=$(dirname "$fpath")
	    if [ "x$sizeTsState" == "x.T"  ]; then
	      log "update $fpath timestamp only"
	      texe touch -r "$src/$file" "$fpath"
    	elif [ "x$sizeTsState" != "x.."  ]; then
	      if [ ! -d "$dpath" ]; then
	        texe sudo -u nobody mkdir -v -m 777 -p "$dpath"
	      fi
	      texe install -o nobody -g users -m 666 -p -D -v "$src/$file" "$fpath"
	    fi
      fi
    fi
done