Jump to content

rsync Directory Creation Question

Recommended Posts

Good day!  I have a question about the order that rsync does things.  I can share my rsync commands if needed.  I'm hoping this is a simple question :)


When I run an rsync between my main and backup Unraid servers, I noticed as I got lower on space that if I was doing a large sync, it would fill up one drive on the backup server rather than respecting my split levels.  What I think I observed is that rsync starts off by creating all of the empty directories, so technically, Unraid is respecting my split level settings, but since all of the directories got configured before the files inside them, they aren't given a chance to split to other drives.


Is there a way to have rsync create one directory and it's files/sub directories at a time?  I've resorted to performing a copy/paste from my Windows workstation between each Unraid share as this appears to one directory at a time, therefore actually moving to a new disk as it hits the split level settings.  Make sense?


If I'm doing small backups, no big deal, but if I end up with a TB of data, it can cause issues if rsync sticks it all on one disk instead of allowing unraid to divvy files up.


Thanks in advance!

Link to comment
  • 4 years later...

I ran into the exact same issue. My solution was to use rsync to only itemize the changes but then apply them myself in a script. While applying, I skip directory creating items until I have a file to be written. I used user scripts plugin to manage the sync


here is an explanation of rsync itemize - http://www.staroceans.org/e-book/understanding-the-output-of-rsync-itemize-changes.html


my example script

function texe() {
  if [ "${run_type}" == "dry-run" ]; then
    echo "dry-run: $@"
  elif [ "${run_type}" == "real-run" ]; then

rsync --dry-run --recursive --itemize-changes --delete --delete-excluded --iconv=utf-8 \
	--exclude '@eaDir' --exclude 'Thumbs.db' $src $dst | while read -r line ; do
    echo "$line"
    read -r op file <<< "$line"

    if [ "x$op" == "x*deleting" ]; then
      log "removing $dst/$file"
      texe rm -rf "$dst/$file"
      op1=$(echo $op | cut -b 1-2)
      sizeTsState=$(echo $op | cut -b 4-5)

      if [ "x$op1" == "xcd" ]; then
	    echo "not eagerly creating $dst/$file"
	    #mkdir -p "$dst/$file"
      elif [ "x$op1" == "x>f" ]; then
        dpath=$(dirname "$fpath")
	    if [ "x$sizeTsState" == "x.T"  ]; then
	      log "update $fpath timestamp only"
	      texe touch -r "$src/$file" "$fpath"
    	elif [ "x$sizeTsState" != "x.."  ]; then
	      if [ ! -d "$dpath" ]; then
	        texe sudo -u nobody mkdir -v -m 777 -p "$dpath"
	      texe install -o nobody -g users -m 666 -p -D -v "$src/$file" "$fpath"


Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Create New...