Jump to content
falcor

One-liner to repair rsync empty directories

1 post in this topic Last Reply

Recommended Posts

It'd be cool if this was some sort of plugin or something, but I recently had a problem where rsync created ALL the directories before moving any files for an 8TB or so network copy, and proceeded to completely fill a 5TB disk until writes failed. Apparently the reason the high-water-mark setting didn't work was the first disk had enough space for all the directory entries, so the entire folder structure ended up on /mnt/disk3. I had it set to only split on top-level directories since this was for TV shows and I want all seasons to be contained in one drive only for each show. After some reading, I decided I needed to shard (love that word 💩) the folders to other disks, which would maintain the view under /mnt/user but allow rsync to put different shows on different drives. I came up with this, run from disk3's tv root:

 

find . -type d -empty -maxdepth 1 -print0 | \
  while IFS= read -r -d '' file; do
    DISK=$((5 + $RANDOM % 3)); 
    mkdir -p /mnt/disk${DISK}/tv && mv "$file" /mnt/disk${DISK}/tv/; 
  done

This successfully put approximately the same number of subdirectories on each disk. Just thought I'd share and see if anyone else has solved this another way or has any suggestions. I think it'd be good if shfs could split out empty directories smartly, but that might be too much to ask or be technically impossible.

Edited by falcor
fixing code

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.