solidus28 Posted July 30, 2018 Share Posted July 30, 2018 Good day! I have a question about the order that rsync does things. I can share my rsync commands if needed. I'm hoping this is a simple question :) When I run an rsync between my main and backup Unraid servers, I noticed as I got lower on space that if I was doing a large sync, it would fill up one drive on the backup server rather than respecting my split levels. What I think I observed is that rsync starts off by creating all of the empty directories, so technically, Unraid is respecting my split level settings, but since all of the directories got configured before the files inside them, they aren't given a chance to split to other drives. Is there a way to have rsync create one directory and it's files/sub directories at a time? I've resorted to performing a copy/paste from my Windows workstation between each Unraid share as this appears to one directory at a time, therefore actually moving to a new disk as it hits the split level settings. Make sense? If I'm doing small backups, no big deal, but if I end up with a TB of data, it can cause issues if rsync sticks it all on one disk instead of allowing unraid to divvy files up. Thanks in advance! Quote Link to comment
solidus28 Posted August 3, 2018 Author Share Posted August 3, 2018 Bumping to see if anyone has any input. Thanks! Quote Link to comment
JorgeB Posted August 4, 2018 Share Posted August 4, 2018 On 7/30/2018 at 4:38 PM, solidus28 said: Is there a way to have rsync create one directory and it's files/sub directories at a time? Not AFAIK, I searched for a way to do it a while back and found none. Quote Link to comment
solidus28 Posted August 6, 2018 Author Share Posted August 6, 2018 Ok, thanks for your feedback Quote Link to comment
airbillion Posted August 7, 2018 Share Posted August 7, 2018 I have had the same issue and was wondering if there was a way to do this also...Sent from my SM-N950U using Tapatalk Quote Link to comment
apandey Posted November 15, 2022 Share Posted November 15, 2022 I ran into the exact same issue. My solution was to use rsync to only itemize the changes but then apply them myself in a script. While applying, I skip directory creating items until I have a file to be written. I used user scripts plugin to manage the sync here is an explanation of rsync itemize - http://www.staroceans.org/e-book/understanding-the-output-of-rsync-itemize-changes.html my example script function texe() { if [ "${run_type}" == "dry-run" ]; then echo "dry-run: $@" elif [ "${run_type}" == "real-run" ]; then "$@" fi } rsync --dry-run --recursive --itemize-changes --delete --delete-excluded --iconv=utf-8 \ --exclude '@eaDir' --exclude 'Thumbs.db' $src $dst | while read -r line ; do echo "$line" read -r op file <<< "$line" if [ "x$op" == "x*deleting" ]; then log "removing $dst/$file" texe rm -rf "$dst/$file" else op1=$(echo $op | cut -b 1-2) sizeTsState=$(echo $op | cut -b 4-5) if [ "x$op1" == "xcd" ]; then echo "not eagerly creating $dst/$file" #mkdir -p "$dst/$file" elif [ "x$op1" == "x>f" ]; then fpath="$dst/$file" dpath=$(dirname "$fpath") if [ "x$sizeTsState" == "x.T" ]; then log "update $fpath timestamp only" texe touch -r "$src/$file" "$fpath" elif [ "x$sizeTsState" != "x.." ]; then if [ ! -d "$dpath" ]; then texe sudo -u nobody mkdir -v -m 777 -p "$dpath" fi texe install -o nobody -g users -m 666 -p -D -v "$src/$file" "$fpath" fi fi fi done Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.