im having trouble rsyncing over to another server

i have my 2nd server set to "Fill up " on the share

 

and the one drive is like 1gb free its supposed to be 50 gb  free

the error log shows

```

[sender] io timeout after 60 seconds -- exiting
rsync error: timeout in data send/receive (code 30) at io.c(200) [sender=3.2.7]

Rsync failure. Backing off and retrying...

```

 

when rsyncing  

does  the rsyncing  bypass rules of the share  like when you go through a root share when you make it it bypass's the rules of the minimum disk space of the share... so  if you set it to 50gb  it will go down to like 200kb  

 

so my code to rsync is

 

while [ 1 ]
do
#    rsync -avz --timeout=60 --partial source dest
    rsync -avzue  ssh --exclude-from='/boot/config/plugins/user.scripts/scripts/exclude1.txt' -s --stats --numeric-ids --progress '/mnt/user/Videos/' root@$serverip:'/mnt/user/Videos/' --delete --timeout=60 --partial

    if [ "$?" = "0" ] ; then
        echo
        echo "Rsync Of The Videos Completed Normally.. Ready To Take Back Home"
        echo
        exit
    else
        echo
        echo "Rsync failure. Backing off and retrying..."
        echo
        sleep 180
    fi
done

 

 

so is it bypassing the rules  and its trying to write to the 1 drive thats outta disk space

if i change to high water seems to still have same issue   keeps writting to the same drive  it will not move to the next drive  i even changed to 100gb free.. it keeps restarting rsync  but it drops down to 100kb  diskspace free or 1gb  free  why is it bypassig the gb rules

Edited December 22, 20241 yr by comet424

What is the split level set to? What happens with rsync is that it creates all the directories first before transferring any files. So if you have anything other then "Automatically split any dictreory as required" as split level, it will try to keep files together until it runs out of space as split level trumps allocation method. 

its set to the default  automaticlly  split any directory as required...

 

doesnt the --delete option on the rsync  also fix that so it can delete files and move them too

 

so i should set it to automatically split any directory as required....  i never know the right option for these so ive always left them at default

 

Do you have cache setup as the primary storage on the remote machine? Do you have any disks in the included/excluded setting and if yes what are those? If split level and the included/excludes settings are set correctly it should hop to the next drive when the minimum space is reached. Maybe post your diagnostics?

Edited December 22, 20241 yr by strike

no cache no include exclude  its set to all... its my backup media server  i had done a

```

for /r %A IN (*.*) do ( "C:\Program Files\MKVToolNix\mkvpropedit.exe" -d title --tags all: "%~A")

```

to fix all my videos i have some dont fix but thats ok...  and then i wanted to re sync to my backupserver of the media videos  so i figured it delete the old file and re copy over the new one updated  with that --delete option

 

it keeps filling the one drives and then rsync craps out with error  always same file  till i move files to a free drive  of more space then it starts resumming fine  almost better  wipe all the the folders and let it re sync  but that wasnt the purpose  figured it could rysnc fine

 

i figured setting limit from 50gb to 100gb  and moving more files off each drive  would help  nope still gets stuck

mitchflix-diagnostics-20241222-0938.zip

im trying to reboot my first server sending the the rsync..  even though i have like 140gb free on each drive  on te remote roughly  it keeps crapping out on the same file when it reachs it in  in the log file  so that shouldnt be happening so i figuring do a clean reboot on the main server see what happens

so after a reboot of the main server  and i run my script...  it rysncs the same 13 files  copies ok  but when gets to the one   it restarts   and starts at the first file it started with  its doing a viscious circle...

here is the diagnostic of the main server..  maybe you see something thats causing it to keep  getting stuck in endless loop... or if i just have to just wipe the folder directory  and re start rsync...  or is there a an attritube to add to rsync  to fix it... i did run scrub on all the drives  but that came up nothing

 

 

rambo-diagnostics-20241222-1034.zip

15 minutes ago, comet424 said:

so after a reboot of the main server  and i run my script...  it rysncs the same 13 files  copies ok  but when gets to the one   it restarts   and starts at the first file it started with  its doing a viscious circle...

As I look at your script (and it has been may long years since I wrote any BASH scripts), It looks like that is what you wanted:

 

Did you try with just the rsync command to do the copy using just the Command line?

 

 

I use the following command line to update/backup the new and changed files:

rsync -avhPX /mnt/remotes/192.168.1.245_Media/All\ Movies/ /mnt/user/BackupMedia/All\ Movies/

 

 

i had copied this rsync script from i think a redit link...  as i wanted something that rsync resumes as i found there was nothing that resumes  if rsyncs cancels  it doesnt resume  on its own  

 

so you use the remote share right?  and what does the avhPX  all do  

and ok ill run it in the terminal and see what it spits out  normally 

if it craps out on same file

25 minutes ago, comet424 said:

and what does the avhPX  all do  

and ok ill run it in the terminal and see what it spits out  normally 

if it craps out on same file

 

Look here for what all of the rsync options are:

 

     https://ss64.com/bash/rsync_options.html

 

The 'X' option is the only one that probably not necessary but (like you) I got those parameters from another source when I was converting data disks formats from Reiserfs to XFS.

 

 

ah ok and when i ran it in the terminal  it started copying the files again  the several before and when it got to the one it gets stuck on it said 12 hours at a 60k/s  transfer and timed out   

 

when i moved that file out of the folder to a different root folder of the folder it was in  and i ran it.. then it rsynced that file file first fine  i wonder what causes that?  so at moment i running through terminal so i can see when it craps out  on the last file  since i thought it was working fine for couple days  only to find out its been looping on the same file  couple days