unRAID OS Version: V6b9

 

Description: Under some circumstances you end up with a partial copy of a file on the array and the full copy on the cache drive. This may be perhaps due to running out of space etc. When this happen the mover will try every time to rsync the file and fail ad nausium. It is not ideal to have two copies of the same file especially when the one on the protected array is faulty in some way.

 

What is especially interesting is if you clear plenty of space on the array the mover will still fail forever until you manually delete the file on the array.

 

How to reproduce: Untested: Fill a user share to nearly 99% then kick of the mover. When this fails you will see it fail every time you run the mover.

 

Expected results: Some sort of feedback in the GUI of a problem with the mover. When space is available the mover should complete. Ideally partial broken files should not be left; better the unprotected cache file is the sole copy.

 

Actual results:

 

Other information:

 

Sep 23 22:42:07 SOLO logger: moving tv.hd/
Sep 23 22:42:07 SOLO logger: ./tv.hd/H/Houdini/Houdini.S01E02..mkv
Sep 23 22:42:07 SOLO logger: .d..t...... tv.hd/
Sep 23 22:42:07 SOLO logger: >f.stpog... tv.hd/H/Houdini/Houdini.S01E02.mkv
Sep 23 22:42:23 SOLO shfs/user0: shfs_write: write: (28) No space left on device
Sep 23 22:42:23 SOLO shfs/user0: shfs_write: write: (28) No space left on device
Sep 23 22:42:26 SOLO logger: rsync: write failed on "/mnt/user0/tv.hd/H/Houdini/Houdini.S01E02..mkv": No space left on device (28)
Sep 23 22:42:26 SOLO logger: rsync error: error in file IO (code 11) at receiver.c(389) [receiver=3.1.0]

 

Note there is a couple of hundred gig available on that share when this log is created.

I was seeing this on mine too. Reboot and run the mover again and it seems to work that time    I haven't check for dupe files yet.

I wouldn't think you would have any duplicates since the mover script is just using rsync and should over write the file on a new run. 

 

I don't think rsync cleans any work files up when it aborts.  Normally it writes to a .file, but the mover is using an inplace, so it is updating the file directly.

I just had the same issue on v610a.  Rebooting and then running mover gave me the same result.  I removed the partial files from the array and tried mover again with the same results.  The issue was that the partial files were sitting on two disks in the array that had reached capacity albeit the array itself had plenty of space.  Solved issue by moving enough files from the full disks to allow mover to move the files to the disks it was initially trying to move to.

...  The issue was that the partial files were sitting on two disks in the array that had reached capacity albeit the array itself had plenty of space.  ...

 

That actually makes a lot of sense I think you have nailed it.

Updating this for all versions form b9 to current b14

 

Fundamentally the issue is that if the mover script fails due to out of disk space rsync leaves a partial copy on the now full destination.

 

This essentially locks that mover action to this now full disk and unless the partial file is manually removed or enough space is cleared from this specific disk the mover will fail forever. This is doubly confusing for the user who sees that the user share may have lots of free space but the mover is complaining that there is none.

 

The bug fix is actually quite simple in the end namely that mover/rsync should not leave incomplete copies on failed actions (for reasons that extend beyond this specific bug)

After running into the "cache didn't empty last night" feature, I must respectfully non-concur with the expected action(s) as stated earlier.

 

I expected the mover to find a disk location having unused capacity sufficient to adhere to the split rules, and locate contents accordingly (even if it meant moving existing data from one data disk to another before moving data from the cache).

 

I would not have cared which data ended up on which data disk: the mover could have chosen any files to move according to any rule set, as long as the split rules were followed.

 

Of course, any partial files _should_ (shall) be removed.

 

Eight TB into a 15 TB move from one server architecture to another (unRAID).  First issue I've seen...well done!

I expected the mover to find a disk location having unused capacity sufficient to adhere to the split rules, and locate contents accordingly (even if it meant moving existing data from one data disk to another before moving data from the cache).

I am afraid that mover is not that intelligent.  Once a file has been placed on a  disk it is NEVER moved except under explicit user action.

 

Of course, any partial files _should_ (shall) be removed.

 

 

My understanding is that the Mover script invokes Rsync with the --inplace flag to do it's moves.

The side effect of which is that --inplace "The option implies --partial (since an interrupted transfer does not  delete  the  file)" - Manpage

 

If someone can think of a better setup that still achieves the goals of mover for mover then perhaps Lime-Tech can update the script.