@ronia Isn't it currently the case with Unraid that if a file exists on user0 and cache, then with /mnt/user/ it prefers cache?

3 hours ago, ronia said:

So basically the torrent will continue to seed off of the cache drive until presumably the user hits the tracker's minimum seed requirements and then the cache copy will also be deleted.  So there is an instance in time in which both an array copy and a cache copy will exist.  The cache copy eventually gets deleted on its own, presumably through qbit_manage or manual process.

 

I can see where this would be useful for some people.  I didn't think of this use case as I perma-seed my content for bon.  I'm not a racer so this didn't even occur to me.

Exactly! You've got it.

 

Essentially, the option could be something like this:

 

======

Always break hardlinks to move files if necessary

 

WARNING: setting to 'Yes' may result in multiple copies of the same file, i.e. two copies of the same file in different paths on the array, or one copy on cache and one copy on array of the same file (this may be useful for cache seeding).

 

You will know if you want this option - most people will keep this set to 'No' (default) which will:

- identify hardlinks

- preserve/recreate hardlinks as they were

- prevent multiple copies of the same file on the same drive

======

 

'No' would be default, how the mover runs (or is intended to run) currently.

 

'Yes' would need to do three things. 1) remove the --hard-link flag (this is a good safety measure, although it shouldn't make a difference, this should help ensure no edge case where multiple copies of the same file is created on the destination drive); and 2) remove at least the log output from this chunk:

6 hours ago, ronia said:

age_mover: processTheMoves() ln1400:

            # Check if list is complete
            NBFILES=$(($(echo "$SYNCED_FILES" | grep -o '" "' | wc -l) + 1))
            if [ $NBFILES != $NBLINKS ]; then
                mvlogger "Expected $NBLINKS files, got $NBFILES. Not moving $SYNCED_FILES to prevent breaking hardlinks"
                continue
            fi

 

But this is only for the log output to not be confusing if someone has set to 'No' - this would not change the functionality.

 

This brings us to 3) the change would have to lie in a different place. It is likely that there is a find command or something else happening that is building a file list, including all hardlinks and inode info, before rsync with chosen exclusions, such as hidden folders or an exclusion list, scans and runs. Normally, rsync on its own does not scan excluded folders, so it wouldn't even find a hardlink in .downloads to skip it because it's also in media/*, it would just process it like a normal file. It would have to have gotten the info elsewhere to see a mismatch before moving, so it skips the file because of that. This is the kicker, and this is what would have to be changed or allowed somehow, if this option were to be possible.

 

I can try to explain it differently or with different examples but I hope that is making a bit more sense now from my end. It's challenging to communicate it.

Edited April 11, 20251 yr by jv6478
duplicate word, formatting

Hmm, IIRC it always worked like that, and I was specifically using a jdupes in a crontab every night to create new hardlinks for files moved to array

20 minutes ago, silverkin said:

@ronia Isn't it currently the case with Unraid that if a file exists on user0 and cache, then with /mnt/user/ it prefers cache?

Yes this is the case and I think I know where you're going with this.  I actually thought that @jv6478 could use this,

 

 

But I suspect this wouldn't work since then you'd have the opposite problem as the library file would then need to be deleted at some point.  Or else you'd just forever have a cache copy and an array copy.

Just now, ronia said:

Yes this is the case and I think I know where you're going with this.  I actually thought that @jv6478 could use this,

 

 

But I suspect this wouldn't work since then you'd have the opposite problem as the library file would then need to be deleted at some point.  Or else you'd just forever have a cache copy and an array copy.

The problem is that currently this option doesn't seem to be working (for me at least) and I have raised an issue on this in this topic and also on GitHub.

5 minutes ago, ronia said:

But I suspect this wouldn't work since then you'd have the opposite problem as the library file would then need to be deleted at some point.  Or else you'd just forever have a cache copy and an array copy.

Imo, that could be a good solution to always have array "synchronized", so all of the files would be laying on array and cache simultaneously, but if cache is going to hit a mover threshold, then this data is just going to evaporate from cache, as it's already on array.

 

That's how I planned to use this functionality, to let me fully utilize the cache to prevent disks from spinning as long as I can and also have a cache data protected by parity

Edited April 11, 20251 yr by silverkin

1 hour ago, silverkin said:

Imo, that could be a good solution to always have array "synchronized", so all of the files would be laying on array and cache simultaneously, but if cache is going to hit a mover threshold, then this data is just going to evaporate from cache, as it's already on array.

 

That's how I planned to use this functionality, to let me fully utilize the cache to prevent disks from spinning as long as I can and also have a cache data protected by parity

This is how I understood it as well, and I feel like it's useful for something like that, but not for the granular control that I'm looking for.

  

1 hour ago, ronia said:

But I suspect this wouldn't work since then you'd have the opposite problem as the library file would then need to be deleted at some point.  Or else you'd just forever have a cache copy and an array copy.

 

That's right, and it wouldn't address my particular case as you mentioned.

 

I tested on my end with a custom rsync script, bypassing the internal mover process, and it works exactly as I intend it when ignoring a file that contains a hardlink. It moves what it finds as I need and expect. I haven't tested with the --hard-link flag, but I presume it wouldn't make a difference in my case because I'm excluding hidden folders before anything is scanned. That's important to be able to make it an option - ensuring there is no passing of hardlinked files to the instruction before rsync scans (this is kind of a double check, where if rsync's scan doesn't find the expected files based on the initial list that's created, it will skip the file).

 

Because I excluded the location where the hardlink resides, rsync moved the file in /media normally because it didn't find a hardlink at all. This means, whatever mechanism (find, etc.) is building a list before the rsync command, needs to be edited to include the exclusions based on the options selected in the plugin (hidden folders, exclusion list, etc.) so the list is built without those files and rsync doesn't see a mismatch when processing. I think this is the key.

Edited April 12, 20251 yr by jv6478
Adding more context

15 hours ago, silverkin said:

The problem is that currently this option doesn't seem to be working (for me at least) and I have raised an issue on this in this topic and also on GitHub.

As we found in github this option is working fine. 

13 hours ago, jv6478 said:

Because I excluded the location where the hardlink resides, rsync moved the file in /media normally because it didn't find a hardlink at all.

I think you have very special case, not sure much users will want this option as I'm afraid of if this option will be changed by user accidentally will break stuff. 

So maybe in your case better to use some script befor/after? 

1 hour ago, Masterwishx said:

As we found in github this option is working fine. 

Intresting, doesn't seem to be working for me either. 

What's the github url, like to read the discussion. Perhaps I mis configured it. 

2 hours ago, Masterwishx said:

As we found in github this option is working fine. 

Yes, sorry for not correcting it here.
 

@BJZwart https://github.com/masterwishx/ca.mover.tuning/issues/50

 

Regarding the hardlink issue - @jv6478 just tested, and that synchronize option works perfectly for my needs - it's leaving stuff on cache, copying it to array and placing the hardlinks there. So e.g. have a Linux ISO on /media and /download on cache with a hardlink and also on array with different inode, but with a hardlink too.

 

@Masterwishx How the script is going to behave after it's going to meet the mover threshold after synchronizing data to array? Is it going to replace the files, or is it hash checking and simply removing files from cache? Wanted to know if there is no double writing with that method :).

Edited April 12, 20251 yr by silverkin

5 minutes ago, silverkin said:

How the script is going to behave after it's going to meet the mover threshold after synchronizing data to array

Not really checked it yet , from what I remember it's saving time of last synchronise to config file to be checked next time, so synchronise work only for files that was changed like wrote in description. 

4 hours ago, Masterwishx said:

Not really checked it yet , from what I remember it's saving time of last synchronise to config file to be checked next time, so synchronise work only for files that was changed like wrote in description. 

Synchronize yes, bur what about the mover if the files are already present on array due to synchronization. Is it going to overwrite them (double write of single file, because of the previous synchronization) or compare and just delete from cache?

24 minutes ago, silverkin said:

Synchronize yes, bur what about the mover if the files are already present on array due to synchronization. Is it going to overwrite them (double write of single file, because of the previous synchronization) or compare and just delete from cache?

You mean if disabled synchronise? 

Yes I think it will move them but 

You check it with test = yes maybe..

38 minutes ago, Masterwishx said:

You mean if disabled synchronise? 

Yes I think it will move them but 

You check it with test = yes maybe..

When it is enabled, I am talking about scenario, when until mover threshold (let's say 90%) the files are going to be just copied to array (synchronized) and when threshold is going to be met, then what's going to happen, is mover going to overwrite or compare them?

//Edit, when tested that, it said in the logs of course the move command, but still have no idea how it's gonna behave, what's the exact mover command? Is the compare flag present in the rsync?

//Edit2: It should work by default, the mtime is the same on both of files, the synchronized one on array and the one on the cache, so "mover" should ignore them, and just simply remove from cache

Edited April 12, 20251 yr by silverkin

43 minutes ago, silverkin said:

When it is enabled, I am talking about scenario, when until mover threshold (let's say 90%) the files are going to be just copied to array (synchronized) and when threshold is going to be met, then what's going to happen, is mover going to overwrite or compare them?

OK I got it. 

When threshold is above it will move to primary if find command + settings will find files on pool, but really not tested it. (not checking files on array as I remember) 

Hey. I used to have it set to move only files older than 60 days. but my cache kept filling. Now i've set it to 15 days and ran the Mover. But nothing is moving...
The logs just show .list files of files?
The cachepool is btrfs.

Lots of files are definitely old enough to move:
-rw-rw-rw- 1 nobody users 7045601667 Feb 25 23:01

I noticed i was yet again using the deprecated version....lets see if it moves files now.

 

Working now.

Edited April 13, 20251 yr by Kees Fluitman

For the life of me, I cannot figure out why mover isn't moving files from my data share from cache > array. Can anyone help me interpret my logs?

 

Unraid Mover logs

12 minutes ago, Ootspunter said:

For the life of me, I cannot figure out why mover isn't moving files from my data share from cache > array. Can anyone help me interpret my logs?

 

Update to the lasted version and delete other/all `loggingFolder="/tmp"` in config file, and run again mover

also delete old overidde configs for cache:prefer

Edited April 13, 20251 yr by Masterwishx

16 minutes ago, Masterwishx said:

 

Update to the lasted version and delete other/all `loggingFolder="/tmp"` in config file, and run again mover

also delete old overidde configs for cache:prefer

 

That did it, thanks!

Edited April 13, 20251 yr by Ootspunter

12 minutes ago, Ootspunter said:

Thanks! How do I get to the config file in this case ?(sorry, I'm still a bit new to unraid)

 

`/boot/config/plugins/ca.mover.tuning/`

On 4/13/2025 at 1:08 AM, Masterwishx said:

OK I got it. 

When threshold is above it will move to primary if find command + settings will find files on pool, but really not tested it. (not checking files on array as I remember) 

Tested this, and it works perfectly, it just checks if files are present on the array, comparing mtime, and then just deleting the stuff from cache, so there is no double write!

50 minutes ago, silverkin said:

Tested this, and it works perfectly

Cool, sorry for mistake

Running the mover like normal when I noticed this error:

 

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.4.1]

log: https://pastebin.com/TAjcQdGx

Any idea why I got this error?

1 hour ago, Triiiple said:

Any idea why I got this error?

In your log:

 

"/mnt/user0/data/torrents/radarr-uhd/The.Bikeriders.2023.REPACK.UHD.BluRay.2160p.TrueHD.Atmos.7.1.DV.HEVC.HYBRID.REMUX-FraMeSToR/.The.Bikeriders.2023.REPACK.UHD.BluRay.2160p.TrueHD.Atmos.7.1.DV.HEVC.HYBRID.REMUX-FraMeSToR.mkv.zruUMe" failed: No such file or directory (2)