1 hour ago, Gorosch said:

The latest. On Unraid V7

I will check it

5 hours ago, JayBriers said:

This seems to resolve the issue, as the strings in the subsequent find command are now double quotes.

Will check it 

19 hours ago, JayBriers said:

This seems to resolve the issue, as the strings in the subsequent find command are now double quotes.

 

What exactly issue do you have or error in log ? Cant reproduce it

20 hours ago, JayBriers said:

I don't know if this is something that can be incorporated into the plugin itself, or not, but I thought I'd fire some feedback over.

Thanks for the tip, if you mean ` eval: line 791: unexpected EOF while looking for matching `''`   ,will be fixed in next version

3 hours ago, Masterwishx said:

Thanks for the tip, if you mean ` eval: line 791: unexpected EOF while looking for matching `''`   ,will be fixed in next version

Apologies, I should have put the error message in too. 

The message I am getting is:
 

/usr/local/emhttp/plugins/ca.mover.tuning/age_mover: eval: line 739: syntax error near unexpected token `('

Essentially errorring at the next open bracket after the single quote, which would be in the next filepath. 

If I limit my ignore file to only two files, the first one being one with a single quote, the find str looks like 
 

find '/mnt/hddcache/media' -type f -depth  -not -name '.placeholder' -not -name '/mnt/hddcache/media/TV/8 Out of 10 Cats Does Countdown (2013) {imdb-tt2345459}/Season 07/8 Out of 10 Cats Does Countdown (2013) - S07E08 - Roisin Conaty Sarah Millican Rhod Gilbert David O'Doherty [WEBDL-1080p][AAC 2.0][x264]-NAF.mkv' -not -name '/mnt/hddcache/media/TV/8 Out of 10 Cats Does Countdown (2013) {imdb-tt2345459}/Season 07/8 Out of 10 Cats Does Countdown (2013) - S07E09 - Seann Walsh Joe Wilkinson Danny Dyer Bill Bailey [WEBDL-1080p][AAC 2.0][x264]-NAF.mkv' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|'         -v SPARSENESS='0'         -v PRIMARYSTORAGENAME='hddcache'         -v SECONDARYSTORAGENAME='user0'         -v SHARENAME='media'         -v SHAREUSECACHE='yes'         -v PRIMARYSIZETHRESH='0' ' { printf "%s|%s|%s|%s|%d|%d|%d|%d|%d|%d|%s\n", PRIMARYSTORAGENAME, SECONDARYSTORAGENAME, SHARENAME, SHAREUSECACHE, $1, PRIMARYSIZETHRESH, $2, $3, $4, $5, $6}'

If I have only the one filepath in the ignorefile, I get the same error as you
 

unexpected EOF while looking for matching `''

 

Edited February 24, 20251 yr by JayBriers

20 hours ago, slikone27 said:

Just to clarify I had it set at 0% to 0%, to try and run just on age for now.

do you using ctime or mtime?

Hi there, I'm a bit confused about the right settings. I searched the Forum and read the FAQ.

 

My scenario:

Unraid 7.0.0 with two ZFS pools, no traditional array:

1. Pioneer with one 2TB NVME
2. Vault with two mirrored 20 TB HDD and one 1TB Cache SSD

Mover has the following setting:

1. Move from primary ("Pioneer") to secondary ("Vault")

Mover tuner has the following settings:

1. Move when primary is above 75%
2. Move until primary is below 60%
3. Don't move based on age
4. Move to secondary based on size
   1. Size is set to 242GB
5. Sync primary to secondary (for "parity")

All shares are accessed via NFS, with the user share mounted by the consumers

 

The goal

1. Keep as many files on primary, only move bigger files.
2. Keep the disks spun down when, for example, Nextcloud access documents or photo sharing immich is accessing thumbnails and videos
3. Spin the disk up to access big files which have been moved to secondary

What I see:

All files, even small ones, are immediately moved to secondary

 

Are there some crucial settings I might be missing or have wrong?

On 2/16/2025 at 12:01 PM, CS01-HS said:

where age_mover uses rsync, with this call: rsync --archive --xattrs --relative --hard-links

which does not preserve atime.

 

After testing a patch to mover I can confirm adding the --atimes option does not solve the problem.

Must be some magic in Unraid's /usr/libexec/unraid/move that rsync doesn't replicate.

1 hour ago, CS01-HS said:

After testing a patch to mover I can confirm adding the --atimes option does not solve the problem.

 

I added option for atime, maybe it can help? 

Hi everyone,

I wanted to ask a few questions about "Sync primary to secondary (for "parity")" settings.

I have few folders that I access often and I would like to always be on the primary disk (the cache SSD). Since they are small folders with small files I prefer to have them on the SSD rather than having the HDDs spun up just to check a document.

 

With this settings I can do that? And if I delete a file or edit I will edit the version on the primary disk while the copy on the array won't be updated until the schedule won't start up again right?

On 2/21/2025 at 9:23 AM, Masterwishx said:

It has another if check after this line

But still checking code... 

 

Thank you so much for the help. I just enabled debugging in my instance to help get some logs. It looks like it is happening when on cache prefer.

 

Feb 24 16:12:46 falcon move: ------------------------------------------------------------ Processing appdata share -----------------------------------------------------------
Feb 24 16:12:46 falcon move: Primary storage: cache_virt - size: 466GiB - used:  14 % (61GiB)
Feb 24 16:12:46 falcon move: Secondary storage: user0
Feb 24 16:12:46 falcon move: Share Information: Name: appdata - Path: /mnt/*/appdata
Feb 24 16:12:46 falcon move: Moving threshold: 80% (373GiB) ; Freeing threshold: 40% (187GiB)
Feb 24 16:12:46 falcon move: Mover action: user0->cache_virt (cache:prefer). Pool is below priming threshold percentage:  14% < 40%.
Feb 24 16:12:46 falcon move: => Will smart move newest files from user0 to cache_virt until threshold. Older files will be moved from cache_virt to user0.
Feb 24 16:12:46 falcon move: FINDSTR: find "/mnt/*/appdata" -type f -not -path '/mnt/user*' -depth  -not -name '.placeholder' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|'         -v SPARSENESS='0'         -v PRIMARYSTORAGENAME='cache_virt'         -v SECONDARYSTORAGENAME='user0'         -v SHARENAME='appdata'         -v SHAREUSECACHE='prefer'         -v PRIMARYSIZETHRESH='200042725376' ' {
Feb 24 16:12:46 falcon move: find: '/mnt/*/appdata': No such file or directory
Feb 24 16:12:46 falcon move: Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2025-02-24T161246.list for appdata

 

I think it is because the find command is wrapping the path in quotes, so find is taking it as a literal instead of expanding the regex.

 

Output from a shell on my unraid box:
 

root@falcon:~# find "/mnt/*/appdata" -type f -not -path '/mnt/user*' -depth  -not -name '.placeholder' -printf '%T@|%s|%S|%n|%i|%p|\0'
find: ‘/mnt/*/appdata’: No such file or directory

 

If I remove the quotes, I get the full output (I cut some out since it has a lot of results)
 

root@falcon:~# find /mnt/*/appdata -type f -not -path '/mnt/user*' -depth  -not -name '.placeholder' -printf '%T@|%s|%S|%n|%i|%p|\0'
66|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/index/0/L0-00000001.tsl|1740272400.3190134880|419|9.77566|1|167072|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/index/0/MANIFEST|1740272400.3180134400|0|1|1|167067|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/index/3/L0-00000001.tsl|1740272400.3190134880|419|9.77566|1|167073|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/index/3/MANIFEST|1740272400.3180134400|0|1|1|167068|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/index/6/L0-00000001.tsl|1740272400.3190134880|419|9.77566|1|167076|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/index/6/MANIFEST|1740272400.3180134400|0|1|1|167069|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/index/1/L0-00000001.tsl|1740272400.3190134880|419|9.77566|1|167077|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/index/1/MANIFEST|1740272400.3180134400|0|1|1|167070|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/index/4/L0-00000001.tsl|1740272400.3200135370|419|9.77566|1|167078|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/index/4/MANIFEST|1740290400.3374298240|442|9.26697|1|167079|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/index/2/MANIFEST|1740290400.3234291290|0|1|1|169947|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/index/2/L0-00000002.tsl|1740290400.3304294770|623|6.57464|1|169948|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/index/2/L1-00000001.tsi|1740272400.3190134880|0|1|1|167074|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/index/7/L0-00000001.tsl|1740272400.3200135370|419|9.77566|1|167080|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/index/7/MANIFEST|1740272400.3190134880|0|1|1|167075|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/index/5/L0-00000001.tsl|1740272400.3200135370|419|9.77566|1|167081|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/index/5/MANIFEST|1740272400.3470148440|99|41.3737|1|167083|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/fields.idx|1740316201.1476871160|1676|2.44391|1|174252|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/autogen/12/000000002-000000002.tsm|1739667600.2250993810|4194304|0.000976562|1|71831|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/_series/00/0000|1739667600.2320997250|4194304|0.000976562|1|71833|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/_series/01/0000|1739667600.2381000200|4194304|0.000976562|1|71835|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/_series/02/0000|1739667600.2451003650|4194304|0.000976562|1|71837|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/_series/03/0000|1739667600.2521007090|4194304|0.000976562|1|71839|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/_series/04/0000|1739667600.2621012010|4194304|0.000976562|1|71841|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/_series/05/0000|1740272400.3360143110|4194304|0.000976562|1|71843|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/_series/06/0000|1739667600.2761018890|4194304|0.000976562|1|71845|/mnt/cache_virt/appdata/scrutiny/engine/data/a2cac7725d3d6544/_series/07/0000|1740384094.0957630970|28672|1|1|36101|/mnt/cache_virt/appdata/scrutiny/scrutiny.db|1740411730.2199417770|3932160|1|1|70904|/mnt/cache_virt/appdata/homarr/db/db.sqlite|1740443270.0799743700|22712|1.08207|1|194465|/mnt/cache_virt/appdata/homarr/redis/du

 

EDIT:
I'll open an issue on github too

Edited February 25, 20251 yr by tehg
being nice

5 hours ago, Masterwishx said:

 

I added option for atime, maybe it can help? 

 

I see an option to moves files based on atime - if that's it, my issue's different: preserving the atime of files and directories when they're moved (which standard mover does but mover-turning doesn't.)

 

Edit: AFAIK unraid doesn't update atime when files are accessed so I don't think this will work as intended.

Edited February 25, 20251 yr by CS01-HS

5 hours ago, tehg said:

. It looks like it is happening when on cache prefer.

Can you enter to this share setting change something and then save, then run mover again 

2 hours ago, CS01-HS said:

I see an option to moves files based on atime

Sorry, for mistake

3 hours ago, CS01-HS said:

(which standard mover does but mover-turning doesn't.

It's maybe issue in code somewhere Becouse rsync -t should save all timea

Need to check it, still trying to understand the code, and why something working and other not... 

3 hours ago, CS01-HS said:

unraid doesn't update atime when files are accessed so I don't think this will work as intended.

If it's useless I will remove it... 

6 hours ago, tehg said:

FINDSTR: find "/mnt/*/appdata"

This should be /mnt/cache_virt/appdata

8 hours ago, Masterwishx said:

It's maybe issue in code somewhere Becouse rsync -t should save all timea

Need to check it, still trying to understand the code, and why something working and other not... 

It looks like Limetech only solved this in 2018 by replacing rsync with a custom "move", so I don't think it's solvable with rsync.

 

 

 

I see the previous mover-tuning plugin also used unraid's "move":

https://github.com/hugenbd/ca.mover.tuning/blob/master/source/ca.mover.tuning/usr/local/emhttp/plugins/ca.mover.tuning/age_mover

 

I wonder if there are other (maybe more significant) differences between rsync and move.

 

52 minutes ago, CS01-HS said:

wonder if there are other (maybe more significant) differences between rsync and move.

Yeah I saw old version maybe by @Squid and @hugenbddwas using Unraid move (binary file), but as I said befor I only about 2 weeks ago found that we have another version of this plugin. 

 

That using find and rsync, maybe I can add option to use unraid move also but then almost all plugin futures will be disabled I think, so needs to think about it if it good idea at all...

Edited February 25, 20251 yr by Masterwishx

@Masterwishx just a BIG 👍 and thank you for taking your time and effort into this ...

 

i know its hard to reverse lookup ... and as this plugin is such a major feature meanwhile and how many coming with issues and wishes and ... so just a great THANK YOU therefore from me.

44 minutes ago, alturismo said:

i know its hard to reverse lookup

Thanks a lot, just wanted to give people fixed plugin at start but will try to make something more if I can.

I wish I have more bash experience, but my knowledge is mostly was c++, so I hope sometime someone with more bash knowledge will take to maintain this plugin 😉

12 hours ago, Masterwishx said:

This should be /mnt/cache_virt/appdata

It won't since this is in bash.

 

Bash will take the quoted text as a literal. It needs to be out of quotes to expand regex.

 

13 hours ago, Masterwishx said:

Can you enter to this share setting change something and then save, then run mover again 

 

Changing the description didn't do it. I mentioned in github, but I think it is tied to the ZFS checks that are there: https://github.com/masterwishx/ca.mover.tuning/issues/9

On 2/24/2025 at 7:22 AM, Masterwishx said:

do you using ctime or mtime?

standard mtime

7 hours ago, slikone27 said:

standard mtime

I will try to check this shold work

On 1/4/2025 at 3:17 PM, Silver226 said:

Just to add my findings as ignore list was working super inconsistently for me. Basically I would see some stuff would be ignored while others would not and there was no rhyme or reason why, as it could be any folder or file. Plus it would vary from 1 day to the next, where a file was not ignored 1 day, would be ignored the next day.

 

The TLDR is the current latest version is bugged for ignore lists if you have a lot of entries in your list. Apart from lowering the amount of entries to ignore, I found no fix bar rewriting the script.

 

The longer version, digging through each Filtered_files_....list and Mover_action_....list, I noticed double entries for ignored files and folders. Essentially 1 saying it will be skipped and another that it will be moved. Going through the age_mover script and adding debugging commands, I got the final find command, it's loooong!

 

The script will add a switch " -not -path \"${skipped_path}\"" or " -not -name \"${skipped_path}\"" for every entry on the ignore list. I have thousands of entries in my list, so no surprise the resultant find command is super chunky. My use case for having a large ignore list is, ctime is super inconsistent on my system, so I wrote a python script that will determine age to move and writes files and folders to the ignore list and then let mover tuning do it's thing.

 

Copying and pasting the full find command into my unraid console displayed the same behaviour of some results being filtered while others not. Turns out find, bash or something else, has, I guess a character limit or maybe just limits amount of switches, that it will handle. When it hits the limit, it truncates and ignores the rest of the find command, so for entries that should be ignored, fall into the truncated section, they are returned in results for what to move.

 

If I limited the number of entries on my ignore list to less than a hundred entries it seems to work fine. I didn't bother trying to work out the exact cut off point though. The code change happened in the 2024-08-15 version and seems to affect all versions after. The previous releases didn't work for me either as they seemed to ignore folders from my ignore list (plus suffered from other bugs).

 

My only fix, was to go ahead and rewrite a large part of the code to use a different method to apply filters for ignored entries. The method I found to work was running a broad find search with all other switches minus the "-not path" and "-not name" elements, write results to a tmp file, read the tmp file and remove entries that match from the ignore list and feed the results to $FILTERFILELIST. Bar some buggy reporting (size left is always wrong), mover now works properly for me and will always ignore files from my list.

 

Note I don't now if this is the best approach but every other thing I tried had some sort of issue. For instance, I only went with writing to tmp files to allow me to debug, when it was working, I tried just writing results to variables in memory, but that ended up being super buggy and inconsistent and at that point I couldn't be motivated to troubleshoot it further so left my version using tmp files.

 

This might also explain why some people have issues with smart caching and etc, as they may be truncated / malformed, if used in combination with a large ignore list. Hopefully this might help someone if a newer version is developed.

 

 

@Masterwishx do you think you can have a fix for this? Basically the plugin doesn't work when we've got a big ignore list. Thanks!