Reynald

After the log, the UI with some new options

Community Apps version is not compatible with unraid 7

This code is here since a while. It will be replaced in next version (coming really soon) Can you check if /tmp/Mover/Custom_Cache_Tuning_(date).list is populated? You can also try to replace this: ((TOTALARRAYSIZE += $CUSTOMFILESIZE)) by that: TOTALARRAYSIZE += $CUSTOMFILESIZE In this file /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 397

Teasing: next version log example: 01:23:22.137 *********************************MOVER START******************************* 01:23:22.139 Age supplied: Automatic 01:23:22.147 --------------------------------- /boot/config/shares/appdata.cfg ----------------------------------------- 01:23:22.154 Cache mode: only. Nothing will be moved. Its usage is taken into account in the calculation of the threshold. 01:23:22.155 Share Information: Name: appdata; Pool Name: cache; Path: /mnt/cache/appdata; Cache mode: only 01:23:22.174 Pool usage: 75 %, size: 723566358528 (673 GiB) 01:23:22.198 cache/appdata usage: 165967560704 bytes (154 GiB) 01:23:22.200 --------------------------------- /boot/config/shares/backup.cfg ----------------------------------------- 01:23:22.202 Cache mode: NO. Skipping. 01:23:22.203 --------------------------------- /boot/config/shares/frigate.cfg ----------------------------------------- 01:23:22.210 Cache mode: Prefer/Array->Cache. Will smart move newest files from Array to Cache until threshold. Older files will be moved from Cache to Array. 01:23:22.211 Share Information: Name: frigate; Pool Name: cache; Path: /mnt/user/frigate; Cache mode: prefer 01:23:22.228 Pool usage: 75 %, size: 723566358528 (673 GiB) 01:23:22.242 Default threshold: 90 %, frigate share threshold: NA % => Threshold selected: 90 %, size: 831559355596 bytes (774 GiB) 01:23:22.955 Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-07-22T012322.list for frigate 01:23:22.956 --------------------------------- /boot/config/shares/media.cfg ----------------------------------------- 01:23:22.963 Cache mode: Prefer/Array->Cache. Will smart move newest files from Array to Cache until threshold. Older files will be moved from Cache to Array. 01:23:22.965 Share Information: Name: media; Pool Name: cache; Path: /mnt/user/media; Cache mode: prefer 01:23:22.984 Pool usage: 75 %, size: 723566358528 (673 GiB) 01:23:22.999 Default threshold: 90 %, media share threshold: NA % => Threshold selected: 90 %, size: 831559355596 bytes (774 GiB) 01:23:58.014 Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-07-22T012322.list for media 01:23:58.015 --------------------------------- /boot/config/shares/nextcloud.borg.cfg ----------------------------------------- 01:23:58.018 Cache mode: NO. Skipping. 01:23:58.019 --------------------------------- /boot/config/shares/nextcloud.cfg ----------------------------------------- 01:23:58.026 Cache mode: only. Nothing will be moved. Its usage is taken into account in the calculation of the threshold. 01:23:58.027 Share Information: Name: nextcloud; Pool Name: cache; Path: /mnt/cache/nextcloud; Cache mode: only 01:23:58.045 Pool usage: 75 %, size: 723587424256 (673 GiB) 01:23:58.068 cache/nextcloud usage: 373098090496 bytes (347 GiB) 01:23:58.071 --------------------------------- /boot/config/shares/system.cfg ----------------------------------------- 01:23:58.077 Cache mode: only. Nothing will be moved. Its usage is taken into account in the calculation of the threshold. 01:23:58.078 Share Information: Name: system; Pool Name: cache; Path: /mnt/cache/system; Cache mode: only 01:23:58.095 Pool usage: 75 %, size: 723587424256 (673 GiB) 01:23:58.117 cache/system usage: 45286526976 bytes (42 GiB) 01:23:58.148 ------------------------------------------------------------------------------------------------------------------------------------ 01:23:58.149 Decide of the action (move/keep) for each file... 01:24:46.024 A total of 381 files representing 246558311837 bytes (229 GiB) will be moved: 01:24:46.026 - 104 older files representing 65750043216 bytes (61 GiB) will be freed from cache and moved to array 01:24:46.028 - 277 new files representing 180808268621 bytes (168 GiB) will then be moved from array to cache See for each share what "Cache mode" does. See how smart are the 3 last lines I've still some work to do on moving function (which send files to Unraid Mover) before releasing this new version

Yes, as stated above, there is a bug in autoage with the actual version of my fork. It's not emptying, but it is not respecting the threshold

Yesm the commands are in the fork: https://github.com/R3yn4ld/ca.mover.tuning/blob/794d8b0fad8ea1c2db8096dbd753ecf75c8a6e22/plugins/ca.mover.tuning.plg#L142 Can you post the output of installation script please? Yes it will show as an update as long as you installed it with plugin manager. It makes no difference if it comes from CA repository (I'm waiting for @Squid validation/integration of my repo) or by URL in plugin manager. I'm struggling with next version however (looping on the custom filelist, lines are missed in my beta), it's nearly a complete rewriting for filelist Coming soon: Automatic (by age) Array -> Cache handling: A file list is created with a little more datafield than previously and sorted by: Modification time Cache: only/prefer/yes/no Inodes (for hardlinks) Hardlink better handling Coming after that: Live monitoring of used spaced Improved process time (actually on my machine: about 5 minutes for 22.4TB in 18409 files ) Edit: I switched to GNU awk for list processing, for same size and number of files, it takes 48 seconds

Haha no problem. My script i. The thread, your in your thread, and mover tuner tend to achieve the same, so it's not really off topic here 😁

Thank you @Terebi! I'm already implementing something similar. I'm still thinking about the way to handle hardlinks (actually if hardlinks are found all the list is sent to mover), and I think you solution is good, it's similar to what I used in the "smart caching script" above. Thank you for sharing with me, that will be inspiration (more than steal haha)

That is probably the reason. Your actual settings are not using the plugin but the original unraid mover. Welcome to the unraid journey

Hi, thank you. That means that mover plugin will not be invoked when you press the move now button (3) and that it will only create files in /tmp/Mover (4) so you can check what mover does when invoked by a schedule Is it really working? It shall not execute if "Move Now button follows plug-in filters" is set to NO

Yes, I plan to implement a "failsafe" setting, using inotify to check cache usage whenever a file is open for writting or closed after writing, just as "File Integrity plugins" works.

At first, before forking this plugin, I was doing the same with a script building a custom "Skip files list". I think the code is ready for Threshold per share, but not yet the UI. Plus the UI isn't ready for overriding settings for a share which is not "cache->array". I need to finish the "cache<-array" age_mover rewritting and then can proceed with these "feature request"

So just set 45% every night ? In a future version, there will be a watchdog that trigger the mover to prevent disk full scenario

Mover tuning plugin will not start by itself, yet. I'm interested to know how you achieved this. Assuming you won't put more than 5-10% per hour on the cache, you can set a hourly mover schedule and set threshold to 90~95 in mover tuning.

1)2)3) thanks for context. 4) You are right, unless you have a very big file that represents some % of disk space. I think I have the same behaviour with too much space freed. The way I handle Automatic Age threshold is to sum the sizes up when looping customFilelist in this function: https://github.com/R3yn4ld/ca.mover.tuning/blob/master/source/ca.mover.tuning/usr/local/emhttp/plugins/ca.mover.tuning/age_mover#L353 Then if total size > threshold, I delete lines in the file list: while IFS= read -r CUSTOMLINE; do #PULL out file size ###echo "$CUSTOMLINE" CUSTOMLINESIZE=$(echo "$CUSTOMLINE" | sed -n 's/.*;\(.*\)/\1/p') CUSTOMLINEDATE=$(echo "$CUSTOMLINE" | sed -n 's/\([^;]*\).*/\1/p') CUSTOMLINETHRESH=$(echo "$CUSTOMLINE" | sed -n 's/^[^;]*;\([^;]*\);.*/\1/p') # Do not add the size of an Hardlink (hardlinks have all the same date). Bordel effect: if 2 files have exactly the same date, shall not happen, if so add previous size) if [ "$PREVIOUSLINEDATE" -ne "$CUSTOMLINEDATE" ] ; then # Append file size to total cache size ((TOTALCACHESIZE += $CUSTOMLINESIZE)) fi # Check if total cache size exceeds the threshold and autoage is selected. * # Date has also to be different from previous file to prevent breaking hardlinks if [ "$TOTALCACHESIZE" -ge "$CUSTOMLINETHRESH" ] && [ "$AGE" -eq -1 ] && [ "$PREVIOUSLINEDATE" -ne "$CUSTOMLINEDATE" ] ; then mvlogger "Threshold met, removing $CUSTOMLINE" # Delete the line from the file (in-place), escape special characters from filename sed -i "/^$(sed 's/[\/&]/\\&/g' <<< "$CUSTOMLINE")$/d" "$CUSTOM_MOVER_FILELIST" fi PREVIOUSLINEDATE=$CUSTOMLINEDATE done <$CUSTOM_MOVER_FILELIST Maybe the modification date is not a good indicator for hardlinks. In next version (I'm on it), ctime timestamp will gain in precision (digits) and the inode will be checked (all from the first 'findstr' scan). For mtime it's not as easy because I get it from 'stat -c' Next version will scan all files on "cache prefer/cache<-array" and "cache yes/cache->array" shares, sort them in reverse, and delete the most recent for which threshold is unmet

Good catch. I will insert your proposal. You may as well fork/PR as well if you like.

If you put this setting to no, then the plugin is inactive when you push "Move now" button. If it doesn't run, it's likely your schedule is not correctly set, or the shares not configured to use mover. Please see some tips there and revert back

Thank you @Freender Your understanding is correct. With these these settings it could achieve your figures, depending of file size. What the script does in your example: For each share with a pool usage > 70%, it moves oldest files until the pool usage is < 70%. So depending on how fast you fill the cache between two schedule, it can start moving anywhere from 71 to 100%. Depending of files size around 70%, it can stop lower than 70%, i.e. if you have a 20GB blue-ray at 71% of a 200GB cache, mover will stop at 61%. I have not test it that much because I don't fill the cache quickly. If you try, please report how it works

Bug spotted: new "Automatic Age" function does not take size of "Cache<-Array / Use cache Prefer" shares into account to calculate the size of files from "Cache->Array/ Use cache Yes" to move. (I personaly don't use cache prefer, yet) It will be corrected soon

I plan to add "Automatic age" to array->cache mover for shares with cache set to "Prefer" (= Array -> Cache). I also plan to add some "Smart caching" option like I did in the following (deprecated) script: https://forums.unraid.net/topic/92126-smart-caching-script/ Both of these function will require a rewrite of plugin internal filelist functions, but I think it worth it. My plan is to, under the hood: build a single list of files that were created/modified since the last plugin run (first run will be quite long) add (in a future version, tunable with option) seeding torrent, played series, and other "preventing to sleep" things sort the list by age (as actually done by "Automatic Age" settings) Then with the list containing all needed information to do smart things: emptying old files from cache to array smartly fill the cache How do you all feel about this? Any advice/feature request?

Pushed new version: https://github.com/R3yn4ld/ca.mover.tuning/raw/master/plugins/ca.mover.tuning.plg 2024-07-10 - Unraid 7.0.0 compatibility (/ 6.x bug?): original mover now works with "Move Now button follows plug-in filters" set to off. Bug discussed here:

Thank you, repo created: https://github.com/R3yn4ld/unraid-plugins and PM sent to @Squid (I lamely reuse original ca.mover.tuning XML)

Great, thank you @SimonF Do you have any guidance to the doc for the XML please?

I'll investigate this one after looking at this error: move: Usage: /usr/local/sbin/mover.old start [-e] <disk_name>

OK so there is a bug with the mover.old invocation for unraid 7.0.0+. As a workaround, you can keep your default parameters and set this to Yes: or remove this plugin as you do not use filters