3 hours ago, Masterwishx said:

also you have wrong thresholds for cache prefer, delete overrides for these if you have and try set moving threshold for some % like 10%

I'm sorry I don't understand what you mean by this. What overrides? Also is it the 'Fill up/prime up to this level of used Primary (cache) space' setting you are referring to? I wouldn't think that would matter as the share I am testing with is 'cache -> array'.

 

I tested it again changing that setting to 10% as you can see in the screenshot below, but otherwise using the same settings as above. I used 5MB files this time. I also reinstalled the plugin for good measure. The commands are included in a screenshot below, and I list the files before and after running the mover.

 

I also attached the logs from this most recent test.

Mover_action_2025-04-23T023816.list Mover_tuning_2025-04-23T023816.log Debug_Mover_tuning_2025-04-23T023816.log Filtered_files_2025-04-23T023816.list

Edited April 23, 20251 yr by fry_the_solid

7 hours ago, Masterwishx said:

Do you see some warning from rsync about them ?

Good question.....I will have to look.

It wasn't a large volume of data so I wasn't stressed about it at the time - I just noticed the pattern.

6 hours ago, fry_the_solid said:

I'm sorry I don't understand what you mean by this. What overrides

No I meaned moving thresholds to 10%

Fillup it's for array to cache it's should be by default 95%.

I added `mover reset` command (default button) for next version that will reset options. 

 

It's seems you have wrong setting for cache prefer (moving, freeing instead of free up) so check if you have override dir in config of plugin 

12 hours ago, Masterwishx said:

No I meaned moving thresholds to 10%

Fillup it's for array to cache it's should be by default 95%.

I added `mover reset` command (default button) for next version that will reset options.

I understand which threshold you mean now, though I don't understand why it needs to be 10%. I'd like it to move any files on my cache to my array, even if it only takes up 1% of the cache so that they will be protected by parity. However, I did set the moving threshold to 10% and ran it again (with the cache >10% full) with the same files as in my above post and got the same results.

 

12 hours ago, Masterwishx said:

It's seems you have wrong setting for cache prefer (moving, freeing instead of free up) so check if you have override dir in config of plugin 

I don't believe I have an override dir in my plugin config. I've attached my config file but looking at it I don't see anything like that.

 

Wouldn't the fact that all my files are moving as expected except those with special characters filtered by the 'validate input filenames' setting imply that the issue is with those special characters?

ca.mover.tuning.cfg

Edited April 24, 20251 yr by fry_the_solid

2 hours ago, fry_the_solid said:

Wouldn't the fact that all my files are moving as expected excepted those with special characters filtered by the 'validate input filenames' setting imply that the issue is with those special characters?

If you disabled files validation they should be moved, you can set both thresholds to 0%. 

 

I can see they should be moving by mover action. And by mover log they are moving!? 

 

I will try to reproduce it. 

 

9 hours ago, fry_the_solid said:

I understand which threshold you mean now, though I don't understand why it needs to be 10%. I'd like it to move any files on my cache to my array, even if it only takes up 1% of the cache so that they will be protected by parity. However, I did set the moving threshold to 10% and ran it again (with the cache >10% full) with the same files as in my above post and got the same results.

 

I don't believe I have an override dir in my plugin config. I've attached my config file but looking at it I don't see anything like that.

 

Wouldn't the fact that all my files are moving as expected except those with special characters filtered by the 'validate input filenames' setting imply that the issue is with those special characters?

ca.mover.tuning.cfg 658 B · 2 downloads

 

Sorry i cant reproduce it , all working fine if validation of files is disabled : 

 

Sorry if this has been asked before, but it seems since upgrading to 7.0 and installing the compatible plugin that the mover is leaving empty folders after the move is finished. Would enabling the setting "Clean Empty Folders" fix this new behavior? Thanks! :D

21 minutes ago, prymordial said:

fix this new behavior?

Yep if you Want to clean empty folders of moved files

Note:

In new version cli fully working now. 

You can run "mover start|stop|reset  etc.. " it will run age_mover or unraid mover depending on options enabled. 

Run `mover --help` for more options. 

 

Also `mover start -e diskX` was added. 

 

Also default button in Gui will reset options to defaults and delete overrides. 

Edited April 25, 20251 yr by Masterwishx

FYI, I believe there is an issue with the CTIME option.  It's not particularly useful and I forget why I had CTIME set to 'yes'; however, if that and 'based on age' is set to AUTO then the file list that is moved over does not seemed to be sorted correctly by CTIME.  

 

I would recommend simply using MTIME as I did not actually see any files that had different MTIMES/CTIMES.

Seems like after updating to Mover Tuning Plugin version 2025.04.24a from Mover Tuning Plugin version 2025.04.05 something broke, as files are no longer being moved

see log output from before and after update

https://pastebin.com/KRvvLtuW

already tried:

• enabling debug log
• running /usr/local/emhttp/plugins/ca.mover.tuning/age_mover reset
• uninstalling/reinstalling
•  
• running in test mode
• disabling the plugin (first option in the config list)

nothing seems to generate any additional detail in the syslog that can guide to possible reasons.

 

running unraid 6.12.10 and 6.12.15 (occuring on both systems)

 

any ideas please?

*edit*
after seeing 

Quote

/usr/local/sbin/mover: line 5: unexpected EOF while looking for matching `"'

in above logs, i did a force mover without the pluging installed, and mover seems to be running normally still.
 

Edited April 25, 20251 yr by Neo_x

8 minutes ago, Neo_x said:

Seems like after updating to Mover Tuning Plugin version 2025.04.24a from Mover Tuning Plugin version 2025.04.05 something broke, as files are no longer being moved

see log output from before and after update

https://pastebin.com/KRvvLtuW

already tried:

• enabling debug log
• running /usr/local/emhttp/plugins/ca.mover.tuning/age_mover reset
• uninstalling/reinstalling
• running in test mode
• disabling the plugin (first option in the config list)

nothing seems to generate any additional detail in the syslog that can guide to possible reasons.

 

running unraid 6.12.10 and 6.12.15 (occuring on both systems)

 

any ideas please?

 

 

 

Your moving thresholds are this:
 

Apr 25 00:00:01 Storage root: Using global (cache:yes) moving threshold: 90 %

Apr 25 00:00:01 Storage root: Using global (cache:yes) freeing threshold: 70 %

Apr 25 00:00:01 Storage root: Using global (cache:prefer) fillup threshold: 95 %

 

presumably nothing is being moved from nvme -> array, but your size calculation is this:

 

Apr 25 00:00:17 Storage root: Mover action: cache->user0 (cache:yes). Pool is below moving threshold percentage: 76% < 90%.

 

 

Nothings going to move since you are below your threshold value of 90%

7 minutes ago, ronia said:

 

Your moving thresholds are this:
 

Apr 25 00:00:01 Storage root: Using global (cache:yes) moving threshold: 90 %

Apr 25 00:00:01 Storage root: Using global (cache:yes) freeing threshold: 70 %

Apr 25 00:00:01 Storage root: Using global (cache:prefer) fillup threshold: 95 %

 

presumably nothing is being moved from nvme -> array, but your size calculation is this:

 

Apr 25 00:00:17 Storage root: Mover action: cache->user0 (cache:yes). Pool is below moving threshold percentage: 76% < 90%.

 

 

Nothings going to move since you are below your threshold value of 90%

 

yes i fully follow that (the log's of the previous version shows exactly that and follows my config). reason i know it is broken, is because my cache is currently above the configured threshold, and files should have been moved. new interations of the mover being run does not give any details on thresholds and/or decisions.

6 minutes ago, Neo_x said:

 

yes i fully follow that (the log's of the previous version shows exactly that and follows my config). reason i know it is broken, is because my cache is currently above the configured threshold, and files should have been moved. new interations of the mover being run does not give any details on thresholds and/or decisions.

 

 

So this entry isn't accurate?

 

Apr 25 00:00:17 Storage root: Primary storage: cache - size: 1.9TiB - used: 76 % (1.4TiB)

 

You don't have roughly 2TB of nvme and using 1.4TB?

8 hours ago, Neo_x said:

new interations of the mover being run does not give any details on thresholds and/or decisions.

Was no changes in last version for thouse, only calculation for cache:prefer fixed that couse to negative values

8 hours ago, Neo_x said:

Seems like after updating to Mover Tuning Plugin version 2025.04.24a from Mover Tuning Plugin version 2025.04.05 something broke, as files are no longer being moved

see log output from before and after update

As @ronia said you have below threshold of moving files. 

If you need more info check /tmp/ca.mover.tuning/* files

10 hours ago, ronia said:

would recommend simply using MTIME as I did not actually see any files that had different MTIMES/CTIMES.

Is it worked befor? As I remember I didn't changed this part only added atime. 

9 hours ago, Neo_x said:

and mover seems to be running normally still.

Unraid mover just move all files. 

Please read help description of options. 

6 hours ago, Masterwishx said:

As @ronia said you have below threshold of moving files. 

If you need more info check /tmp/ca.mover.tuning/* files

 

It really seems like it is not running.

Installed the plugin again now, and only enabled debug logging (thus threshold is 0%), and clicked on move now.

 

adding some detail

Quote

root@Storage:~# du -h -d 1 /mnt/cache/
3.1G    /mnt/cache/downloads_tmp
160G    /mnt/cache/Movies
11G     /mnt/cache/downloads
265M    /mnt/cache/Series
174G    /mnt/cache/
root@Storage:~# du -h -d 1 /mnt/nvme/
453G    /mnt/nvme/app2
35G     /mnt/nvme/docker
8.8G    /mnt/nvme/app
496G    /mnt/nvme/
root@Storage:~# cat "/boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg"
testmode="yes"
logging="yes"
movenow="yes"
version="2025.04.24a"
moverDisabled="no"
validateFilenames="yes"
debuglogging="yes"
notify="yes"
loggingFolder="/tmp"
logfilesDaysold="5"
listfilesDaysold="10"
advancedSettings="no"
parity="yes"
moverNice="0"
moverIO="-c 2 -n 0"
force="no"
beforeScript=""
afterScript=""
movingThreshold="0"
freeingThreshold=""
fillupThreshold="95"
age="no"
advancedFilters="no"
sizef="no"
sparsnessf="no"
filelistf="no"
filetypesf="no"
ignoreHidden="no"
omovercfg="no"
enableTurbo="no"
cleanFolders="no"
cleanDatasets="no"
synchronizeCache="no"
advancedOptions="no"
rebalanceShares="no"
resynchronizeCache="no"
root@Storage:~# date
Sat Apr 26 16:19:06 SAST 2025
root@Storage:~#

 

SYSLOG

 

Quote

Apr 26 16:18:46 Storage root: Starting Mover ...
Apr 26 16:18:46 Storage root: Options:
Apr 26 16:18:46 Storage root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover
Apr 26 16:18:46 Storage move: Mover Tuning Plugin version 2025.04.24a
Apr 26 16:18:46 Storage move:
Apr 26 16:18:46 Storage move: Usage: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start [-e <disk_name>]
Apr 26 16:18:46 Storage move:        /usr/local/emhttp/plugins/ca.mover.tuning/age_mover stop|softstop|status|reset
Apr 26 16:18:46 Storage move:        <disk_name> must match pattern 'disk[0-9]*'
Apr 26 16:18:46 Storage move:
Apr 26 16:18:46 Storage move: reset  - reset config file to default values and delete override config folder

 

TMP folder (Empty)

Quote

root@Storage:~# cd /tmp/ca.mover.tuning/
root@Storage:/tmp/ca.mover.tuning# ls
root@Storage:/tmp/ca.mover.tuning# ls -lisah
total 0
2959349 0 drwxr-xr-x  2 root root  40 Apr 26 01:15 ./
   2508 0 drwxrwxrwt 24 root root 720 Apr 26 16:28 ../
root@Storage:/tmp/ca.mover.tuning#

 

 

 

Edited April 26, 20251 yr by Neo_x

30 minutes ago, Neo_x said:

not running.

Please set test mode to "no" 

5 minutes ago, Masterwishx said:

Please set test mode to "no" 

adjusted, same result (usage on /mnt/cache is roughy 8 %)

 

Quote

root@Storage:/tmp/ca.mover.tuning# date
Sat Apr 26 17:09:56 SAST 2025
root@Storage:/tmp/ca.mover.tuning# cat "/boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg"
testmode="no"
logging="yes"
movenow="yes"
version="2025.04.24a"
moverDisabled="no"
validateFilenames="yes"
debuglogging="yes"
notify="yes"
loggingFolder="/tmp"
logfilesDaysold="5"
listfilesDaysold="10"
advancedSettings="no"
parity="yes"
moverNice="0"
moverIO="-c 2 -n 0"
force="no"
beforeScript=""
afterScript=""
movingThreshold="0"
freeingThreshold="0"
fillupThreshold="95"
age="no"
advancedFilters="no"
sizef="no"
sparsnessf="no"
filelistf="no"
filetypesf="no"
ignoreHidden="no"
omovercfg="no"
enableTurbo="no"
cleanFolders="no"
cleanDatasets="no"
synchronizeCache="no"
advancedOptions="no"
rebalanceShares="no"
resynchronizeCache="no"
root@Storage:/tmp/ca.mover.tuning# tail -n 50 /var/log/syslog
Apr 26 16:59:09 Storage avahi-daemon[14266]: avahi-daemon 0.8 exiting.
Apr 26 16:59:09 Storage root: Starting Avahi mDNS/DNS-SD Daemon: /usr/sbin/avahi-daemon -D
Apr 26 16:59:09 Storage avahi-daemon[19991]: Found user 'avahi' (UID 61) and group 'avahi' (GID 214).
Apr 26 16:59:09 Storage avahi-daemon[19991]: Successfully dropped root privileges.
Apr 26 16:59:09 Storage avahi-daemon[19991]: avahi-daemon 0.8 starting up.
Apr 26 16:59:09 Storage avahi-daemon[19991]: Successfully called chroot().
Apr 26 16:59:09 Storage avahi-daemon[19991]: Successfully dropped remaining capabilities.
Apr 26 16:59:09 Storage avahi-daemon[19991]: Loading service file /services/sftp-ssh.service.
Apr 26 16:59:09 Storage avahi-daemon[19991]: Loading service file /services/smb.service.
Apr 26 16:59:09 Storage avahi-daemon[19991]: Loading service file /services/ssh.service.
Apr 26 16:59:09 Storage avahi-daemon[19991]: *** WARNING: Detected another IPv4 mDNS stack running on this host. This makes mDNS unreliable and is thus not recommended. ***
Apr 26 16:59:09 Storage avahi-daemon[19991]: Joining mDNS multicast group on interface br0.IPv4 with address 192.168.0.3.
Apr 26 16:59:09 Storage avahi-daemon[19991]: New relevant interface br0.IPv4 for mDNS.
Apr 26 16:59:09 Storage avahi-daemon[19991]: Network interface enumeration completed.
Apr 26 16:59:09 Storage avahi-daemon[19991]: Registering new address record for 192.168.0.3 on br0.IPv4.
Apr 26 16:59:09 Storage emhttpd: shcmd (9394886): /etc/rc.d/rc.avahidnsconfd restart
Apr 26 16:59:09 Storage root: Stopping Avahi mDNS/DNS-SD DNS Server Configuration Daemon: stopped
Apr 26 16:59:09 Storage root: Starting Avahi mDNS/DNS-SD DNS Server Configuration Daemon:  /usr/sbin/avahi-dnsconfd -D
Apr 26 16:59:09 Storage avahi-dnsconfd[20002]: Successfully connected to Avahi daemon.
Apr 26 16:59:09 Storage unassigned.devices: Updating share settings...
Apr 26 16:59:09 Storage unassigned.devices: Share settings updated.
Apr 26 16:59:10 Storage avahi-daemon[19991]: Server startup complete. Host name is Storage.local. Local service cookie is 4228167732.
Apr 26 16:59:11 Storage avahi-daemon[19991]: Service "Storage" (/services/ssh.service) successfully established.
Apr 26 16:59:11 Storage avahi-daemon[19991]: Service "Storage" (/services/smb.service) successfully established.
Apr 26 16:59:11 Storage avahi-daemon[19991]: Service "Storage" (/services/sftp-ssh.service) successfully established.
Apr 26 17:00:01 Storage root: Starting Mover ...
Apr 26 17:00:01 Storage root: Options:
Apr 26 17:00:01 Storage root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover
Apr 26 17:00:01 Storage root: Mover Tuning Plugin version 2025.04.24a
Apr 26 17:00:01 Storage root:
Apr 26 17:00:01 Storage root: Usage: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start [-e <disk_name>]
Apr 26 17:00:01 Storage root:        /usr/local/emhttp/plugins/ca.mover.tuning/age_mover stop|softstop|status|reset
Apr 26 17:00:01 Storage root:        <disk_name> must match pattern 'disk[0-9]*'
Apr 26 17:00:01 Storage root:
Apr 26 17:00:01 Storage root: reset  - reset config file to default values and delete override config folder
Apr 26 17:00:02 Storage mergerfs[32302]: running basic garbage collection
Apr 26 17:06:33 Storage emhttpd: spinning down /dev/sdf
Apr 26 17:06:54 Storage emhttpd: read SMART /dev/sdk
Apr 26 17:07:42 Storage emhttpd: read SMART /dev/sdf
Apr 26 17:10:21 Storage emhttpd: shcmd (9400543): /usr/local/sbin/mover |& logger -t move &
Apr 26 17:10:21 Storage root: Starting Mover ...
Apr 26 17:10:21 Storage root: Options:
Apr 26 17:10:21 Storage root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover
Apr 26 17:10:21 Storage move: Mover Tuning Plugin version 2025.04.24a
Apr 26 17:10:21 Storage move:
Apr 26 17:10:21 Storage move: Usage: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start [-e <disk_name>]
Apr 26 17:10:21 Storage move:        /usr/local/emhttp/plugins/ca.mover.tuning/age_mover stop|softstop|status|reset
Apr 26 17:10:21 Storage move:        <disk_name> must match pattern 'disk[0-9]*'
Apr 26 17:10:21 Storage move:
Apr 26 17:10:21 Storage move: reset  - reset config file to default values and delete override config folder
root@Storage:/tmp/ca.mover.tuning# uame -r
-bash: uame: command not found
root@Storage:/tmp/ca.mover.tuning# uname -r
6.1.79-Unraid
root@Storage:/tmp/ca.mover.tuning#

 

I will add in next version : warning about test mode.

In global setting section after setting or better to add at the end where analyzing moving data? 

52 minutes ago, Neo_x said:

adjusted, same result (usage on /mnt/cache is roughy 8 %)

No need to post syslog. 

You need to post files from /tmp/ca.mover.tuning. + config will try to add debug zip for diagnostic in next version 

8 hours ago, Masterwishx said:

Is it worked befor? As I remember I didn't changed this part only added atime. 

It's very possible that it never worked and I just never paid attention to it.  Not a groundbreaking bug, but maybe something to look into if you have free time.  

Easiest workaround is just to simply use MTIME.

Hello,

i have an issue with CA Mover Tuning :  data are not moved from cache to array :
 

Unraid Version: 7.0.1

Mover Tuning Version: 2025.04.24a
Issue: eg. data are not moved from cache to array :

 

Here my cache :


 

CA mover Tuning settings : 

 

 

Logs :

 

Apr 26 19:00:04 Saroumane move: ------------------------------------------------------------ Processing [media] share -----------------------------------------------------------
Apr 26 19:00:05 Saroumane move: Primary storage: cache - size: 457GiB - used:  83 % (367GiB)
Apr 26 19:00:05 Saroumane move: Secondary storage: user0
Apr 26 19:00:05 Saroumane move: Share Information: Name: media - Path: /mnt/cache/media
Apr 26 19:00:05 Saroumane move: Moving threshold: 70% (320GiB) ; Freeing threshold: 55% (252GiB)
Apr 26 19:00:05 Saroumane move: Mover action: cache->user0 (cache:yes). Pool is above moving threshold percentage:   83% >= 70%.
Apr 26 19:00:05 Saroumane move: => Will smart move old files from cache to user0. Nothing will be moved from user0 to cache
Apr 26 19:00:05 Saroumane move: ------------------------------------------------------ Debugging [media] share Find Function ----------------------------------------------------
Apr 26 19:00:05 Saroumane move: FINDSTR: find "/mnt/cache/media" -type f -depth -not -name '.placeholder' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|'         -v SPARSENESS='0'         -v PRIMARYSTORAGENAME='cache'         -v SECONDARYSTORAGENAME='user0'         -v SHARENAME='media'         -v SHAREUSECACHE='yes'         -v PRIMARYSIZETHRESH='269557680128' ' { printf "%s|%s|%s|%s|%d|%d|%d|%d|%d|%d|%s\n", PRIMARYSTORAGENAME, SECONDARYSTORAGENAME, SHARENAME, SHAREUSECACHE, $1, PRIMARYSIZETHRESH, $2, $3, $4, $5, $6}'
Apr 26 19:00:05 Saroumane move:                                                                                                                                                  
Apr 26 19:00:05 Saroumane move: Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2025-04-26T190002.list for media
Apr 26 19:00:05 Saroumane move: ------------------------------------------------------------ Processing [system] share ----------------------------------------------------------
Apr 26 19:00:05 Saroumane move: Primary storage: cache - size: 457GiB - used:  83 % (367GiB)
Apr 26 19:00:05 Saroumane move: Secondary storage: user0
Apr 26 19:00:05 Saroumane move: Share Information: Name: system - Path: /mnt/*/system
Apr 26 19:00:05 Saroumane move: Moving threshold: % (0B) ; Filling up threshold: 60% (55GiB)
Apr 26 19:00:05 Saroumane move: ----------------------------------------------------- Debugging [system] share Find Function ----------------------------------------------------
Apr 26 19:00:05 Saroumane move: FINDSTR: find "/mnt/disk2/system" -type f -depth -not -name '.placeholder' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|'         -v SPARSENESS='0'         -v PRIMARYSTORAGENAME='cache'         -v SECONDARYSTORAGENAME='user0'         -v SHARENAME='system'         -v SHAREUSECACHE='prefer'         -v PRIMARYSIZETHRESH='58052586700' ' { printf "%s|%s|%s|%s|%d|%d|%d|%d|%d|%d|%s\n", PRIMARYSTORAGENAME, SECONDARYSTORAGENAME, SHARENAME, SHAREUSECACHE, $1, PRIMARYSIZETHRESH, $2, $3, $4, $5, $6}'
Apr 26 19:00:05 Saroumane move:                                                                                                                                                  
Apr 26 19:00:05 Saroumane move: --------------------------------------------------------- Debugging [system] count files --------------------------------------------------------
Apr 26 19:00:05 Saroumane move: Amount of files on secondary storage: 0
Apr 26 19:00:05 Saroumane move:                                                                                                                                                  
Apr 26 19:00:05 Saroumane move: ************************************************************ ANALYSING MOVING ACTIONS ***********************************************************
Apr 26 19:00:05 Saroumane move: Deciding the action (move/sync/keep) for each file. There are 131 files, it can take a while...
Apr 26 19:00:06 Saroumane move: No new files will be moved/synced from primary to secondary
Apr 26 19:00:06 Saroumane move: No new files will be moved/synced from secondary to primary
Apr 26 19:00:06 Saroumane move: Cleaning lock and stop files
Apr 26 19:00:06 Saroumane move: ****************************************************************** WE ARE DONE ! ****************************************************************

 

 

I know that in media i have 135G all data than can be fluh to array. But nothing happen.

I don't see any errors.

 

maybe i have misunderstood something in the settings ?!

 

regards