[Plugin] Mover Tuning

Reynald · July 30

21 hours ago, Terebi said:

So I set my "media" share to array->cache, but the tuner will actually empty it from cache->array to keep it below 80%?

Yes it is:

The tuner list all files from cache:prefer and cache:yes from the more recent to the oldest, then it compare the cumulative size to the threshold, for each sharepool.

If you want to keep below 80% and that while running the scheduled mover you are at 92%, then the oldest files from cache:prefer and cache:yes representing 12% of space will be moved to array.
If you want to keep below 80% and that while running the scheduled mover you are at 67%, then the most recent files from cache:prefer shares on array, representing 13% of cache size, will be moved to cache, not those from cache:yes because they are meant to be moved from cache to array and that is..

Edited July 31 by Reynald
(grammar)

Reynald · August 1

On 7/30/2024 at 7:52 AM, Reynald said:

Your error is because you delete the share content but not the share via the UnRaid GUI. I'm adding error control in the next version

Update from last night:

###2024-08-01

- Deleted share error control

- SoftStop improvement

How softstop works:

It will cleanly exit loops (Filtering, Deciding, Moving). While moving/syncing, the file transfer is not interrupted. Can be launched by running this command from terminal or from a script (it was already existing previously):

/usr/local/emhttp/plugins/ca.mover.tuning/age_mover softstop

there are also, to start age_mover (the engine of Mover Tuning Plugin)

/usr/local/emhttp/plugins/ca.mover.tuning/age_mover start

and to kill all the process (can lead to unfinished or corrupted file transferts while moving):

/usr/local/emhttp/plugins/ca.mover.tuning/age_mover stop

nmkaufman · August 1

Are there plans to create an option to disable all the new cache juggling? Or should I just keep using the old plugin?

I'd made the assumption it would be deprecated at some point, since it hasn't been updated for 7.0.0.

I run my mover every night after a backup script, and I really don't expect/want it to do anything until my cache is 80% full.

It seems that having overrides set on any of my shares causes the contents of those shares to be scanned, each night, even if my cache filled % is below the threshold.

I respect all the hard work, it's just not for me is all. I'd prefer my disks just keep sleeping unless the cache is full.

Edited August 1 by nmkaufman

alturismo · August 1

19 minutes ago, nmkaufman said:

I respect all the hard work, it's just not for me is all. I'd prefer my disks just keep sleeping unless the cache is full.

my disks wont spinup during the mover lookup action ... just as note

nmkaufman · August 1

The lookup may not be waking mine, either, but it's waking disks because it keeps moving stuff that doesn't need moved.

Mover tuner is configured to move >30 day files, with a threshold of 80%.

I have multiple share overrides configured to move >0 days (but also with a threshold of 80%)

I don't want anything to happen unless my cache is 80% full.

At that point some shares are completely emptied, and others retain any files newer than 30 days.

The new plugin keeps emptying these 0-day shares, every night, which I do not want.

Edited August 1 by nmkaufman

alturismo · August 1

15 minutes ago, nmkaufman said:

(but also with a threshold of 80%)

ok, then im out ... i have mine currently set to 90 % treehold, and it shouldnt matter which Shares behind are .. as its cache drive based afaik ... so either my cache drive is 90 % filled or not, currently its not and so nothing happening.

i guess im missunderstanding your usecase, so im rather out and lets see who can say something about it

nmkaufman · August 1

You're not misunderstanding; That's exactly how mine used to work until a few days ago.

There's now a per-share threshold %, and the behavior seems to have completely changed.

It may just be a bug that the per-share threshold % is not being respected.

alturismo · August 1

1 hour ago, nmkaufman said:

and the behavior seems to have completely changed.

prolly true, lets see ... i wait a little and 1st have to look why its still failing here so much, i assume my filter list and "old" share configs are bad ... need some cleanup todo here and then i can report back here

alturismo · August 1

may as question, when i check now, i always see something like

... [: =: unary operator expected

Aug  1 17:49:13 AlsServerII move: ---------------------------------------------------------- Settings for Dokumente share ---------------------------------------------------------
Aug  1 17:49:13 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 362: [: =: unary operator expected
Aug  1 17:49:13 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 364: [: =: unary operator expected
Aug  1 17:49:13 AlsServerII move: Share Information: Name: Dokumente - Path: /mnt/cache/Dokumente - Cache mode: only
Aug  1 17:49:13 AlsServerII move: Pool Information: Name: cache - pool used:  52 % - size: 1035657900032 bytes (964 GiB)
Aug  1 17:49:13 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 416: [: =: unary operator expected
Aug  1 17:49:13 AlsServerII move: Adding Ignore Hidden Files: yes
Aug  1 17:49:13 AlsServerII move: Adding Skip Folder List. List Path: /mnt/user/system/mover/mover_ignore.txt
Aug  1 17:49:13 AlsServerII move: Adding Skip File List. List Path: /mnt/user/system/mover/mover_ignore.txt
Aug  1 17:49:13 AlsServerII move: Skipfilelist string: find "/mnt/cache/Dokumente" -type f -depth -not -path '*/\.*' -not -path '/mnt/cache/Media/EBOOKS/*' -not -path '/mnt/cache/Media/RECORD/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/system/mover/mover_ignore.txt')
Aug  1 17:49:13 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 514: [: =: unary operator expected
Aug  1 17:49:13 AlsServerII move: ------------------------------------------------------------ Settings for Media share -----------------------------------------------------------
Aug  1 17:49:13 AlsServerII move: Cache mode: Yes/Cache->Array. Will smart move old files from Cache to Array. Nothing will be moved from Array to Cache
Aug  1 17:49:13 AlsServerII move: Share Information: Name: Media - Path: /mnt/cache/Media - Cache mode: yes
Aug  1 17:49:13 AlsServerII move: Pool Information: Name: cache - pool used:  52 % - size: 1035657900032 bytes (964 GiB)
Aug  1 17:49:13 AlsServerII move: Default threshold: 90 % - Media share threshold: NA % => Threshold selected: 90 %,s size: 1799479015833 bytes (1 TiB)
Aug  1 17:49:13 AlsServerII move: Adding Ignore Hidden Files: yes
Aug  1 17:49:13 AlsServerII move: Adding Skip Folder List. List Path: /mnt/user/system/mover/mover_ignore.txt
Aug  1 17:49:13 AlsServerII move: Adding Skip File List. List Path: /mnt/user/system/mover/mover_ignore.txt
Aug  1 17:49:13 AlsServerII move: Skipfilelist string: find "/mnt/cache/Media" -type f -depth -not -path '*/\.*' -not -path '/mnt/cache/Media/EBOOKS/*' -not -path '/mnt/cache/Media/RECORD/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/system/mover/mover_ignore.txt')
Aug  1 17:49:13 AlsServerII move: Filtering Media files... (can take a moment)
Aug  1 17:49:15 AlsServerII move: Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-08-01T174913.list for Media
Aug  1 17:49:15 AlsServerII move: ---------------------------------------------------------- Settings for Nextcloud share ---------------------------------------------------------
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 362: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 364: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: Share Information: Name: Nextcloud - Path: /mnt/cache/Nextcloud - Cache mode: only
Aug  1 17:49:15 AlsServerII move: Pool Information: Name: cache - pool used:  52 % - size: 1035657900032 bytes (964 GiB)
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 416: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: Adding Ignore Hidden Files: yes
Aug  1 17:49:15 AlsServerII move: Adding Skip Folder List. List Path: /mnt/user/system/mover/mover_ignore.txt
Aug  1 17:49:15 AlsServerII move: Adding Skip File List. List Path: /mnt/user/system/mover/mover_ignore.txt
Aug  1 17:49:15 AlsServerII move: Skipfilelist string: find "/mnt/cache/Nextcloud" -type f -depth -not -path '*/\.*' -not -path '/mnt/cache/Media/EBOOKS/*' -not -path '/mnt/cache/Media/RECORD/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/system/mover/mover_ignore.txt')
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 514: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: ----------------------------------------------------------- Settings for appdata share ----------------------------------------------------------
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 362: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 364: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: Share Information: Name: appdata - Path: /mnt/cache/appdata - Cache mode: only
Aug  1 17:49:15 AlsServerII move: Pool Information: Name: cache - pool used:  52 % - size: 1035657900032 bytes (964 GiB)
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 416: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: Adding Ignore Hidden Files: yes
Aug  1 17:49:15 AlsServerII move: Adding Skip Folder List. List Path: /mnt/user/system/mover/mover_ignore.txt
Aug  1 17:49:15 AlsServerII move: Adding Skip File List. List Path: /mnt/user/system/mover/mover_ignore.txt
Aug  1 17:49:15 AlsServerII move: Skipfilelist string: find "/mnt/cache/appdata" -type f -depth -not -path '*/\.*' -not -path '/mnt/cache/Media/EBOOKS/*' -not -path '/mnt/cache/Media/RECORD/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/system/mover/mover_ignore.txt')
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 514: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: ----------------------------------------------------------- Settings for domains share ----------------------------------------------------------
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 362: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 364: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: Share Information: Name: domains - Path: /mnt/cache/domains - Cache mode: only
Aug  1 17:49:15 AlsServerII move: Pool Information: Name: cache - pool used:  52 % - size: 1035657900032 bytes (964 GiB)
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 416: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: Adding Ignore Hidden Files: yes
Aug  1 17:49:15 AlsServerII move: Adding Skip Folder List. List Path: /mnt/user/system/mover/mover_ignore.txt
Aug  1 17:49:15 AlsServerII move: Adding Skip File List. List Path: /mnt/user/system/mover/mover_ignore.txt
Aug  1 17:49:15 AlsServerII move: Skipfilelist string: find "/mnt/cache/domains" -type f -depth -not -path '*/\.*' -not -path '/mnt/cache/Media/EBOOKS/*' -not -path '/mnt/cache/Media/RECORD/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/system/mover/mover_ignore.txt')
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 514: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: ------------------------------------------------------------- Settings for isos share -----------------------------------------------------------
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 362: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 364: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: Share Information: Name: isos - Path: /mnt/cache/isos - Cache mode: only
Aug  1 17:49:15 AlsServerII move: Pool Information: Name: cache - pool used:  52 % - size: 1035657900032 bytes (964 GiB)
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 416: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: Adding Ignore Hidden Files: yes
Aug  1 17:49:15 AlsServerII move: Adding Skip Folder List. List Path: /mnt/user/system/mover/mover_ignore.txt
Aug  1 17:49:15 AlsServerII move: Adding Skip File List. List Path: /mnt/user/system/mover/mover_ignore.txt
Aug  1 17:49:15 AlsServerII move: Skipfilelist string: find "/mnt/cache/isos" -type f -depth -not -path '*/\.*' -not -path '/mnt/cache/Media/EBOOKS/*' -not -path '/mnt/cache/Media/RECORD/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/system/mover/mover_ignore.txt')
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 514: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: ------------------------------------------------------------- Settings for lxc share ------------------------------------------------------------
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 362: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 364: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: Share Information: Name: lxc - Path: /mnt/cache/lxc - Cache mode: only
Aug  1 17:49:15 AlsServerII move: Pool Information: Name: cache - pool used:  52 % - size: 1035657900032 bytes (964 GiB)
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 416: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: Adding Ignore Hidden Files: yes
Aug  1 17:49:15 AlsServerII move: Adding Skip Folder List. List Path: /mnt/user/system/mover/mover_ignore.txt
Aug  1 17:49:15 AlsServerII move: Adding Skip File List. List Path: /mnt/user/system/mover/mover_ignore.txt
Aug  1 17:49:15 AlsServerII move: Skipfilelist string: find "/mnt/cache/lxc" -type f -depth -not -path '*/\.*' -not -path '/mnt/cache/Media/EBOOKS/*' -not -path '/mnt/cache/Media/RECORD/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/system/mover/mover_ignore.txt')
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 514: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: ------------------------------------------------------------ Settings for system share ----------------------------------------------------------
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 362: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 364: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: Share Information: Name: system - Path: /mnt/cache/system - Cache mode: only
Aug  1 17:49:15 AlsServerII move: Pool Information: Name: cache - pool used:  52 % - size: 1035657900032 bytes (964 GiB)
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 416: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: Adding Ignore Hidden Files: yes
Aug  1 17:49:15 AlsServerII move: Adding Skip Folder List. List Path: /mnt/user/system/mover/mover_ignore.txt
Aug  1 17:49:15 AlsServerII move: Adding Skip File List. List Path: /mnt/user/system/mover/mover_ignore.txt
Aug  1 17:49:15 AlsServerII move: Skipfilelist string: find "/mnt/cache/system" -type f -depth -not -path '*/\.*' -not -path '/mnt/cache/Media/EBOOKS/*' -not -path '/mnt/cache/Media/RECORD/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/system/mover/mover_ignore.txt')
Aug  1 17:49:15 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 514: [: =: unary operator expected
Aug  1 17:49:15 AlsServerII move: ************************************************************ ANALYSING MOVING ACTIONS ***********************************************************
Aug  1 17:49:15 AlsServerII move: Deciding the action (move/sync/keep) for each file. There are 720 files, it can take a while...
Aug  1 17:49:18 AlsServerII move: No new files will be moved/synced from cache to array
Aug  1 17:49:18 AlsServerII move: No new files will be moved/synced from array to cache
Aug  1 17:49:18 AlsServerII move: ****************************************************************** WE ARE DONE ! ****************************************************************

and in the current state i have no idea what is causing it

here the ignore ...

while there is only 1 Share involved in a mover action anyhow ...

--------------------------------

and to clarify

auto age ... will only move short under treehold ... so once you reach it you would always trigger it with a extra file/s again ... more like a "ping pong" once you reached treehold ...

while this setting would move all files off to array > 180 days age ... nevermind the treehold

image.png.5f3451ce4fd9ccfb7d26c3d04005f2df.png

correct ?

Reynald · August 1

7 hours ago, nmkaufman said:

Are there plans to create an option to disable all the new cache juggling? Or should I just keep using the old plugin?

I'd made the assumption it would be deprecated at some point, since it hasn't been updated for 7.0.0.

I run my mover every night after a backup script, and I really don't expect/want it to do anything until my cache is 80% full.

It seems that having overrides set on any of my shares causes the contents of those shares to be scanned, each night, even if my cache filled % is below the threshold.

I respect all the hard work, it's just not for me is all. I'd prefer my disks just keep sleeping unless the cache is full.

No plan for option about the "cache juggling" yet, as if you set this to anything else than Auto and Synchronise cache files to array, the behavior is the same as before:

This will wake the array to synchronise whats on cache to it if files are newer than the last scan/sync.

All share content (on cache) will be scanned at each run.
Content of Use Cache:Prefer (Array->Cache) will be scan and may or may not (if Folder Caching plugin installed) wake the array up.
Set Use cache:Yes in the share settings to not wake the spinners. Cache:Prefer scanning is a new feature, as before Mover Tuning only focussed on Cache:Yes shares. Feature for smart caching was requested for a while, several script attempt (creating dynamic ignore list) have been made, now it's live.

Per share threshold was implemented before I "arrive" in this project, but there were no gui option for it.

And finally, no, hugenbdd won't update his version, we've hand over, and I have no plan to deprecate it at the moment.

Hope this help, do not hesitate if you need some more support :)

Reynald · August 1

6 hours ago, nmkaufman said:

The lookup may not be waking mine, either, but it's waking disks because it keeps moving stuff that doesn't need moved.

Does it move or does it sync? Please check the log, the mover action list (in /tmp/ca.mover.tuning), and sync option, I'd be happy to correct a bug or bad behavior, testing on my own may not be enough

Reynald · August 1

5 hours ago, nmkaufman said:

There's now a per-share threshold %, and the behavior seems to have completely changed.

The log will indicate the Default threshold, the share threshold, and the choosen one (if share threshold exists, it choose it, else it choose the default one)

Reynald · August 1

4 hours ago, alturismo said:

while this setting would move all files off to array > 180 days age ... nevermind the treehold

correct ?

No, the threshold, as before, is a condition to start or not the age_mover

image.png.662ec5f2f6264d502768f1c7e51ac53d.png

Reynald · August 1

4 hours ago, alturismo said:

may as question, when i check now, i always see something like

... [: =: unary operator expected

Aug  1 17:49:13 AlsServerII move: ---------------------------------------------------------- Settings for Dokumente share ---------------------------------------------------------
Aug  1 17:49:13 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 362: [: =: unary operator expected
Aug  1 17:49:13 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 364: [: =: unary operator expected
Aug  1 17:49:13 AlsServerII move: Share Information: Name: Dokumente - Path: /mnt/cache/Dokumente - Cache mode: only
Aug  1 17:49:13 AlsServerII move: Pool Information: Name: cache - pool used:  52 % - size: 1035657900032 bytes (964 GiB)
Aug  1 17:49:13 AlsServerII move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 416: [: =: unary operator expected

and in the current state i have no idea what is causing it

Something with this option probably:

I guess you updated without uninstalling previous version (that's completly supported, normally...), so that during update the plugin install keep the previous config right?

If so that's a bug, I see what I missed (default values when config not set is empty instead of "no").
Is this section in your log says:

----------------------------------------------------------------- Global settings --------------------------------------------------------------
[...]
No Repair argument provided
[...]

In any case can you please:

Set Repair Cache Only/No share option to yes, apply, to no, apply? just to confirm my guess? Normally unless I made a typo in variable names in the code, the "Unrary" error should disappear.

Edited August 1 by Reynald

alturismo · August 2

7 hours ago, Reynald said:

Something with this option probably:

you where right

7 hours ago, Reynald said:

I guess you updated without uninstalling previous version (that's completly supported, normally...), so that during update the plugin install keep the previous config right?

yes

7 hours ago, Reynald said:

Set Repair Cache Only/No share option to yes, apply, to no, apply? just to confirm my guess? Normally unless I made a typo in variable names in the code, the "Unrary" error should disappear.

worked out, thanks, error is gone, sample now

Aug  2 06:14:43 AlsServerII move: ------------------------------------------------------------ Settings for Media share -----------------------------------------------------------
Aug  2 06:14:43 AlsServerII move: Cache mode: Yes/Cache->Array. Will smart move old files from Cache to Array. Nothing will be moved from Array to Cache
Aug  2 06:14:43 AlsServerII move: Share Information: Name: Media - Path: /mnt/cache/Media - Cache mode: yes
Aug  2 06:14:43 AlsServerII move: Pool Information: Name: cache - pool used:  52 % - size: 1035156312064 bytes (964 GiB)
Aug  2 06:14:43 AlsServerII move: Default threshold: 90 % - Media share threshold: NA % => Threshold selected: 90 %,s size: 1799479015833 bytes (1 TiB)
Aug  2 06:14:43 AlsServerII move: Adding Ignore Hidden Files: yes
Aug  2 06:14:43 AlsServerII move: Adding Skip Folder List. List Path: /mnt/user/system/mover/mover_ignore.txt
Aug  2 06:14:43 AlsServerII move: Adding Skip File List. List Path: /mnt/user/system/mover/mover_ignore.txt
Aug  2 06:14:43 AlsServerII move: Skipfilelist string: find "/mnt/cache/Media" -type f -depth -not -path '*/\.*' -not -path '/mnt/cache/Media/EBOOKS/*' -not -path '/mnt/cache/Media/RECORD/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/system/mover/mover_ignore.txt')
Aug  2 06:14:43 AlsServerII move: Filtering Media files... (can take a moment)
Aug  2 06:14:44 AlsServerII move: Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-08-02T061442.list for Media
Aug  2 06:14:44 AlsServerII move: ---------------

alturismo · August 2

8 hours ago, Reynald said:

No, the threshold, as before, is a condition to start or not the age_mover

that is clear, just as sample's

global

mover schedule >> daily once

treehold >> 90 %

>> mover runs once a day, check treehold, if > 90 % it ll start moving by "rules"

>> while im on 91 % as sample and fill daily 2 %

rules sample

image.png.fa8c41648676bba0d99cd4fca20d27b9.png

this will now daily move old files until im at 89 %, next day (again 91 %) it ll move again, ...

thats my daily "ping pong" sample, correct ?

image.png.b5156b507720e208f8238ce97fc6360d.png

this will now move all files > 180 days old once i hit the 91 %

which can be anything from 0 to 100 % (depending how many files are 180+ days old), correct ?

so here the result is "variable", after a scheduled move it still can be 91 % or 80 % or 50 % or 0 % treehold.

thats my interpretation from this setting, so in my usecase (i dont want ping pong) i like to move ~ 20 - 30 % off the cache depending on age once i hit the 91 % as there is no target fillrate in the plugin.

i have to look what my fillrate is in average and find something to come around 60 - 70 % after mover schedule triggered ...

if my interpretation is correct

Notwist · August 2

I cannot figure out why mover tuning plugin is moving my hidden folders despite my specifically enabling "Ignore hidden files and folders" for the specific share.

Reynald · August 2

2 hours ago, alturismo said:

rules sample

this will now daily move old files until im at 89 %, next day (again 91 %) it ll move again, ...

thats my daily "ping pong" sample, correct ?

Correct, and in my opinion, this is an efficient way to use cache: set the threshold at 100% minus your period (daily, hourly) usage minus some margin

Quote

this will now move all files > 180 days old once i hit the 91 %

which can be anything from 0 to 100 % (depending how many files are 180+ days old), correct ?

so here the result is "variable", after a scheduled move it still can be 91 % or 80 % or 50 % or 0 % treehold.

Correct, and in my opinion, this is a waste of cache free space until you fill it up again

Quote

thats my interpretation from this setting, so in my usecase (i dont want ping pong) i like to move ~ 20 - 30 % off the cache depending on age once i hit the 91 % as there is no target fillrate in the plugin.

i have to look what my fillrate is in average and find something to come around 60 - 70 % after mover schedule triggered ...

if my interpretation is correct

I do prefer ping pong because it takes advantage of cache space. You can set threshold to 65% and set your schedule so that you avoid out of disk space situation.

You can think about two scenarios:
- 90~95% hourly
- 60~70 daily
I think the result (depending of your cache filling speed) is quite similar but the first one take better advantage of the cache.

Anyway, I understood your wish of playing tennis (60~93% daily) instead of ping pong (89~92% hourly) and will think about bringing a low threshold.
That could be:
- schedule + high threshold trigger the mover

- cache is emptied (cache:prefer, cache:yes) or filled (cache:prefer) down/up to low threshold

Any thought anyone?

Reynald · August 2

3 hours ago, helpful-tune3401 said:

I cannot figure out why mover tuning plugin is moving my hidden folders despite my specifically enabling "Ignore hidden files and folders" for the specific share.

OK I will investigate.

Was this the case with older versions (prior to 2024.06) ?

alturismo · August 2

14 minutes ago, Reynald said:

You can think about two scenarios:
- 90~95% hourly
- 60~70 daily
I think the result (depending of your cache filling speed) is quite similar but the first one take better advantage of the cache.

i understand your case, my usecase is more

fillrate i would guess now, may 10 % / month, even less ...

and my array is more or less really just a collection keeper, so will be used around 1x month for a movie or so, if i want to rewatch a series again, i usually move them manually backwards to the cache ... so normal usecase here is a wakeup/spinup from my array in a 1x / month period ... not daily or weekly

also a reason why i liked the idea of "TV Show" mover array > cache when started ... and already resides on the array

thanks for looking into it

Reynald · August 2

OK I think I get your point.

TV Shows and torrent caching will come later, for sure.

Can you please explain how would you set Mover Tuning for your use case if there where two thresholds?

alturismo · August 2

Just now, Reynald said:

Can you please explain how would you set Mover Tuning for your use case if there where two thresholds?

personally

mover run hourly

treehold 90 % (keep cache until ... if 90+ start moving until)

treehold 70 % (stop moving as soon 69 is reached)

sorted by age

excluding the filtered list ...

simple as this

---

while im watching a "old" tvshow which i moved manually to the cache, the tvshow is inserted in the exluded list until final episode is reached then i remove it and it ll be moved away again as its "old" anyhow.

that would be my current usecase if there would be a "target" treehold ...

like this, my cache will always be ~ 70 - 90 % fillrated and real moving actions will only be triggered like every 2 - 3 month here ...

nmkaufman · August 2

19 hours ago, Reynald said:

Does it move or does it sync? Please check the log, the mover action list (in /tmp/ca.mover.tuning), and sync option, I'd be happy to correct a bug or bad behavior, testing on my own may not be enough

It is moving. It does not seem to be respecting the threshold %.

I removed the plugin, and set back up from scratch, same behavior.

Quote

------------------------------------------------------------ Settings for Games share -----------------------------------------------------------
Settings override:
------------------
Share threshold: 90 %
Age: no = 0 ; daysold: 30
------------------------------------------------------------------------ -----------------------------------------------------------------------
Cache mode: Yes/Cache->Array. Will smart move old files from Cache to Array. Nothing will be moved from Array to Cache
Share Information: Name: Games - Path: /mnt/cache/Games - Cache mode: yes
Pool Information: Name: cache - pool used: 74 % - size: 732397322240 bytes (682 GiB)
Default threshold: 90 % - Games share threshold: 90 % => Threshold selected: 90 %,s size: 910985549414 bytes (848 GiB)
Global skipfiletypes string: find "/mnt/cache/Games" -type f -depth -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -iv -e '\.ass' -e '\.ASS' -e '\.gif' -e '\.GIF' -e '\.idx' -e '\.IDX' -e '\.jfif' -e '\.JFIF' -e '\.jpeg' -e '\.JPEG' -e '\.jpg' -e '\.JPG' -e '\.nfo' -e '\.NFO' -e '\.png' -e '\.PNG' -e '\.srt' -e '\.SRT' -e '\.ssa' -e '\.SSA' -e '\.sub' -e '\.SUB' -e '\.svg' -e '\.SVG' -e '\.txt' -e '\.TXT' -e '\.vtt' -e '\.VTT' -e '\.ffs_db' -e '\.ffs_lock'
Filtering Games files... (can take a moment)
Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-08-02T100913.list for Games
------------------------------------------------------------------------ -----------------------------------------------------------------------
Restore global settings
-----------------------
Default threshold: 90 %
Age: 30
Skip filetypes: .ass,.ASS,.gif,.GIF,.idx,.IDX,.jfif,.JFIF,.jpeg,.JPEG,.jpg,.JPG,.nfo,.NFO,.png,.PNG,.srt,.SRT,.ssa,.SSA,.sub,.SUB,.svg,.SVG,.txt,.TXT,.vtt,.VTT,.ffs_db,.ffs_lock
Test Mode: yes
************************************************************ ANALYSING MOVING ACTIONS ***********************************************************
Deciding the action (move/sync/keep) for each file. There are 26 files, it can take a while...
A total of 18 files representing 32564397598 bytes (30 GiB) will be moved/synced:
- 18 files representing 32564397598 bytes (30 GiB) will be moved/sync from cache to array
- No new files will be moved/synced from array to cache
*********************************************************** LET THE MOVING SHOW BEGIN ! *********************************************************
Moving "/mnt/cache/./Games/XXXX.zip" to /mnt/user0/
Moving "/mnt/cache/./Games/XXX.zip" to /mnt/user0/
Moving "/mnt/cache/./Games/XXX.zip" to /mnt/user0/
Moving "/mnt/cache/./Games/XXX.zip" to /mnt/user0/
...
...
...
...
****************************************************************** WE ARE DONE ! ****************************************************************
Script Finished Aug 02, 2024 10:23.10

Full logs for this script are available at /tmp/user.scripts/tmpScripts/1111 MOVER FIX/log.txt

Reynald · August 2

6 hours ago, alturismo said:

treehold 90 % (keep cache until ... if 90+ start moving until)

treehold 70 % (stop moving as soon 69 is reached)

[...]
while im watching a "old" tvshow which i moved manually to the cache, the tvshow is inserted in the exluded list until final episode is reached then i remove it and it ll be moved away again as its "old" anyhow.

that would be my current usecase if there would be a "target" treehold ...

OK, the low treshold option will land soon

for the TVShow, I need to make a separate beta as it's a major modification. This can happen mid august. Can you please PM me to give me some details about the player, the Alturismo's way to make the excluded list... ?

Reynald · August 2

43 minutes ago, nmkaufman said:

It is moving. It does not seem to be respecting the threshold %.

I removed the plugin, and set back up from scratch, same behavior.

Analyzing your logs:

------------------------------------------------------------ Settings for Games share -----------------------------------------------------------
Settings override:
------------------
Share threshold: 90 %
Age: no = 0 ; daysold: 30

OK here you set Age = NO and daysold =30, which is ignored.

You probably want this settings for your game share:

(or maybe 90% threshold according to your use case, I let the default here, focusing on age)

[Plugin] Mover Tuning

Recommended Posts

Link to comment

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

Squid

hugenbdd

hugenbdd

Posted Images

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation