[Plugin] Mover Tuning

Reynald · July 29

5 hours ago, Freender said:
3) In /boot/config/plugins/ca.mover.tuning/shareOverrideConfig/media.cfg

threshold is set:
shareThreshold="85"

3) Corrected in 2024-07-29.b
I had to make some adaptation in getMoverSettings() function to not overwrite defaults when no set for share override.

Result:

13:03:47.499 ------------------------------------------------------------ Settings for media share -----------------------------------------------------------
13:03:47.505 Settings override:
13:03:47.509 ------------------
13:03:47.520 Age: Automatic (smart caching)
13:03:47.537 Share threshold: 60 %
13:03:47.552 ------------------------------------------------------------------------  -----------------------------------------------------------------------
13:03:47.562 Cache mode: Yes/Cache->Array. Will smart move old files from Cache to Array. Nothing will be moved from Array to Cache
13:03:47.566 Share Information: Name: media - Path: /mnt/cache/media - Cache mode: yes
13:03:47.590 Pool Information: Name: cache - pool used: 89 % - size: 856872009728 bytes (798 GiB)
13:03:47.606 Default threshold: 95 % - media share threshold: 60 % => Threshold selected: 60 %,s size: 554372903731 bytes (516 GiB)
13:03:48.597 Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-07-29T130347.list for media
13:03:48.606 ----------------------------------------------------------------- Global settings ---------------------------------------------------------------

Edited July 29 by Reynald

Reynald · July 29

New version: 2024-07-30.0000

Implement above fixes and automatically recreate config file if broken from 2024-07-29 version fresh install

Edited July 29 by Reynald

Freender · July 29

@Reynald thanks for your support.

1) Greatly appreciate these 2 settings:

Show advanced filters
Show advanced options:

This makes the plugin cleaner and focuses on the main functions.

2) To make debugging easier I simplified my config (disabled override shares).

My current pool usage is 84% and threshold is 85%.

Expected Result: I don't expect any files to be moved from cache to array with this config

Actual Results: "/tmp/ca.mover.tuning/Filtered_files_2024-07-29T125104.list for media" file has created.

Size - 205K

-rw-r--r--  1 root root 205K Jul 29 12:51 Filtered_files_2024-07-29T125104.list

Full Log:

Jul 29 12:51:04 NAS move: ***************************************************************** FILTERING FILES ***************************************************************
Jul 29 12:51:04 NAS move: ----------------------------------------------------------- Settings for appdata share ----------------------------------------------------------
Jul 29 12:51:04 NAS move: Cache mode: Only. Nothing will be moved. Its usage is taken into account in the calculation of the threshold.
Jul 29 12:51:04 NAS move: Share Information: Name: appdata - Path: /mnt/cache/appdata - Cache mode: only
Jul 29 12:51:04 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3372671008256 bytes (3 TiB)
Jul 29 12:51:04 NAS move: cache/appdata usage: 277430949136 bytes (258 GiB)
Jul 29 12:51:04 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:04 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:04 NAS move: Skipfilelist string: find "/mnt/cache/appdata" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 12:51:04 NAS move: ------------------------------------------------------------ Settings for backup share ----------------------------------------------------------
Jul 29 12:51:04 NAS move: Cache mode: Yes/Cache->Array. Will smart move old files from Cache to Array. Nothing will be moved from Array to Cache
Jul 29 12:51:04 NAS move: Share Information: Name: backup - Path: /mnt/cache/backup - Cache mode: yes
Jul 29 12:51:04 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3372671008256 bytes (3 TiB)
Jul 29 12:51:04 NAS move: Default threshold: 85 % - backup share threshold: NA % => Threshold selected: 85 %,s size: 3288519933952 bytes (2 TiB)
Jul 29 12:51:04 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:04 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:04 NAS move: Skipfilelist string: find "/mnt/cache/backup" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 12:51:04 NAS move: Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-07-29T125104.list for backup
Jul 29 12:51:04 NAS move: ------------------------------------------------------------ Settings for docker share ----------------------------------------------------------
Jul 29 12:51:04 NAS move: Cache mode: Only. Nothing will be moved. Its usage is taken into account in the calculation of the threshold.
Jul 29 12:51:04 NAS move: Share Information: Name: docker - Path: /mnt/cache/docker - Cache mode: only
Jul 29 12:51:04 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3372671008256 bytes (3 TiB)
Jul 29 12:51:04 NAS move: cache/docker usage: 7979181760 bytes (7 GiB)
Jul 29 12:51:04 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:04 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:04 NAS move: Skipfilelist string: find "/mnt/cache/docker" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 12:51:04 NAS move: ------------------------------------------------------------- Settings for isos share -----------------------------------------------------------
Jul 29 12:51:04 NAS move: Cache mode: Yes/Cache->Array. Will smart move old files from Cache to Array. Nothing will be moved from Array to Cache
Jul 29 12:51:04 NAS move: Share Information: Name: isos - Path: /mnt/cache/isos - Cache mode: yes
Jul 29 12:51:04 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3372671008256 bytes (3 TiB)
Jul 29 12:51:04 NAS move: Default threshold: 85 % - isos share threshold: NA % => Threshold selected: 85 %,s size: 3288519933952 bytes (2 TiB)
Jul 29 12:51:04 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:04 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:04 NAS move: Skipfilelist string: find "/mnt/cache/isos" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 12:51:04 NAS move: Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-07-29T125104.list for isos
Jul 29 12:51:04 NAS move: ------------------------------------------------------------ Settings for media share -----------------------------------------------------------
Jul 29 12:51:04 NAS move: Cache mode: Yes/Cache->Array. Will smart move old files from Cache to Array. Nothing will be moved from Array to Cache
Jul 29 12:51:04 NAS move: Share Information: Name: media - Path: /mnt/cache/media - Cache mode: yes
Jul 29 12:51:04 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3372671008256 bytes (3 TiB)
Jul 29 12:51:04 NAS move: Default threshold: 85 % - media share threshold: NA % => Threshold selected: 85 %,s size: 3288519933952 bytes (2 TiB)
Jul 29 12:51:05 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:05 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:05 NAS move: Skipfilelist string: find "/mnt/cache/media" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 12:51:08 NAS move: Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-07-29T125104.list for media
Jul 29 12:51:08 NAS move: ----------------------------------------------------------- Settings for pictures share ---------------------------------------------------------
Jul 29 12:51:08 NAS move: Cache mode: Yes/Cache->Array. Will smart move old files from Cache to Array. Nothing will be moved from Array to Cache
Jul 29 12:51:08 NAS move: Share Information: Name: pictures - Path: /mnt/cache/pictures - Cache mode: yes
Jul 29 12:51:08 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3372671668432 bytes (3 TiB)
Jul 29 12:51:08 NAS move: Default threshold: 85 % - pictures share threshold: NA % => Threshold selected: 85 %,s size: 3288519933952 bytes (2 TiB)
Jul 29 12:51:08 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:08 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:08 NAS move: Skipfilelist string: find "/mnt/cache/pictures" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 12:51:08 NAS move: ------------------------------------------------------------ Settings for steam share -----------------------------------------------------------
Jul 29 12:51:08 NAS move: Cache mode: Only. Nothing will be moved. Its usage is taken into account in the calculation of the threshold.
Jul 29 12:51:08 NAS move: Share Information: Name: steam - Path: /mnt/cache/steam - Cache mode: only
Jul 29 12:51:08 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3372671668432 bytes (3 TiB)
Jul 29 12:51:08 NAS move: cache/steam usage: 55265793616 bytes (51 GiB)
Jul 29 12:51:08 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:08 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:08 NAS move: Skipfilelist string: find "/mnt/cache/steam" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 12:51:08 NAS move: ------------------------------------------------------------ Settings for system share ----------------------------------------------------------
Jul 29 12:51:08 NAS move: Cache mode: Only. Nothing will be moved. Its usage is taken into account in the calculation of the threshold.
Jul 29 12:51:08 NAS move: Share Information: Name: system - Path: /mnt/cache/system - Cache mode: only
Jul 29 12:51:08 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3372671668432 bytes (3 TiB)
Jul 29 12:51:08 NAS move: cache/system usage: 152768 bytes (149 KiB)
Jul 29 12:51:08 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:08 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:08 NAS move: Skipfilelist string: find "/mnt/cache/system" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 12:51:08 NAS move: --------------------------------------------------------- Settings for time_machine share -------------------------------------------------------
Jul 29 12:51:08 NAS move: Cache mode: Yes/Cache->Array. Will smart move old files from Cache to Array. Nothing will be moved from Array to Cache
Jul 29 12:51:08 NAS move: Share Information: Name: time_machine - Path: /mnt/cache/time_machine - Cache mode: yes
Jul 29 12:51:08 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3372671668432 bytes (3 TiB)
Jul 29 12:51:08 NAS move: Default threshold: 85 % - time_machine share threshold: NA % => Threshold selected: 85 %,s size: 3288519933952 bytes (2 TiB)
Jul 29 12:51:08 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:08 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:08 NAS move: Skipfilelist string: find "/mnt/cache/time_machine" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 12:51:08 NAS move: ------------------------------------------------------- Settings for time_machine_pc share ------------------------------------------------------
Jul 29 12:51:08 NAS move: Cache mode: Yes/Cache->Array. Will smart move old files from Cache to Array. Nothing will be moved from Array to Cache
Jul 29 12:51:08 NAS move: Share Information: Name: time_machine_pc - Path: /mnt/cache/time_machine_pc - Cache mode: yes
Jul 29 12:51:08 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3372671668432 bytes (3 TiB)
Jul 29 12:51:08 NAS move: Default threshold: 85 % - time_machine_pc share threshold: NA % => Threshold selected: 85 %,s size: 3288519933952 bytes (2 TiB)
Jul 29 12:51:08 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:08 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 12:51:08 NAS move: Skipfilelist string: find "/mnt/cache/time_machine_pc" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 12:51:08 NAS move: ---------------------------------------------------------- Settings for zfs_backup share --------------------------------------------------------
Jul 29 12:51:08 NAS move: No shareCachePool entry found in config file, defaulting to cache
Jul 29 12:51:08 NAS move: Cache mode: No. Skipping
Jul 29 12:51:08 NAS move: ************************************************************ ANALYSING MOVING ACTIONS ***********************************************************
Jul 29 12:51:08 NAS move: Deciding the action (move/sync/keep) for each file. There are 1065 files, it can take a while...
Jul 29 12:51:10 NAS move: No new files will be moved/synced from cache to array
Jul 29 12:51:10 NAS move: No new files will be moved/synced from array to cache
Jul 29 12:51:10 NAS move: ****************************************************************** WE ARE DONE ! ****************************************************************

Reynald · July 29

2 hours ago, Freender said:
@Reynald thanks for your support.

1) Greatly appreciate these 2 settings:

Show advanced filters

Show advanced options:

This makes the plugin cleaner and focuses on the main functions.

2) To make debugging easier I simplified my config (disabled override shares).

My current pool usage is 84% and threshold is 85%.

Expected Result: I don't expect any files to be moved from cache to array with this config

Actual Results: "/tmp/ca.mover.tuning/Filtered_files_2024-07-29T125104.list for media" file has created.

Size - 205K
-rw-r--r--  1 root root 205K Jul 29 12:51 Filtered_files_2024-07-29T125104.list
Full Log:
Jul 29 12:51:08 NAS move: ************************************************************ ANALYSING MOVING ACTIONS ***********************************************************
Jul 29 12:51:08 NAS move: Deciding the action (move/sync/keep) for each file. There are 1065 files, it can take a while...
Jul 29 12:51:10 NAS move: No new files will be moved/synced from cache to array
Jul 29 12:51:10 NAS move: No new files will be moved/synced from array to cache
Jul 29 12:51:10 NAS move: ****************************************************************** WE ARE DONE ! ****************************************************************

Edited July 29 by Reynald

Reynald · July 29

@Freender reading your log again.

I don't think it's necessary to log "Skipfilelist string:" as the find string (and not the found string 👙) is really large and only usefull for debugging purpose.

I may remove that in a next version. What do you (and all others?) think?

Freender · July 29

Yeah, all files in Mover_action_*.list have 'keep' option

Case #2 (Auto moving enabled, there are no share overrides)

• Pool usage: 84%

• Threshold: 80%

Actual Result:

All files in Mover_action_*.list have the ‘keep’ action. Just in case, I disabled Test Mode - nothing is moved.

Expected Result:

After running the mover, some files in Mover_action_*.list should have the ‘move’ action.

Jul 29 16:32:01 NAS move: ***************************************************************** FILTERING FILES ***************************************************************
Jul 29 16:32:01 NAS move: ----------------------------------------------------------- Settings for appdata share ----------------------------------------------------------
Jul 29 16:32:01 NAS move: Cache mode: Only. Nothing will be moved. Its usage is taken into account in the calculation of the threshold.
Jul 29 16:32:01 NAS move: Share Information: Name: appdata - Path: /mnt/cache/appdata - Cache mode: only
Jul 29 16:32:01 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3373779198240 bytes (3 TiB)
Jul 29 16:32:01 NAS move: cache/appdata usage: 277686311760 bytes (258 GiB)
Jul 29 16:32:01 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:01 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:01 NAS move: Skipfilelist string: find "/mnt/cache/appdata" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 16:32:01 NAS move: ------------------------------------------------------------ Settings for backup share ----------------------------------------------------------
Jul 29 16:32:01 NAS move: Cache mode: Yes/Cache->Array. Will smart move old files from Cache to Array. Nothing will be moved from Array to Cache
Jul 29 16:32:01 NAS move: Share Information: Name: backup - Path: /mnt/cache/backup - Cache mode: yes
Jul 29 16:32:01 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3373779198240 bytes (3 TiB)
Jul 29 16:32:01 NAS move: Default threshold: 80 % - backup share threshold: NA % => Threshold selected: 80 %,s size: 3095077584896 bytes (2 TiB)
Jul 29 16:32:01 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:01 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:01 NAS move: Skipfilelist string: find "/mnt/cache/backup" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 16:32:01 NAS move: Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-07-29T163201.list for backup
Jul 29 16:32:01 NAS move: ------------------------------------------------------------ Settings for docker share ----------------------------------------------------------
Jul 29 16:32:01 NAS move: Cache mode: Only. Nothing will be moved. Its usage is taken into account in the calculation of the threshold.
Jul 29 16:32:01 NAS move: Share Information: Name: docker - Path: /mnt/cache/docker - Cache mode: only
Jul 29 16:32:01 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3373779198240 bytes (3 TiB)
Jul 29 16:32:01 NAS move: cache/docker usage: 7973245632 bytes (7 GiB)
Jul 29 16:32:01 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:01 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:01 NAS move: Skipfilelist string: find "/mnt/cache/docker" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 16:32:01 NAS move: ------------------------------------------------------------- Settings for isos share -----------------------------------------------------------
Jul 29 16:32:01 NAS move: Cache mode: Yes/Cache->Array. Will smart move old files from Cache to Array. Nothing will be moved from Array to Cache
Jul 29 16:32:01 NAS move: Share Information: Name: isos - Path: /mnt/cache/isos - Cache mode: yes
Jul 29 16:32:01 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3373779198240 bytes (3 TiB)
Jul 29 16:32:01 NAS move: Default threshold: 80 % - isos share threshold: NA % => Threshold selected: 80 %,s size: 3095077584896 bytes (2 TiB)
Jul 29 16:32:01 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:01 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:01 NAS move: Skipfilelist string: find "/mnt/cache/isos" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 16:32:01 NAS move: Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-07-29T163201.list for isos
Jul 29 16:32:01 NAS move: ------------------------------------------------------------ Settings for media share -----------------------------------------------------------
Jul 29 16:32:01 NAS move: Cache mode: Yes/Cache->Array. Will smart move old files from Cache to Array. Nothing will be moved from Array to Cache
Jul 29 16:32:01 NAS move: Share Information: Name: media - Path: /mnt/cache/media - Cache mode: yes
Jul 29 16:32:01 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3373779198240 bytes (3 TiB)
Jul 29 16:32:01 NAS move: Default threshold: 80 % - media share threshold: NA % => Threshold selected: 80 %,s size: 3095077584896 bytes (2 TiB)
Jul 29 16:32:01 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:01 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:01 NAS move: Skipfilelist string: find "/mnt/cache/media" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 16:32:04 NAS move: Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-07-29T163201.list for media
Jul 29 16:32:04 NAS move: ----------------------------------------------------------- Settings for pictures share ---------------------------------------------------------
Jul 29 16:32:04 NAS move: Cache mode: Yes/Cache->Array. Will smart move old files from Cache to Array. Nothing will be moved from Array to Cache
Jul 29 16:32:04 NAS move: Share Information: Name: pictures - Path: /mnt/cache/pictures - Cache mode: yes
Jul 29 16:32:04 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3373779198240 bytes (3 TiB)
Jul 29 16:32:04 NAS move: Default threshold: 80 % - pictures share threshold: NA % => Threshold selected: 80 %,s size: 3095077584896 bytes (2 TiB)
Jul 29 16:32:04 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:04 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:04 NAS move: Skipfilelist string: find "/mnt/cache/pictures" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 16:32:04 NAS move: Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-07-29T163201.list for pictures
Jul 29 16:32:04 NAS move: ------------------------------------------------------------ Settings for steam share -----------------------------------------------------------
Jul 29 16:32:04 NAS move: Cache mode: Only. Nothing will be moved. Its usage is taken into account in the calculation of the threshold.
Jul 29 16:32:04 NAS move: Share Information: Name: steam - Path: /mnt/cache/steam - Cache mode: only
Jul 29 16:32:04 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3373779198240 bytes (3 TiB)
Jul 29 16:32:04 NAS move: cache/steam usage: 55265793616 bytes (51 GiB)
Jul 29 16:32:04 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:04 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:04 NAS move: Skipfilelist string: find "/mnt/cache/steam" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 16:32:04 NAS move: ------------------------------------------------------------ Settings for system share ----------------------------------------------------------
Jul 29 16:32:04 NAS move: Cache mode: Only. Nothing will be moved. Its usage is taken into account in the calculation of the threshold.
Jul 29 16:32:04 NAS move: Share Information: Name: system - Path: /mnt/cache/system - Cache mode: only
Jul 29 16:32:04 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3373779198240 bytes (3 TiB)
Jul 29 16:32:04 NAS move: cache/system usage: 152768 bytes (149 KiB)
Jul 29 16:32:04 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:04 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:04 NAS move: Skipfilelist string: find "/mnt/cache/system" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 16:32:04 NAS move: --------------------------------------------------------- Settings for time_machine share -------------------------------------------------------
Jul 29 16:32:04 NAS move: Cache mode: Yes/Cache->Array. Will smart move old files from Cache to Array. Nothing will be moved from Array to Cache
Jul 29 16:32:04 NAS move: Share Information: Name: time_machine - Path: /mnt/cache/time_machine - Cache mode: yes
Jul 29 16:32:04 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3373779198240 bytes (3 TiB)
Jul 29 16:32:04 NAS move: Default threshold: 80 % - time_machine share threshold: NA % => Threshold selected: 80 %,s size: 3095077584896 bytes (2 TiB)
Jul 29 16:32:04 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:04 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:04 NAS move: Skipfilelist string: find "/mnt/cache/time_machine" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 16:32:04 NAS move: ------------------------------------------------------- Settings for time_machine_pc share ------------------------------------------------------
Jul 29 16:32:04 NAS move: Cache mode: Yes/Cache->Array. Will smart move old files from Cache to Array. Nothing will be moved from Array to Cache
Jul 29 16:32:04 NAS move: Share Information: Name: time_machine_pc - Path: /mnt/cache/time_machine_pc - Cache mode: yes
Jul 29 16:32:04 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3373779198240 bytes (3 TiB)
Jul 29 16:32:04 NAS move: Default threshold: 80 % - time_machine_pc share threshold: NA % => Threshold selected: 80 %,s size: 3095077584896 bytes (2 TiB)
Jul 29 16:32:04 NAS move: Adding Skip Folder List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:04 NAS move: Adding Skip File List. List Path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 16:32:04 NAS move: Skipfilelist string: find "/mnt/cache/time_machine_pc" -type f -depth -not -path '/mnt/cache/media/downloads/*' -printf '%T@|%s|%S|%n|%i|%p|\0' | awk -v RS='\0' -v FS='|' -v SPARSENESS=$SPARSENESS -v CACHEPOOLNAME=$CACHEPOOLNAME -v SHARENAME=$SHARENAME -v SHAREUSECACHE=$SHAREUSECACHE -v CACHESIZETHRESH=$CACHESIZETHRESH '{cmd="stat -c%Y \""$6"\""; cmd | getline result; close(cmd); sub(/\n$/, "", result); printf "%s|%s|%s|%d|%d|%d|%s|%d|%d|%s\n", CACHEPOOLNAME, SHARENAME, SHAREUSECACHE, result, CACHESIZETHRESH, $2, $3, $4, $5, $6}' | grep -vFx -f <(sed 's/\/*$//' '/mnt/user/backup/mover-ignore/mover_ignore.txt')
Jul 29 16:32:04 NAS move: ---------------------------------------------------------- Settings for zfs_backup share --------------------------------------------------------
Jul 29 16:32:04 NAS move: No shareCachePool entry found in config file, defaulting to cache
Jul 29 16:32:04 NAS move: Cache mode: No. Skipping
Jul 29 16:32:04 NAS move: ************************************************************ ANALYSING MOVING ACTIONS ***********************************************************
Jul 29 16:32:04 NAS move: Deciding the action (move/sync/keep) for each file. There are 1203 files, it can take a while...
Jul 29 16:32:06 NAS move: No new files will be moved/synced from cache to array
Jul 29 16:32:06 NAS move: No new files will be moved/synced from array to cache

Action Log:

CACHEPOOLNAME|SHARENAME|SHAREUSECACHE|MODIFICATIONTIME|FILESIZE|POOLSUM|THRESHOLD|NBLINKS|INODE|ACTION|FROM|TO|FILEPATH
cache|appdata|only|9999999999|277686311760|277686311760|3095077584896|0|0|keep|/mnt/cache|/mnt/cache|appdata
cache|docker|only|9999999999|7973245632|285659557392|3095077584896|0|0|keep|/mnt/cache|/mnt/cache|docker
cache|steam|only|9999999999|55265793616|340925351008|3095077584896|0|0|keep|/mnt/cache|/mnt/cache|steam
cache|system|only|9999999999|152768|340925503776|3095077584896|0|0|keep|/mnt/cache|/mnt/cache|system
cache|pictures|yes|1722283458|396327|340925900103|3095077584896|1|862|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/05/f3/05f365ca-6848-4f61-8d66-651dc1c61b43.mp4
cache|pictures|yes|1722283457|247340|340926147443|3095077584896|1|1000|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/cc/d8/ccd8c0a4-ff17-49f2-8d3f-c0d8ec604e51.mp4
cache|pictures|yes|1722283454|11277940|340937425383|3095077584896|1|1002|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/eb/c6/ebc6e997-c421-455e-a102-044d1eec1af7.mp4
cache|pictures|yes|1722283331|328314|340937753697|3095077584896|1|226|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/cb/9d/cb9dfcad-3931-453c-af93-eef97eb3dff1.mp4
cache|pictures|yes|1722283328|444276|340938197973|3095077584896|1|1426|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/e0/0e/e00efad8-18b6-49a6-acde-932b094f8ffd.mp4
cache|pictures|yes|1722283325|471987|340938669960|3095077584896|1|1260|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/80/b5/80b57dca-7fea-4d4f-ba74-9cb154a15e0b.mp4
cache|pictures|yes|1722283323|847081|340939517041|3095077584896|1|998|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/4c/27/4c272d3e-cef8-4b3d-8ab9-482cd179ad0b.mp4
cache|pictures|yes|1722283319|5452932|340944969973|3095077584896|1|1098|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/4d/f9/4df9f2da-73fa-41e5-a7cb-09a858566934.mp4
cache|pictures|yes|1722283222|169987|340945139960|3095077584896|1|852|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/13/eb/13ebc94e-2058-4079-b69e-9e1ef8fabcb9.mp4
cache|pictures|yes|1722283220|317874|340945457834|3095077584896|1|370|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/0b/78/0b78bb97-8e8a-4300-adbd-34a2e30bfd4e.mp4
cache|pictures|yes|1722283218|965163|340946422997|3095077584896|1|1418|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/8a/25/8a25a257-22ef-45f1-955b-239ad4e473ca.mp4
cache|pictures|yes|1722283215|63614495|341010037492|3095077584896|1|1412|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/86/61/86616870-4569-4444-8e3e-3ba8dcc33a30.mp4
cache|pictures|yes|1722283034|17769864|341027807356|3095077584896|1|850|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/7a/c3/7ac3575f-e331-430e-8253-20641257c074.mp4
cache|pictures|yes|1722282979|29671005|341057478361|3095077584896|1|426|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/22/75/22755539-de27-4f3e-94ed-c4308112e2ec.mp4
cache|pictures|yes|1722282746|748213|341058226574|3095077584896|1|62|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/c7/8d/c78ddcce-4082-4fcc-aa70-6340d477cc37.mp4
cache|pictures|yes|1722282742|719743|341058946317|3095077584896|1|1410|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/a6/9b/a69b3765-36da-4a50-9084-9f5dfd6eb353.mp4
cache|pictures|yes|1722282738|696341|341059642658|3095077584896|1|992|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/9a/f7/9af744de-8973-49d0-b9e7-dbfc1be26974.mp4
cache|pictures|yes|1722282736|1008149|341060650807|3095077584896|1|1408|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/87/00/870092eb-744e-4868-81b4-87e66a759539.mp4
cache|pictures|yes|1722282732|745384|341061396191|3095077584896|1|224|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/64/cb/64cbd6e8-e60f-4a0c-8d2d-df7385784b37.mp4
cache|pictures|yes|1722282729|1532399|341062928590|3095077584896|1|844|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/72/2f/722f77e1-d8d6-492e-b760-99b1a2b1b97d.mp4
cache|pictures|yes|1722282726|564063|341063492653|3095077584896|1|424|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/84/05/8405a84f-9731-48cf-9cf1-887b5a661ace.mp4
cache|pictures|yes|1722282723|332964|341063825617|3095077584896|1|630|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/39/5f/395f4cbd-4307-47c4-96ce-38eeab8d7394.mp4
cache|pictures|yes|1722282717|227759|341064053376|3095077584896|1|628|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/6e/42/6e4287eb-8eaf-4328-9f6e-e069d66135cd.mp4
cache|pictures|yes|1722282716|710732|341064764108|3095077584896|1|1246|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/6b/01/6b01113e-2d6c-42d6-a9ad-ff2959461d75.mp4
cache|pictures|yes|1722282713|239270|341065003378|3095077584896|1|758|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/b9/f1/b9f1442c-1f51-4920-8615-c4e8d107b4ae.mp4
cache|pictures|yes|1722282711|716480|341065719858|3095077584896|1|1244|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/2c/0b/2c0be43b-a718-4312-bbba-281d37e9d57b.mp4
cache|pictures|yes|1722282708|985418|341066705276|3095077584896|1|60|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/f3/e9/f3e9f6d4-829b-4527-a280-805c7f852b07.mp4
cache|pictures|yes|1722282706|499796|341067205072|3095077584896|1|1090|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/3b/39/3b3914a9-6bed-47a7-82dd-30472b34e4e1.mp4
cache|pictures|yes|1722282703|526965|341067732037|3095077584896|1|364|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/30/d0/30d0f58f-a5a4-4cab-901d-f2da02e34d4a.mp4
cache|pictures|yes|1722282701|262976|341067995013|3095077584896|1|842|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/be/08/be0832b6-a144-4eaf-8925-072b02a1f0be.mp4
cache|pictures|yes|1722282698|376841|341068371854|3095077584896|1|362|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/46/35/46355910-534a-41f7-b364-578f1e391cbd.mp4
cache|pictures|yes|1722282690|263129|341068634983|3095077584896|1|978|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/99/4e/994e661a-c258-4038-839e-d2553c827b59.mp4
cache|pictures|yes|1722282669|293243|341068928226|3095077584896|1|972|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/f2/00/f20000ba-1dbf-45fc-ae44-4cca95480c98.mp4
cache|pictures|yes|1722282665|305887|341069234113|3095077584896|1|1088|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/80/0a/800af07a-74bc-4106-b91e-67137a7e477b.mp4
cache|pictures|yes|1722282658|424926|341069659039|3095077584896|1|744|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/41/fd/41fddc1e-b924-40d4-a136-612e245f267e.mp4
cache|pictures|yes|1722282650|413132|341070072171|3095077584896|1|1242|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/0a/05/0a05aedd-6240-4c49-9a9f-6cf61b841985.mp4
cache|pictures|yes|1722282634|750048|341070822219|3095077584896|1|742|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/eb/9b/eb9bd1ae-a280-481a-afae-ddb63bd72492.mp4
cache|pictures|yes|1722282611|555211|341071377430|3095077584896|1|618|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/1a/f9/1af9f462-3ab3-4f72-b629-623373b494bb.mp4
cache|pictures|yes|1722282595|222918|341071600348|3095077584896|1|356|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/de/bf/debf44f6-01f1-40e0-b49e-1d3a2b17569b.mp4
cache|pictures|yes|1722282580|343680|341071944028|3095077584896|1|1236|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/dc/a8/dca875f0-d107-4dd5-9309-974a4c01f396.mp4
cache|pictures|yes|1722282570|536552|341072480580|3095077584896|1|740|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/7a/8f/7a8f46f4-c35e-4bb5-a2fb-d70b5fa0127b.mp4
cache|pictures|yes|1722282559|379264|341072859844|3095077584896|1|20|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/51/29/5129d3f8-e82c-4c6c-abd0-60ecccdf60fb.mp4
cache|pictures|yes|1722282531|673632|341073533476|3095077584896|1|1062|keep|/mnt/cache|/mnt/cache|pictures/encoded-video/239e7fe0-5449-4db9-a2df-b6ffbc90c873/6a/23/6a23f136-b12f-42dd-8b94-bc6a02360a79.mp4
cache|pictures|yes|1722282509|10166262|341083699738|3095077584896|1|54|keep|/mnt/cache|/mnt/cache|pictures/upload/239e7fe0-5449-4db9-a2df-b6ffbc90c873/8d/32/8d32443b-af33-45b7-a460-57930a704c72.MOV
...

Reynald · July 29

If no sensible data inside, can you post Mover_action please?

Else, please tail it.

image.png.d3fcad85bf9f08d63dbbd9622a54b494.png
At some point, green number shall be > yellow number and action be move,

Freender · July 29

sent via PM

Reynald · July 29

OK, got it !

https://github.com/R3yn4ld/ca.mover.tuning/blob/5747150459a4a362de4fc5020feb8381c0eaa2d0/source/ca.mover.tuning/usr/local/emhttp/plugins/ca.mover.tuning/age_mover#L573

# Make decision to move or keep:
action = ($3 != "only" && ($3 == "prefer" && (pool_sums[$1] >= $5 && AGE == -1) || AGE >= 0)) ? "move" : "keep"

No decision taken for cache = no and cache = yes shares...

Reynald · July 29

Above bug corrected, thank you @Freender

2024-07-30.0038:

Various bug fixes R3yn4ld Freender

Enhance internal mover function R3yn4ld

Freender · July 30

Fantastic! I have set the threshold to 0% for all shares (e.g. I like when appdata backups moved to array overnight) and overridden it to 85% for media. According to the logs (test mode), everything should work as expected. Let’s check the test results tonight.

One more unrelated question: How is the ‘Synchronize cache files to array’ feature designed to work?

It would be great to have all files on the ‘Media’ share synchronized with the array, so if there isn’t enough space available on the cache, the mover could simply delete the file from the cache without needing to write anything to the array (if the file is already in sync).

Reynald · July 30

2 hours ago, Freender said:

One more unrelated question: How is the ‘Synchronize cache files to array’ feature designed to work?

It would be great to have all files on the ‘Media’ share synchronized with the array, so if there isn’t enough space available on the cache, the mover could simply delete the file from the cache without needing to write anything to the array (if the file is already in sync).

Synchronize cache files to array will rsync cache=prefer and cache=yes shares to array.

Move will rsync --remove-source-files (delete) them, which is basically a deletion for file that have not been modified.

Rsync flags are similar to 6.x mover and Unbalanced plugin.

This is not yet a per share setting.

It's there in the code: https://github.com/R3yn4ld/ca.mover.tuning/blob/b611aaba62b083622598900b8146edd32b2d180e/source/ca.mover.tuning/usr/local/emhttp/plugins/ca.mover.tuning/age_mover#L718

This is specifically for this feature and a better hardkinks management that I stopped using the UnRaid mover

Edited July 30 by Reynald

alturismo · July 30

may a question how you fetch available disks / shares, i see the following error here

/mnt/usb_cache

which is not existing (anymore), since a few month actually ...

root@AlsServerII:~# ls -la /mnt/
total 0
drwxr-xr-x 13 root   root  260 Jul 29 06:16 ./
drwxr-xr-x 22 root   root  480 Jul 30 00:38 ../
drwxr-xr-x  6 nobody users 140 Jul 29 06:15 RecycleBin/
drwxrwxrwt 11 nobody users 220 Jul 29 06:15 addons/
drwxr-xr-x  2 root   root   40 Jul 29 06:16 boot/
drwxrwxrwx 12 nobody users 153 May  6 06:25 cache/
drwxrwxrwx  3 nobody users  19 Dec  7  2023 disk1/
drwxrwxrwx  3 nobody users  19 Dec  7  2023 disk2/
drwxrwxrwt  3 nobody users  60 Jul 30 06:27 disks/
drwxrwxrwt  4 nobody users  80 Jul 29 06:15 remotes/
drwxrwxrwt  2 nobody users  40 Jul 29 06:15 rootshare/
drwxrwxrwx  1 nobody users  19 May  6 06:25 user/
drwxrwxrwx  1 nobody users  19 Dec  7  2023 user0/
root@AlsServerII:~#

alturismo · July 30

and while im on it, may i ask if i have something wrong in mind.

before there was a treehold how much to free up

sample

trigger at 90 %

free up to 70 % (1 file lower)

the 70 % option is gone ? so will it move now all away ? or is the age option only left and it ll move just a little lower 90 % now ?

Reynald · July 30

59 minutes ago, alturismo said:

may a question how you fetch available disks / shares, i see the following error here

/mnt/usb_cache

which is not existing (anymore), since a few month actually ...

Available shares are found by reading the config files:

for SHARECFG in /boot/config/shares/*; do
	[...]
	SHARENAME="$(basename "$SHARECFG" .cfg)"

Your error is because you delete the share content but not the share via the UnRaid GUI. I'm adding error control in the next version

Reynald · July 30

57 minutes ago, alturismo said:

and while im on it, may i ask if i have something wrong in mind.

before there was a treehold how much to free up

sample

trigger at 90 %

free up to 70 % (1 file lower)

the 70 % option is gone ? so will it move now all away ? or is the age option only left and it ll move just a little lower 90 % now ?

I'm not aware of the 70% option. I guess you mix trigger and free up with another plugin/script (that's how my former "Smart caching script" worked before, and there are some script intended to be ran by this mover tuning plugin, like Terebi's one).

Here is an explanation, to make it simple I assume there is no other (size, sparness, etc..) filters.
Since previous versions, this plugin filters the cache file list and the move of filtered files are triggered based on threshold. If you filter with AGE=7, passed the threshold all files older than 7 days will be moved. If AGE=0 everything is move.
Now (since I maintain this plugin) if you filter with AGE=Automatic, only old files that are above the threshold will be moved.

That means that if you set threshold to 70%, when mover is invoked, if you are at 87% it will free the cache down to 70%.

alturismo · July 30

1 hour ago, Reynald said:

Available shares are found by reading the config files:

root@AlsServerII:~# ls -la /boot/config/shares/
total 76
drw-r--r--  2 root root 4096 Apr 25 12:38 ./
drw-r--r-- 12 root root 4096 Jul 29 06:15 ../
-rw-r--r--  1 root root  445 Nov  3  2023 Daten.cfg
-rw-r--r--  1 root root  445 Nov  3  2023 Dokumente.cfg
-rw-r--r--  1 root root  449 Dec 28  2023 Download.cfg
-rw-r--r--  1 root root  444 Nov  3  2023 Media.cfg
-rw-r--r--  1 root root  428 Apr  5 23:04 Nextcloud.cfg
-rw-r--r--  1 root root  418 Apr  1 23:51 Nextcloud_Backup.cfg
-rw-r--r--  1 root root  449 Dec 10  2023 _old_cache.cfg
-rw-r--r--  1 root root  436 Nov  3  2023 appdata.cfg
-rw-r--r--  1 root root  435 Dec 10  2023 backup_backup.cfg
-rw-r--r--  1 root root  464 Nov  3  2023 domains.cfg
-rw-r--r--  1 root root  435 Dec 10  2023 domains_backup.cfg
-rw-r--r--  1 root root  455 Nov  3  2023 isos.cfg
-rw-r--r--  1 root root  435 Dec 10  2023 luckybackup.cfg
-rw-r--r--  1 root root  445 Nov  3  2023 lxc.cfg
-rw-r--r--  1 root root  435 Dec 10  2023 lxc_Backup.cfg
-rw-r--r--  1 root root  457 Nov  3  2023 system.cfg
-rw-r--r--  1 root root  435 Dec 10  2023 vu_backup.cfg
root@AlsServerII:~#

ok ... there are some ... i ll clean them up too, nice hint due the error message

alturismo · July 30

2 hours ago, Reynald said:

Here is an explanation, to make it simple I assume there is no other (size, sparness, etc..) filters.
Since previous versions, this plugin filters the cache file list and the move of filtered files are triggered based on threshold. If you filter with AGE=7, passed the threshold all files older than 7 days will be moved. If AGE=0 everything is move.
Now (since I maintain this plugin) if you filter with AGE=Automatic, only old files that are above the threshold will be moved.

That means that if you set threshold to 70%, when mover is invoked, if you are at 87% it will free the cache down to 70%.

ok, thanks for the Info.

so my usecase wont be really handled then as expected anymore, thanks for clarifying

AnnabellaRenee87 · July 30

Was running the latest version of the plugin as of writting this post, If I manually invoked the Mover it would just gray the button out and not actually move the files over to the array.

I had to go to the Media Share (that's the share that was being updated) and hit Move in there and it cancled the move in progress, I did this like 2 or 3 times and it would never move, I removed the Plugin and found that after removing it the files are moving again, i'm on the latest UNRAID 7 Beta 2 and here's my logs if they can help.

server-diagnostics-20240730-0755.zip

RonneBlaze · July 30

When i run 'mover start' from terminal i see all the output, it lists Settings for shares that I do not even have.

---------------------------------------------------------- Settings for Nextcloud share ---------------------------------------------------------
Cache mode: No. Skipping
------------------------------------------------------------ Settings for Photos share ----------------------------------------------------------
Cache mode: No. Skipping
------------------------------------------------------------ Settings for mysql share -----------------------------------------------------------
Cache mode: No. Skipping
------------------------------------------------------ Settings for performance_schema share ----------------------------------------------------
Cache mode: No. Skipping
------------------------------------------------------------- Settings for sys share ------------------------------------------------------------
Cache mode: No. Skipping

EDIT: i found where these were coming from. on the flash drive config > shares. I guess the cfg files did not ge deleted when i removed the shares.

Edited July 30 by RonneBlaze

Terebi · July 30

@Reynald

Quote

Automatic Array to Cache

I am interpreting that to mean that if I have the auto age set to 80%, and I am at 20%, the mover will fill the cache back up to 80% with the newest files. Is that accurate? If so, how do I enable that? In the logs for the relevant share I see :

Quote

Cache mode: Yes/Cache->Array. Will smart move old files from Cache to Array. Nothing will be moved from Array to Cache
Share Information: Name: data - Path: /mnt/cache/data - Cache mode: yes
Pool Information: Name: cache - pool used: 28 % - size: 269765861376 bytes (251 GiB)
Default threshold: 80 % - data share threshold: NA % => Threshold selected: 80 %,s size: 800163069952 bytes (745 GiB)
Filtering data files... (can take a moment)
Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-07-30T101859.list for data

Reynald · July 30

7 hours ago, AnnabellaRenee87 said:

Was running the latest version of the plugin as of writting this post, If I manually invoked the Mover it would just gray the button out and not actually move the files over to the array.

I had to go to the Media Share (that's the share that was being updated) and hit Move in there and it cancled the move in progress, I did this like 2 or 3 times and it would never move, I removed the Plugin and found that after removing it the files are moving again, i'm on the latest UNRAID 7 Beta 2 and here's my logs if they can help.

server-diagnostics-20240730-0755.zip 243.76 kB · 0 downloads

I cannot see mover in the log you provided. Are you sure of your configuration? (schedule mover and mover tuning)

Reynald · July 30

5 hours ago, RonneBlaze said:

EDIT: i found where these were coming from. on the flash drive config > shares. I guess the cfg files did not ge deleted when i removed the shares.

Next version will be verbose about that: if it find a config but the share is not on system

Reynald · July 30

4 hours ago, Terebi said:

@Reynald

I am interpreting that to mean that if I have the auto age set to 80%, and I am at 20%, the mover will fill the cache back up to 80% with the newest files. Is that accurate? If so, how do I enable that? In the logs for the relevant share I see :

Yes, but for Cache:Prefer (Array->Cache) shares only. For cache:yes (Cache->Array), it will keep them if it's not needed to free the cache (i.e usage below threshold). In any case, for freeing the cache, oldest Cache:Yes and Cache:Prefer are moved to array in chronological order until threshold is met

Terebi · July 30

1 hour ago, Reynald said:

Yes, but for Cache:Prefer (Array->Cache) shares only. For cache:yes (Cache->Array), it will keep them if it's not needed to free the cache (i.e usage below threshold). In any case, for freeing the cache, oldest Cache:Yes and Cache:Prefer are moved to array in chronological order until threshold is met

So I set my "media" share to array->cache, but the tuner will actually empty it from cache->array to keep it below 80%?

[Plugin] Mover Tuning

Recommended Posts

Link to comment

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

Squid

hugenbdd

hugenbdd

Posted Images

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation