[Plugin] Mover Tuning

Terebi · July 19

@Reynald I rebooted and reinstalled and the link appears to be working now.

Edited July 19 by Terebi

Reynald · July 19

Yes, as stated above, there is a bug in autoage with the actual version of my fork.

It's not emptying, but it is not respecting the threshold

Reynald · July 21

On 7/19/2024 at 5:05 PM, Reynald said:

Coming soon:

Automatic (by age) Array -> Cache handling:
A file list is created with a little more datafield than previously and sorted by:

Modification time

Cache: only/prefer/yes/no

Inodes (for hardlinks

Teasing: next version log example:

01:23:22.137 *********************************MOVER START*******************************
01:23:22.139 Age supplied: Automatic
01:23:22.147 --------------------------------- /boot/config/shares/appdata.cfg -----------------------------------------
01:23:22.154 Cache mode: only. Nothing will be moved. Its usage is taken into account in the calculation of the threshold.
01:23:22.155 Share Information: Name: appdata; Pool Name: cache; Path: /mnt/cache/appdata; Cache mode: only
01:23:22.174 Pool usage: 75 %, size: 723566358528 (673 GiB)
01:23:22.198 cache/appdata usage: 165967560704 bytes (154 GiB)
01:23:22.200 --------------------------------- /boot/config/shares/backup.cfg -----------------------------------------
01:23:22.202 Cache mode: NO. Skipping.
01:23:22.203 --------------------------------- /boot/config/shares/frigate.cfg -----------------------------------------
01:23:22.210 Cache mode: Prefer/Array->Cache. Will smart move newest files from Array to Cache until threshold. Older files will be moved from Cache to Array.
01:23:22.211 Share Information: Name: frigate; Pool Name: cache; Path: /mnt/user/frigate; Cache mode: prefer
01:23:22.228 Pool usage: 75 %, size: 723566358528 (673 GiB)
01:23:22.242 Default threshold: 90 %, frigate share threshold: NA % => Threshold selected: 90 %, size: 831559355596 bytes (774 GiB)
01:23:22.955 Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-07-22T012322.list for frigate
01:23:22.956 --------------------------------- /boot/config/shares/media.cfg -----------------------------------------
01:23:22.963 Cache mode: Prefer/Array->Cache. Will smart move newest files from Array to Cache until threshold. Older files will be moved from Cache to Array.
01:23:22.965 Share Information: Name: media; Pool Name: cache; Path: /mnt/user/media; Cache mode: prefer
01:23:22.984 Pool usage: 75 %, size: 723566358528 (673 GiB)
01:23:22.999 Default threshold: 90 %, media share threshold: NA % => Threshold selected: 90 %, size: 831559355596 bytes (774 GiB)
01:23:58.014 Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-07-22T012322.list for media
01:23:58.015 --------------------------------- /boot/config/shares/nextcloud.borg.cfg -----------------------------------------
01:23:58.018 Cache mode: NO. Skipping.
01:23:58.019 --------------------------------- /boot/config/shares/nextcloud.cfg -----------------------------------------
01:23:58.026 Cache mode: only. Nothing will be moved. Its usage is taken into account in the calculation of the threshold.
01:23:58.027 Share Information: Name: nextcloud; Pool Name: cache; Path: /mnt/cache/nextcloud; Cache mode: only
01:23:58.045 Pool usage: 75 %, size: 723587424256 (673 GiB)
01:23:58.068 cache/nextcloud usage: 373098090496 bytes (347 GiB)
01:23:58.071 --------------------------------- /boot/config/shares/system.cfg -----------------------------------------
01:23:58.077 Cache mode: only. Nothing will be moved. Its usage is taken into account in the calculation of the threshold.
01:23:58.078 Share Information: Name: system; Pool Name: cache; Path: /mnt/cache/system; Cache mode: only
01:23:58.095 Pool usage: 75 %, size: 723587424256 (673 GiB)
01:23:58.117 cache/system usage: 45286526976 bytes (42 GiB)
01:23:58.148 ------------------------------------------------------------------------------------------------------------------------------------
01:23:58.149 Decide of the action (move/keep) for each file...
01:24:46.024 A total of 381 files representing 246558311837 bytes (229 GiB) will be moved:
01:24:46.026 - 104 older files representing 65750043216 bytes (61 GiB) will be freed from cache and moved to array
01:24:46.028 - 277 new files representing 180808268621 bytes (168 GiB) will then be moved from array to cache

See for each share what "Cache mode" does.
See how smart are the 3 last lines

I've still some work to do on moving function (which send files to Unraid Mover) before releasing this new version

Notwist · July 22

Log shows an error when running mover: "/usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 397: ((: TOTALARRAYSIZE += : syntax error: operand expected (error token is "+= ")"

Any idea what this means and how to prevent it from occurring?

Reynald · July 22

7 hours ago, helpful-tune3401 said:

Log shows an error when running mover: "/usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 397: ((: TOTALARRAYSIZE += : syntax error: operand expected (error token is "+= ")"

Any idea what this means and how to prevent it from occurring?

This code is here since a while. It will be replaced in next version (coming really soon)

Can you check if /tmp/Mover/Custom_Cache_Tuning_(date).list is populated?

You can also try to replace this:

((TOTALARRAYSIZE += $CUSTOMFILESIZE))

by that:

TOTALARRAYSIZE += $CUSTOMFILESIZE

In this file

/usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 397

Edited July 22 by Reynald

Grobalt · July 22

i removed the old and installed this version from the community apps (not manually with a link from the forum).

Jul 22 20:38:27 unraid emhttpd: shcmd (329893): /usr/local/sbin/mover start |& logger -t move &
Jul 22 20:38:27 unraid kernel: mdcmd (37): set md_write_method 1
Jul 22 20:38:27 unraid kernel:
Jul 22 20:38:28 unraid move: Log Level: 1
Jul 22 20:38:28 unraid move: mover: started
Jul 22 20:38:29 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 260: /usr/local/sbin/move: No such file or directory
*repeated About 20 times*
Jul 22 20:38:30 unraid move: mover: finished

Reynald · July 22

36 minutes ago, Grobalt said:

i removed the old and installed this version from the community apps (not manually with a link from the forum).
[..]
Jul 22 20:38:29 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 260: /usr/local/sbin/move: No such file or directory
*repeated About 20 times*
Jul 22 20:38:30 unraid move: mover: finished

Community Apps version is not compatible with unraid 7

Reynald · July 22

19 hours ago, Reynald said:

Teasing: next version log example:

[...]
See for each share what "Cache mode" does.
See how smart are the 3 last lines

I've still some work to do on moving function (which send files to Unraid Mover) before releasing this new version

After the log, the UI with some new options

Grobalt · July 22

no changes with this one:

https://github.com/R3yn4ld/ca.mover.tuning/blob/794d8b0fad8ea1c2db8096dbd753ecf75c8a6e22/plugins/ca.mover.tuning.plg

First test yes - then with test no:

Quote

Jul 22 21:52:16 unraid emhttpd: shcmd (331937): /usr/local/sbin/mover start |& logger -t move &
Jul 22 21:52:16 unraid kernel: mdcmd (51): set md_write_method 1
Jul 22 21:52:16 unraid kernel:
Jul 22 21:52:16 unraid move: Log Level: 1
Jul 22 21:52:16 unraid move: mover: started
Jul 22 21:52:20 unraid move: mover: finished
Jul 22 21:52:20 unraid kernel: mdcmd (52): set md_write_method auto
Jul 22 21:52:20 unraid kernel:
Jul 22 21:52:41 unraid emhttpd: shcmd (331938): /usr/local/sbin/mover start |& logger -t move &
Jul 22 21:52:41 unraid kernel: mdcmd (53): set md_write_method 1
Jul 22 21:52:41 unraid kernel:
Jul 22 21:52:41 unraid move: Log Level: 1
Jul 22 21:52:41 unraid move: mover: started
Jul 22 21:52:42 unraid move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 260: /usr/local/sbin/move: No such file or directory
*repeated*
Jul 22 21:52:43 unraid move: mover: finished
Jul 22 21:52:43 unraid kernel: mdcmd (54): set md_write_method auto
Jul 22 21:52:43 unraid kernel:

Edited July 22 by Grobalt

Reynald · July 22

2 hours ago, Grobalt said:

no changes with this one:

https://github.com/R3yn4ld/ca.mover.tuning/blob/794d8b0fad8ea1c2db8096dbd753ecf75c8a6e22/plugins/ca.mover.tuning.plg

First test yes - then with test no:

Very strange because creating symlinks is part of the installation of the latest plugin.

Please try this code in console and report:

if [[ -e /usr/local/bin/move ]]; then
  echo "Creating Soft link for 6.12 RC8 or newer since /usr/local/sbin/move is now located in /usr/local/bin/move"
  if [[ -e /usr/local/sbin/move ]]; then rm /usr/local/sbin/move; fi
  ln -s /usr/local/bin/move /usr/local/sbin/move
elif [[ -e /usr/libexec/unraid/move ]]; then
  echo "Creating Soft link for 7.0 or newer since /usr/local/sbin/move is now located in /usr/libexec/unraid/move"
  if [[ -e /usr/local/sbin/move ]]; then rm /usr/local/sbin/move; fi
  ln -s /usr/libexec/unraid/move /usr/local/sbin/move
fi

Magic815 · July 23

So I believe I've tried mover tuning before, but had to remove it due to bugs/issues at the time with my setup. As a note, I'm still on UnRAID version 6.12.10. I intend to stick to 6.12.xx until a couple minor releases into version 7.

So these are my pool devices:

And these are my share settings:

Is the plugin (either the Squid version or this more recent fork) able to handle the use case of me setting a % threshold on the 'download_drive' hard drive for triggering the mover? My drive named 'cache' holds my appdata, domains, and systems files - which is pretty consistently 200GB / 1TB. It's the `Download_drive` that is constantly having new media added to it - and that is the one I'd want to have watched for when it gets close to being filled (i.e. when the 'Download_drive' gets over 70%, start moving files off of it to the array). But I'm not sure if the plugin can be set up for that use case?

Edited July 23 by Magic815

Reynald · July 23

4 hours ago, Magic815 said:

It's the `Download_drive` that is constantly having new media added to it - and that is the one I'd want to have watched for when it gets close to being filled (i.e. when the 'Download_drive' gets over 70%, start moving files off of it to the array). But I'm not sure if the plugin can be set up for that use case?

Actually 'per pool threshold' is not working, so with this plugin, you would set a 70% threshold and the files of each shares above 70% would be moved to array. That's not a problem as you cache drive is about 20%

Grobalt · July 24

On 7/23/2024 at 12:15 AM, Reynald said:

Very strange because creating symlinks is part of the installation of the latest plugin.

Please try this code in console and report:

if [[ -e /usr/local/bin/move ]]; then
  echo "Creating Soft link for 6.12 RC8 or newer since /usr/local/sbin/move is now located in /usr/local/bin/move"
  if [[ -e /usr/local/sbin/move ]]; then rm /usr/local/sbin/move; fi
  ln -s /usr/local/bin/move /usr/local/sbin/move
elif [[ -e /usr/libexec/unraid/move ]]; then
  echo "Creating Soft link for 7.0 or newer since /usr/local/sbin/move is now located in /usr/libexec/unraid/move"
  if [[ -e /usr/local/sbin/move ]]; then rm /usr/local/sbin/move; fi
  ln -s /usr/libexec/unraid/move /usr/local/sbin/move
fi

./test.sh
Creating Soft link for 7.0 or newer since /usr/local/sbin/move is now located in /usr/libexec/unraid/move

Reynald · July 24

Ok.

And is the age_mover running now?

Grobalt · July 24

fresh reinstalled and a run with test mode:

Jul 24 21:19:17 unraid plugin-manager: ca.mover.tuning.plg installed
Jul 24 21:20:03 unraid kernel: mdcmd (37): nocheck pause
Jul 24 21:20:03 unraid kernel: md: recovery thread: exit status: -4
Jul 24 21:20:06 unraid emhttpd: shcmd (1338): /usr/local/sbin/mover start |& logger -t move &
Jul 24 21:20:06 unraid move: Log Level: 1
Jul 24 21:20:06 unraid move: /tmp/Mover Directory does not exist. Creating it
Jul 24 21:20:06 unraid move: mover: started
Jul 24 21:20:10 unraid flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Jul 24 21:20:13 unraid move: mover: finished

on my cache disk are around 400GB to be moved .. dont know what the test mode should report - i expected filenames to be moved

i am not using "age" as criteria, i am using size above 14M to keep media library pictures on the flash drives

Edited July 24 by Grobalt

Gobblerpl · July 24

Is it possible to configure the mover to move files based on age and when the cache is over 60% full?

Currently, when I set it to move files based on age and Move All from Cache-yes shares pool percentage, it only works based on age.

Reynald · July 24

2 hours ago, Grobalt said:

fresh reinstalled and a run with test mode:

on my cache disk are around 400GB to be moved .. dont know what the test mode should report - i expected filenames to be moved

i am not using "age" as criteria, i am using size above 14M to keep media library pictures on the flash drives

The report are in the file in /tmp/Mover. You can check the list of what would be moved with test mode: no

to keep media library pictures on ssd drives, set the share(s) to "Cache=only". In latest unraid version, do not set a secondary storage for it.

Reynald · July 24

1 hour ago, Gobblerpl said:

Is it possible to configure the mover to move files based on age and when the cache is over 60% full?

Currently, when I set it to move files based on age and Move All from Cache-yes shares pool percentage, it only works based on age.

"Move All from Cache-yes" will move all based on the threhold for this option.
Do not select this option and just set "Only move at/to this threshold of used cache space:" and "Move files off cache based on age?"

Grobalt · July 24

3 minutes ago, Reynald said:

The report are in the file in /tmp/Mover. You can check the list of what would be moved with test mode: no

to keep media library pictures on ssd drives, set the share(s) to "Cache=only". In latest unraid version, do not set a secondary storage for it.

that will not work as the movie scraper store the pictures in the same folder as the movie - otherwise it would be super easy to create an flash only share. the mover worked and moved all files above 14M size to array.

Reynald · July 24

Ah OK, understood. Then the file size filter is effectivly your friend

Gobblerpl · July 28

On 7/24/2024 at 11:33 PM, Reynald said:

"Move All from Cache-yes" will move all based on the threhold for this option.
Do not select this option and just set "Only move at/to this threshold of used cache space:" and "Move files off cache based on age?"

Doesn't work.

Reynald · July 29

Updated:

2024-07-29

Automatic Array to Cache
Complete rewrite of file listing functions (find, decide to move..) (R3yn4ld)
Updated zfs functions getting usage of a pool (R3yn4ld, greetings to (Freender on forums) for pointing out and proposal)
Added cache mode "prefer" smart moving in "Automatic age" mode (R3yn4ld)
Added option to "repair" Cache:Only (moving everything on share to cache) and Cache:No (moving everything on share to array) shares (R3yn4ld)
Added option to synchronize Cache:Yes and Cache:Prefer shares to array so data are parity protected (R3yn4ld)
Turbo write mode forcing improvement to not wake spinners if not needed (R3yn4ld)
UI improvements (R3yn4ld)

Full changelog: R3yn4ld/ca.mover.tuning at automatic-array-to-cache (github.com)

It is a huge huge update. I've tested for a couple of weeks. Hopefully you won't find many bugs, please report in any case.

Installation: ca.mover.tuning.plg

Please remove Community Application one before installing this one (at the moment via Plugins/Install plugin/insert above URL).

Please note that you'll have to reconfigure mover tuning.

image.png.f42e625ee4e873c6acd0cdaa0ea1fe3c.png

Update: if already installed with above method, you should be able to update normally:

(above is the setting page of 2024-07-29 already )

Advise: enable Test Mode as on picture above and verify the Mover_action_{date}T{time}list in the /tmp/ca.mover.tuning directory. This file contain the files that would be moved with Test Mode off.

If everything seems OK you can turn off Test mode.

Edited July 29 by Reynald

Freender · July 29

1) After a fresh installation (2024.07.29), I am not able to see the Mover settings in Settings > Scheduler.

I believe the issue is with the freshly generated /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg. The content inside is written as a single line. After I broke down the lines, it started working.
2) I also suggest setting the “Move Now button follows plug-in filters” option to ‘Yes’ by default.

Edited July 29 by Freender
added plugin version and suggestion

Freender · July 29

3) In /boot/config/plugins/ca.mover.tuning/shareOverrideConfig/media.cfg

threshold is set:

shareThreshold="85"

However in https://github.com/R3yn4ld/ca.mover.tuning/blob/master/source/ca.mover.tuning/usr/local/emhttp/plugins/ca.mover.tuning/age_mover

threshold is misspelled (shareTreshold instead of shareThreshold)

    if [ -z $(cfg shareTreshold) ]; then
        SHAREPCTLIMIT="NA"
    else
        SHAREPCTLIMIT=$(cfg shareTreshold)
        mvlogger "Share threshold: $SHAREPCTLIMIT %"

As a result mover fails if a share has an override.

Jul 29 00:58:46 NAS move: ---------- Settings override for media: ----------
Jul 29 00:58:46 NAS move: Default threshold: 
Jul 29 00:58:46 NAS move: Age: Automatic (smart caching)
Jul 29 00:58:46 NAS move: No Turbo Mode argument supplied
Jul 29 00:58:46 NAS move: Skip file list path: /mnt/user/backup/mover-ignore/mover_ignore.txt
Jul 29 00:58:46 NAS move: Skip folder list path: /mnt/cache/media/downloads/*
Jul 29 00:58:46 NAS move: No Repair argument provided
Jul 29 00:58:46 NAS move: No Sparness argument supplied
Jul 29 00:58:46 NAS move: No Synchronize Cache argument provided
Jul 29 00:58:46 NAS move: No Test Mode argument supplied
Jul 29 00:58:46 NAS move: Cache mode: Yes/Cache->Array. Will smart move old files from Cache to Array. Nothing will be moved from Array to Cache
Jul 29 00:58:46 NAS move: Share Information: Name: media - Path: /mnt/cache/media - Cache mode: yes
Jul 29 00:58:46 NAS move: Pool Information: Name: cache - pool used: 84 % - size: 3362321969248 bytes (3 TiB)
Jul 29 00:58:46 NAS move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 396: 3868846981120 *  /100: syntax error: operand expected (error token is "/100")

I am going to sleep, and I will check if this helps tomorrow.

Reynald · July 29

6 hours ago, Freender said:

1) After a fresh installation (2024.07.29), I am not able to see the Mover settings in Settings > Scheduler.

I believe the issue is with the freshly generated /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg. The content inside is written as a single line. After I broke down the lines, it started working.
2) I also suggest setting the “Move Now button follows plug-in filters” option to ‘Yes’ by default.

1) corrected in 2024.07.29.b

2) good idea, implemented !

[Plugin] Mover Tuning

Recommended Posts

Link to comment

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

Squid

hugenbdd

hugenbdd

Posted Images

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation