[Plugin] Mover Tuning

[Brandon_K]

2 hours ago, d3m3zs said:

Can someone tell me how to push plugin move data?

Turned off plugin, pressed MOVE, saw grey disable button

 

If you have "Move now button follow plugin filters" set to "yes", by setting "Disable mover running on a schedule" to "yes" also, has effectively disabled the entire mover system.

Set "move now follows plugin filters" to *NO* and "Disable Mover running on a schedule" to *NO*.

[Tufinger]

On 12/13/2022 at 6:10 AM, Brandon_K said:

I'm still having some issues that I've yet to be able to solve.

What I want is pretty simple; when the cache is >70% full, move everything (all "cache: yes" shares) to the array.  I've been playing with options and I can get it to move, but it only moves data that is above 70% and leaves everything else on cache. I'm mostly just trying to avoid spinning up disks every day if I don't have to.  It usually takes 3-5 days to get my cache disk to ~70% utilization.

My settings are below;

 

No
 *Note that setting this to yes effectively disables this plugin

Priority for mover process: Normal
Priority for disk I/O: Normal
Only move at this threshold of used cache space: 70 %
Move files off cache based on age? No
Move files that are greater than this many days old: 1
Use CTIME: No
Move files based on minimum size? No
Move files that are larger than this size (In M). 1
Move files off cache based on sparsenes? No
Move files that are greather than this sparseness: .1
Ignore files listed inside of a text file: No
File list path:
Ignore file types: No
comma seperated list of file types:
Let scheduled mover run during a parity check / rebuild: No
Force turbo write on during mover: No
Log when not moving due to rules: Yes
Force move of all files on a schedule: No
Cron Schedule to force move all of files:
Allow force mover schedule to run during a parity check/rebuild: No
Ignore All hiden files and directories: No
Script to run before mover (No checks, always runs):
Script to run after mover (No checks, always runs):
Move All from Cache-Yes shares when disk is above a certain percentage: Yes
Move All from Cache-yes shares pool percentage: 75 %
Move Now button follows plug-in filters: Yes
Test Mode: No

 

What am I missing or doing wrong?

same problem here

[hugenbdd]

4 hours ago, Tufinger said:

same problem here

Would need to see some of the mover log entries.  Mostly the individual find commands for each share.

[chocorem]

to be able to debug, is it possible to log the rule result ? so we can see zhen the script is started, what is the condition that triggers or do not trigger the mover start ?

[hugenbdd]

7 hours ago, chocorem said:

to be able to debug, is it possible to log the rule result ? so we can see zhen the script is started, what is the condition that triggers or do not trigger the mover start ?

Yes, each rule is logged when it's triggered.  If you have logging enabled, and run in test mode, you can then review the logs and change your rules accordingly.  Also, if you set the "Move Now button follows Rules" then you can make lots of edits very quickly by hitting the move now button.

[Anon]

Hi, just a headsup that caused me some headache if anyone else comes across it:

When working with the ignore files list I accidentally had a empty line at the bottom that I didn't notice. This caused no files to be moved.

 

[-Daedalus]

Has anyone much experience with the custom scripts before/after mover?

I've got one I'm trying to run that stops/starts my download client before/after.

Rather than two separate scripts, I'm passing the same one with a viable, so In the two fields, I have:

 

 

 

Looking at the log file, the variable is empty, so it's not being passed through.

I can make a second one, but maybe a feature request would be to alter that field to allow more than just a path?

Edited January 6, 2023 by -Daedalus

[stefan.tomko]

I installed this plugin and basically made only few amendments - I want to move things off of cache when cache is more than 50% utilized.

 

My cache is 19% utilized so I would expect that nothing gets moved. However from logs, I observe following:

 

Quote

Jan 31 03:40:01 Tower root: mvlogger: *********************************MOVER START*******************************
Jan 31 03:40:01 Tower root: mvlogger: Age supplied 0
Jan 31 03:40:01 Tower root: mvlogger: Size supplied
Jan 31 03:40:01 Tower root: mvlogger: Sparness supplied
Jan 31 03:40:01 Tower root: mvlogger: No Skipfiles Argument Supplied
Jan 31 03:40:01 Tower root: mvlogger: No Skipfiles Argument Supplied
Jan 31 03:40:01 Tower root: mvlogger: No Before Script Argument Supplied
Jan 31 03:40:01 Tower root: mvlogger: No After Script Argument Supplied
Jan 31 03:40:01 Tower root: mvlogger: CTIME Argument: no
Jan 31 03:40:01 Tower root: mvlogger: CACHE THRESH to Move all Cache-Yes shares to array: 5
Jan 31 03:40:01 Tower root: mvlogger: No Test Mode Argument Supplied
Jan 31 03:40:01 Tower root: mvlogger: No Ignore Hidden Files Argument Supplied
Jan 31 03:40:01 Tower root: mvlogger: No Script to Run.
Jan 31 03:40:01 Tower root: mvlogger: CACHETHRESH: 5
Jan 31 03:40:01 Tower root: mvlogger: Checking to see if we should all files from share.
Jan 31 03:40:01 Tower root: mvlogger: Move All Cache Threshold: 5
Jan 31 03:40:01 Tower root: mvlogger: /mnt/cache PCT USED:  19
Jan 31 03:40:01 Tower root: mvlogger: Pool /mnt/cache is above 'move all' percentage, moving all cache-yes shares to array
Jan 31 03:40:01 Tower root: mvlogger: Not in Test mode
Jan 31 03:40:01 Tower root: mvlogger: Complete Mover Command: find "/mnt/cache/backups/" -depth | /usr/local/sbin/move -d 1
Jan 31 03:40:02 Tower root: mvlogger: Not in Test mode
Jan 31 03:40:02 Tower root: mvlogger: Complete Mover Command: find "/mnt/cache/nextcloud/" -depth | /usr/local/sbin/move -d 1
Jan 31 03:40:02 Tower root: mvlogger: Done checking Move All from Share
Jan 31 03:40:02 Tower root: mvlogger: Share Name Only: backups
Jan 31 03:40:02 Tower root: mvlogger: Cache Pool Name: cache
Jan 31 03:40:02 Tower root: mvlogger: cache Threshold Pct:
Jan 31 03:40:02 Tower root: mvlogger: OVERALL Threshold: 50
Jan 31 03:40:02 Tower root: mvlogger: Share Path: /mnt/cache/backups
Jan 31 03:40:02 Tower root: mvlogger: Pool Pct Used:  19 %
Jan 31 03:40:02 Tower root: mvlogger: DFTPCT LIMIT USED FOR SETTING: 50
Jan 31 03:40:02 Tower root: mvlogger: Threshold Used: 50
Jan 31 03:40:02 Tower root: mvlogger: Mover not Needed.

Why are first logs reporting that threshold is 5%? And obviously moving files...while later for other shares, it is correctly displaying 50% threshold and not taking any action?

 

Only move at this threshold of used cache space: is set to 50%

Move All from Cache-yes shares pool percentage: is set to 55%

 

Anyone has any idea?

 

[hugenbdd]

2 hours ago, stefan.tomko said:

I installed this plugin and basically made only few amendments - I want to move things off of cache when cache is more than 50% utilized.

 

My cache is 19% utilized so I would expect that nothing gets moved. However from logs, I observe following:

 

Why are first logs reporting that threshold is 5%? And obviously moving files...while later for other shares, it is correctly displaying 50% threshold and not taking any action?

 

Only move at this threshold of used cache space: is set to 50%

Move All from Cache-yes shares pool percentage: is set to 55%

 

Anyone has any idea?

 

 

"Move All Cache Threshold: 5"

 

Might be a bug somewhere so that the right value did not get saved in the config file.  Try changing from 55% to 60% and hitting apply.

[stefan.tomko]

I have changes to 55 and 60%, seems like it worked

[hugenbdd]

Looking for a few testers with the following.

 

1.) Basic understanding of bash/shell (To install the files and change their permissions. chown, chmod etc...)

2.) That have mover tuning plug-in installed. (Preferrable have been using it for a while) (Required as the GUI update file is created based off this plug-in).

2b.) Have unRAID 6.11.3 or above installed.

3.) That can test moving files off cache shares to the array.

4.) That can test moving files TO cache pool FROM the array.

5.) Bonus - if you have multiple pools to test with.

 

[WexfordStyle]

Am I understanding the help notes correctly that if I have pre-post scripts that they will be run at every check regardless of whether the conditions are met to run the mover? Ie. If I have the schedule set to run every hour then my torrent docker would stop and start every hour (if that was my script) regardless of weather the mover runs?

 

[BleachingLlama]

Hello,

 

Thank you for providing this script! I have a question about the `Script to run before mover` and `Script to after before mover` feature.

 

I would like to stop my docker container called `qbittorrentvpn` when mover starts, and start it again after.

 

So I have the following path set and the script looks like so

Script to run before mover:

`/boot/config/plugins/user.scripts/scripts/Stop-Qbittorrent/script`

```

root@Tower:~# cat /boot/config/plugins/user.scripts/scripts/Stop-Qbittorrent/script
#!/bin/bash
docker stop -t 60 qbittorrentvpn

```

 

Script to run after mover:

`/boot/config/plugins/user.scripts/scripts/Start-Qbittorrent/script`

```

root@Tower:~# cat /boot/config/plugins/user.scripts/scripts/Start-Qbittorrent/script
#!/bin/bash
docker start qbittorrentvpn

```

 

At the bottom of the settings there is a setting for "Move Now button follows plug-in filters." Its defaulted to no, and I changed this to Yes.

 

But if I hit "Run Mover" in the UI my docker container is not killed. Are the rules only applied if the Mover is run on the schedule? 

 

Thank you!

 

Edited February 26, 2023 by BleachingLlama
I found another setting but same problem

[WexfordStyle]

18 hours ago, BleachingLlama said:

Hello,

 

Thank you for providing this script! I have a question about the `Script to run before mover` and `Script to after before mover` feature.

 

I would like to stop my docker container called `qbittorrentvpn` when mover starts, and start it again after.

 

So I have the following path set and the script looks like so

Script to run before mover:

`/boot/config/plugins/user.scripts/scripts/Stop-Qbittorrent/script`

```

root@Tower:~# cat /boot/config/plugins/user.scripts/scripts/Stop-Qbittorrent/script
#!/bin/bash
docker stop -t 60 qbittorrentvpn

```

 

Script to run after mover:

`/boot/config/plugins/user.scripts/scripts/Start-Qbittorrent/script`

```

root@Tower:~# cat /boot/config/plugins/user.scripts/scripts/Start-Qbittorrent/script
#!/bin/bash
docker start qbittorrentvpn

```

 

At the bottom of the settings there is a setting for "Move Now button follows plug-in filters." Its defaulted to no, and I changed this to Yes.

 

But if I hit "Run Mover" in the UI my docker container is not killed. Are the rules only applied if the Mover is run on the schedule? 

 

Thank you!

 

EDIT:

 

So I am working through something similar and I believe the issue you will find in the logs if you turn them on is that the script cannot be run because of permissions issues. My understanding is that recent updates to Unraid prevent scripts from being run on the flash drive without specifying a run command which as far as I can tell is not possible to do within the mover tuning plugin. See below.

 

 

Therefore all I did to get around this was move the script from the flash drive to my cache drive and made in executable. Update the path in the mover tuning plugin and now it works well.

 

ORIGINAL POST:

 

Try turning on logging and see what’s in the syslog. This should let you know if and why the scripts are not being run.

 

If you are using qbittorrent with any kind of automation, I would highly recommend you check out trash guides.

 

https://trash-guides.info/Downloaders/qBittorrent/Tips/How-to-run-the-unRaid-mover-for-qBittorrent/#the-script

Edited February 26, 2023 by WexfordStyle
Plausible root cause + solution added

[BleachingLlama]

8 hours ago, WexfordStyle said:

EDIT:

 

So I am working through something similar and I believe the issue you will find in the logs if you turn them on is that the script cannot be run because of permissions issues. My understanding is that recent updates to Unraid prevent scripts from being run on the flash drive without specifying a run command which as far as I can tell is not possible to do within the mover tuning plugin. See below.

 

 

Therefore all I did to get around this was move the script from the flash drive to my cache drive and made in executable. Update the path in the mover tuning plugin and now it works well.

 

ORIGINAL POST:

 

Try turning on logging and see what’s in the syslog. This should let you know if and why the scripts are not being run.

 

If you are using qbittorrent with any kind of automation, I would highly recommend you check out trash guides.

 

https://trash-guides.info/Downloaders/qBittorrent/Tips/How-to-run-the-unRaid-mover-for-qBittorrent/#the-script

I appreciate the detailed reply. Do you mind sharing which path on the Cache drive you used?

 

Do I need to turn on something to see better mover logs? When I try to press the "Move" button with the scripts enabled, the mover does not actually start, I assume its some sort of script issue

 

I set the following paths:

/mnt/cache/system/scripts/stop-downloaders-for-mover.sh

/mnt/cache/system/scripts/start-downloaders-for-mover.sh

 

I believe I have permissions set up properly, here is following the paths and looking at chmod:

root@Tower:/mnt# cd cache/
root@Tower:/mnt/cache# ls -l
total 0
drwxrwxrwx 1 nobody users 402 Feb 26 11:38 appdata/
drwxrwxrwx 1 nobody users  26 Feb 25 17:45 data/
drwxrwxrwx 1 nobody users  12 Feb 24 02:41 domains/
drwxrwxrwx 1 nobody users  40 Feb 26 11:50 system/
root@Tower:/mnt/cache# cd system/
root@Tower:/mnt/cache/system# ls -l
total 0
drwxrwxrwx 1 nobody users  20 Feb 23 19:27 docker/
drwxrwxrwx 1 nobody users  22 Feb 23 19:27 libvirt/
drwxrwxrwx 1 root   root  118 Feb 26 13:43 scripts/
root@Tower:/mnt/cache/system# cd scripts/
root@Tower:/mnt/cache/system/scripts# ls -l
total 8
-rwxrwxrwx 1 root root 508 Feb 26 13:43 start-downloaders-for-mover.sh*
-rwxrwxrwx 1 root root 508 Feb 26 13:43 stop-downloaders-for-mover.sh*

 

 

 

 

I'll share my scripts I spent some time on them and think they are quite good (maybe jq can be optimized).

 

Kills any container connected to my VPN, then the VPN:

#!/bin/bash

# Docker ID for qbittorrentvpn
VPN_CONTAINER_ID=$(docker inspect qbittorrentvpn | jq -r '.[].Id')
# Looks for all containers using the network by ID, uses JQ to extract those container IDs and passes them to docker stop
docker container ls -aq | xargs docker container inspect --format='{{json .}}' | jq -r "select(.HostConfig.NetworkMode == \"container:$VPN_CONTAINER_ID\")" | jq -s '.' | jq '. | map(.Id)' | jq -r '.[]' | xargs docker stop
# Now stop qbittorrentvpn
docker stop $VPN_CONTAINER_ID

Starts the VPN, then containers connected to VPN

#!/bin/bash

# Docker ID for qbittorrentvpn
VPN_CONTAINER_ID=$(docker inspect qbittorrentvpn | jq -r '.[].Id')
# Start qbittorrentvpn
docker start $VPN_CONTAINER_ID
# Looks for all containers using the network by ID, uses JQ to extract those container IDs and passes them to docker start
docker container ls -aq | xargs docker container inspect --format='{{json .}}' | jq -r "select(.HostConfig.NetworkMode == \"container:$VPN_CONTAINER_ID\")" | jq -s '.' | jq '. | map(.Id)' | jq -r '.[]' | xargs docker start

 

Edited February 26, 2023 by BleachingLlama

[WexfordStyle]

On 2/26/2023 at 4:52 PM, BleachingLlama said:

I appreciate the detailed reply. Do you mind sharing which path on the Cache drive you used?

 

Do I need to turn on something to see better mover logs? When I try to press the "Move" button with the scripts enabled, the mover does not actually start, I assume its some sort of script issue

 

I set the following paths:

/mnt/cache/system/scripts/stop-downloaders-for-mover.sh

/mnt/cache/system/scripts/start-downloaders-for-mover.sh

 

I believe I have permissions set up properly, here is following the paths and looking at chmod:

root@Tower:/mnt# cd cache/
root@Tower:/mnt/cache# ls -l
total 0
drwxrwxrwx 1 nobody users 402 Feb 26 11:38 appdata/
drwxrwxrwx 1 nobody users  26 Feb 25 17:45 data/
drwxrwxrwx 1 nobody users  12 Feb 24 02:41 domains/
drwxrwxrwx 1 nobody users  40 Feb 26 11:50 system/
root@Tower:/mnt/cache# cd system/
root@Tower:/mnt/cache/system# ls -l
total 0
drwxrwxrwx 1 nobody users  20 Feb 23 19:27 docker/
drwxrwxrwx 1 nobody users  22 Feb 23 19:27 libvirt/
drwxrwxrwx 1 root   root  118 Feb 26 13:43 scripts/
root@Tower:/mnt/cache/system# cd scripts/
root@Tower:/mnt/cache/system/scripts# ls -l
total 8
-rwxrwxrwx 1 root root 508 Feb 26 13:43 start-downloaders-for-mover.sh*
-rwxrwxrwx 1 root root 508 Feb 26 13:43 stop-downloaders-for-mover.sh*

 

 

 

 

I'll share my scripts I spent some time on them and think they are quite good (maybe jq can be optimized).

 

Kills any container connected to my VPN, then the VPN:

#!/bin/bash

# Docker ID for qbittorrentvpn
VPN_CONTAINER_ID=$(docker inspect qbittorrentvpn | jq -r '.[].Id')
# Looks for all containers using the network by ID, uses JQ to extract those container IDs and passes them to docker stop
docker container ls -aq | xargs docker container inspect --format='{{json .}}' | jq -r "select(.HostConfig.NetworkMode == \"container:$VPN_CONTAINER_ID\")" | jq -s '.' | jq '. | map(.Id)' | jq -r '.[]' | xargs docker stop
# Now stop qbittorrentvpn
docker stop $VPN_CONTAINER_ID

Starts the VPN, then containers connected to VPN

#!/bin/bash

# Docker ID for qbittorrentvpn
VPN_CONTAINER_ID=$(docker inspect qbittorrentvpn | jq -r '.[].Id')
# Start qbittorrentvpn
docker start $VPN_CONTAINER_ID
# Looks for all containers using the network by ID, uses JQ to extract those container IDs and passes them to docker start
docker container ls -aq | xargs docker container inspect --format='{{json .}}' | jq -r "select(.HostConfig.NetworkMode == \"container:$VPN_CONTAINER_ID\")" | jq -s '.' | jq '. | map(.Id)' | jq -r '.[]' | xargs docker start

 

 

The reason I said that was because in your initial post, you showed paths on the flash drive (/boot).

 

Now that you moved them and made them executable I think you should be good.

 

You need to enable mover logging on the main ‘Mover Settings’ tab and then you will see everything it does in the syslog.

 

As a side note you really should check out the Trash Guides link that I placed above. His script is well vetted and supported by a large community and much more advanced. It uses the qBittorrent api to pause your torrents, run the mover, then resume your torrents. No shutting down or shopping any services and no before after scripting in the somewhat buggy mover tuning plugin.

 

 

 

[threefoursixeight]

I'm thinking of upgrading my system and adding an additional SSD to a new pool to store Sabnzbd downloads and Plex Media and curious if my thoughts below are correct:

 

1. I'd like my shares like nextcloud, etc. to behave like they do now - files transferred into unraid stay on cache until the nightly move where they will always move to the array.
   1.  If I leave this share as "Yes: Cache" and in the plugin set "Move all from Cache-Yes shares when disc is above a certain percentage" to "Yes" and set the % to 0, will that achieve this?
2. I'd like my media share that contains downloads and Plex media to stay on cache until X% full.
   1. Would I need to set this share to "Prefer: Cache", and then play with "Only move at this threshold of used cache space" and/or the moved based on age setting?

Basically, I'd like some shares to move nightly no matter what, and then other shares stay on cache until a certain age/used space is reached.

 

Thanks!

[SP67]

Hello, 

 

Thanks for the amazing plug-in. It adds some really useful stuff!

 

Is there any way to exclude a specific folder from the mover? Currently I see the option to ignore files based on size, text file, file type, but I don't know if there is any way to exclude a folder.

 

Thanks!

 

EDIT: Nevermind, found a solution on page 36.

Edited March 10, 2023 by SP67

[Mamaun]

Hi,

 

I'm planning my unraid server and I wonder if I can do the following:

- Start mover only when my cache is full at 70%

- Move older files first 

- Only move files until my cache is 70% again and keep the rest.

 

So if my caché is at 80% the mover only moves the older files until my cache is at 70% again.

 

Thanks and sorry about my english

Edited March 10, 2023 by Mamaun

[SP67]

Not directly. You can play with the age of the file and the % to get something similar, but not exactly what you want. Read the last pages as it has been discussed. 

[danimal86]

Looking for some help on why my mover tuning settings aren't working.  I've got 4tb cache and my idea is that it will keep new media on the cache drive for hopefully as long as possible so that no disks need to be spun up if you are watching something thats current (at least within the last few months).  

I looked at one of my directories on the cache and there's a file in there thats 68 days old.  My cache is getting full so i've been dropping the "Move files that are greater than this many days old" down and clicking save thinking that it would drop the usage down but it doesn't.  Is there a setting im missing?

[Kilrah]

Do you actually have the mover scheduled to run? The Tuning plugin only adds filters to a normal run, it still needs to actually be run.

Edited March 13, 2023 by Kilrah

[danimal86]

3 hours ago, Kilrah said:

Do you actually have the mover scheduled to run? The Tuning plugin only adds filters to a normal run, it still needs to actually be run.

I do have it turned on.  Should i set it to something like "Daily" so that it will check for old media frequently?

[Kilrah]

Definitely, your current settings mean it will never run and check for your conditions more than once a month.

 

The schedule is the "master", i.e. whenever the schedule runs the mover it will check and do things based on the tuning plugin settings, the plugin is not a trigger.

Edited March 13, 2023 by Kilrah

[danimal86]

21 minutes ago, Kilrah said:

Definitely, your current settings mean it will never run and check for your conditions more than once a month.

 

The schedule is the "master", i.e. whenever the schedule runs the mover it will check and do things based on the tuning plugin settings, the plugin is not a trigger.

I changed the master schedule to daily in a few min....will test to see if it works.