[Plugin] Mover Tuning

[b0mb]

On 5/12/2020 at 3:16 PM, IronBeardKnight said:

A Small Guide

 

For those that want to use the new features but are a little lost or perhaps this will save you some time.
 

 

 

Open Unraid CMD.

    cd /boot/config/plugins/ca.mover.tuning

    sudo nano skiplist.txt

 

In the nano editor you can list your locations to skip like the following:

 

 

   Ctl + o 

   Ctl +x

 

Note: The list of locations may include or not include /mnt/cache/ as this is already catered for withing the mover script

 

Find Specific files of a name or kind you may find for example all .png in a location  and then put them in your exclusion list incase they are mixed in with other files and such example below.

  find "/mnt/cache" -type f \( -name "*.png"\) -print

Open CA Mover Tuning GUI

In CA Mover Tuning Plugin Set the location of your skiplist.txt as below

    File list path: /boot/config/plugins/ca.mover.tuning/skiplist.txt

 

So can i put whole folders in to that textfile for the skipping list?

[IronBeardKnight]

2 hours ago, b0mb said:

 

So can i put whole folders in to that textfile for the skipping list?

Yes

[sasbro97]

Hey guys. I guess somebody has probably answered already but the search function does not show me anything useful. I have the same error as this user: 

mv: cannot stat '/usr/local/bin/mover': No such file or directory

I see it in the boot logs. It still boots but why is this error appearing? How to fix it?

 

[CS01-HS]

I noticed folder exclusion using the File list path option stopped working.

 

Enabling test mode I see this in the find command (where my exclude file is exclude.txt)

grep -vFx -f <(sed 's/\/*$//' '/boot/extras/mover_tuner/exclude.txt') 

 

I don't see how this can work with the -x option:

-x, --line-regexp 

Select only those matches that exactly match 
the whole line. For a regular expression 
pattern, this is like parenthesizing the pattern 
and then surrounding it with ^ and $.

 

Even files in exclude.txt won't satisfy the exact match unless they're prefaced with the cache path, e.g. /mnt/cache/sharename/file.txt vs simply sharename/file.txt, which used to work.

 

Am I missing something?

Edited December 10, 2023 by CS01-HS

[hugenbdd]

20 hours ago, sasbro97 said:

Hey guys. I guess somebody has probably answered already but the search function does not show me anything useful. I have the same error as this user: 

mv: cannot stat '/usr/local/bin/mover': No such file or directory

I see it in the boot logs. It still boots but why is this error appearing? How to fix it?

 

Mover location was changed somewhere between 6.9 and 6.11.  So older versions still need an older path, hence the "missing".

 

Once you run mover it creates a soft link.  If you search back several pages I think I posted how to create the link.  Will still go missing on bootup though.  DM me if you can't find how to create it.

[hugenbdd]

24 minutes ago, CS01-HS said:

I noticed folder exclusion using the File list path option stopped working.

 

Enabling test mode I see this in the find command (where my exclude file is exclude.txt)

grep -vFx -f <(sed 's/\/*$//' '/boot/extras/mover_tuner/exclude.txt') 

 

I don't see how this can work with the -x option:

-x, --line-regexp 

Select only those matches that exactly match 
the whole line. For a regular expression 
pattern, this is like parenthesizing the pattern 
and then surrounding it with ^ and $.

 

Even files in exclude.txt won't satisfy the exact match unless they're prefaced with the cache path, e.g. /mnt/cache/sharename/file.txt vs simply sharename/file.txt, which used to work.

 

Am I missing something?

I believe it should have always been /mnt/cache and not the sharename.  At least that is how I coded and tested it.  If sharename/file.txt was working, I don't think it was intentional.

[CS01-HS]

Just now, hugenbdd said:

I believe it should have always been /mnt/cache and not the sharename.  At least that is how I coded and tested it.  If sharename/file.txt was working, I don't think it was intentional.

 

Okay let's set that aside.

Exact match (-x) means folder exclusion can't work because while the folder will match none of the files within it will.

[Swarles]

21 hours ago, CS01-HS said:

 

Okay let's set that aside.

Exact match (-x) means folder exclusion can't work because while the folder will match none of the files within it will.

What issue are you noticing?

Inside the "exclusion.txt" file you will need to make sure that each path for exclusion is going through the "/mnt/cache" pool name share and not the "/mnt/user" share. You also need to make sure that if you are using the override settings for a particular share, you should include the exclude list inside those override settings too because it won't use the global exclude list.

[CS01-HS]

1 hour ago, Swarles said:

What issue are you noticing?

 

File-exclusion works as expected but folder-exclusion only excludes the folder itself, nothing under it.

 

e.g. given the folder structure:

MyShare/KeepOnCache/
		File1.txt
		File2.txt

 

And this entry in the exclude file:

/mnt/cache/MyShare/KeepOnCache

 

the find command (with sed evaluated) look like this:

> find "/mnt/cache/MyShare" -depth | grep -vFx "/mnt/cache/MyShare/KeepOnCache"

/mnt/cache/MyShare/KeepOnCache/File1.txt
/mnt/cache/MyShare/KeepOnCache/File2.txt

So File1.txt and File2.txt are not excluded from mover.

 

Is that intended and if so what's the point of folder-exclusion?

[Swarles]

5 hours ago, CS01-HS said:

 

File-exclusion works as expected but folder-exclusion only excludes the folder itself, nothing under it.

 

e.g. given the folder structure:

MyShare/KeepOnCache/
		File1.txt
		File2.txt

 

And this entry in the exclude file:

/mnt/cache/MyShare/KeepOnCache

 

the find command (with sed evaluated) look like this:

> find "/mnt/cache/MyShare" -depth | grep -vFx "/mnt/cache/MyShare/KeepOnCache"

/mnt/cache/MyShare/KeepOnCache/File1.txt
/mnt/cache/MyShare/KeepOnCache/File2.txt

So File1.txt and File2.txt are not excluded from mover.

 

Is that intended and if so what's the point of folder-exclusion?

Have you tried testing it or are you just going off the code?

 

That part of the code is only meant to exclude the top level folder path or individual files if they have been specified in the exclude list, it doesn't exclude anything underneath a folder. The find string will then use a -not -path "/mnt/cache/MyShare/KeepOnCache/*" to exclude the files underneath the folder.

[CS01-HS]

1 hour ago, Swarles said:

The find string will then use a -not -path "/mnt/cache/MyShare/KeepOnCache/*" to exclude the files underneath the folder.

 

Oh, I didn't realize.

I was going off the debug output in test mode.

 

I tried a real run (now with full paths) and it works. 

Thanks!

 

[Scheev]

Is there a way to use this tuner to:

 

1. Move only when the cache fills to a certain amount

2. Move only the oldest files

 

any help is appreciated!

[kizer]

11 hours ago, Scheev said:

Is there a way to use this tuner to:

 

1. Move only when the cache fills to a certain amount

2. Move only the oldest files

 

any help is appreciated!

 

That's exactly what the script does. I have mine set to only move items if the drive is 5% used. Meaning it will try and run every day. Then I have it set to only move files that are 15days or older. 

 

So with my settings it'll try and run every day because I have more than 5% on my drive, but hold all files until they are at least 15 days. Files that are 0-14days old will still be on the drive. Obviously I could set the thresh hold to 15% or more, but I'm more concerned with the how many days vs how full my drive is. 

 

I want files to sit on my drive for quick access and so when I watch something in plex I don't have to wait for my drives to spool up because they are sitting on my SSD. Anything older than 15days will spool up the drives, but I'm taking less of a risk that my files could be lost if my SSD dies. However, I do have a backup unraid machine that I backup all my media to and it just happens to look at my SSD to so even if I lost my SSD I'd have it all backed up anyways. 

[martinkeat]

Steps Tried: 1-6

System Log: Put in a spoiler. 
Mover Tuning Log: 

 

Quote

mvlogger: *********************************MOVER START*******************************
mvlogger: Age supplied: 0
mvlogger: Size supplied: 0
mvlogger: Sparness supplied: 0
mvlogger: No Skipfiles Argument Supplied
mvlogger: No Skip File Types Argument Supplied
mvlogger: No Before Script Argument Supplied
mvlogger: No After Script Argument Supplied
mvlogger: No CTIME Argument Supplied
mvlogger: CACHE THRESH to Move all Cache-Yes shares to array: 25
mvlogger: No Test Mode Argument Supplied
mvlogger: No Ignore Hidden Files Argument Supplied
mvlogger: Threshold Percent: 20
mvlogger: No Script to Run.
mvlogger: CACHETHRESH: 25
mvlogger: Checking to see if we should move all files from share.
mvlogger: Move All Cache Threshold: 25
mvlogger: The fstype of /mnt/cache is zfs.
mvlogger: /mnt/cache PCT USED: 36
mvlogger: Pool /mnt/cache is above 'move all' percentage, moving all cache-yes shares to array
mvlogger: Done checking Move All from Share
mvlogger: Share Name Only: Ripped
mvlogger: Cache Pool Name: cache 
mvlogger: cache Threshold Pct: 
mvlogger: OVERALL Threshold: 20
mvlogger: Share Path: /mnt/cache/Ripped
mvlogger: cache is zfs.
mvlogger: Pool Pct Used: 36 %
mvlogger: DFTPCT LIMIT USED FOR SETTING: 20
mvlogger: Threshold Used: 20
mvlogger: Share Name Only: system
mvlogger: Cache Pool Name: cache 
mvlogger: cache Threshold Pct: 
mvlogger: OVERALL Threshold: 20
mvlogger: Share Path: /mnt/cache/system
mvlogger: cache is zfs.
mvlogger: Pool Pct Used: 36 %
mvlogger: DFTPCT LIMIT USED FOR SETTING: 20
mvlogger: Threshold Used: 20
mvlogger: after customFilelist
mvlogger: After totalsizeFilelist
mvlogger: -----------------------------------------Running Array Files to Cache drives----------------------------------------------
mvlogger: Running to cache drive
mvlogger: File to Cache filelist created
mvlogger: Cache File List: /tmp/Mover/Cache_Tuning_2023-12-12T234731.list
mvlogger: make customArray File List
mvlogger: write first status file after to array
mvlogger: Hard Link Status: false
mvlogger: No /tmp/Mover/Custom_Mover_Directory_2023-12-12T234731.list created as no empty directories
mvlogger: No After Script to Run.
 

Extra Details: eg. Only stopped working after latest update.

[Scheev]

8 hours ago, kizer said:

 

That's exactly what the script does. I have mine set to only move items if the drive is 5% used. Meaning it will try and run every day. Then I have it set to only move files that are 15days or older. 

 

So with my settings it'll try and run every day because I have more than 5% on my drive, but hold all files until they are at least 15 days. Files that are 0-14days old will still be on the drive. Obviously I could set the thresh hold to 15% or more, but I'm more concerned with the how many days vs how full my drive is. 

 

I want files to sit on my drive for quick access and so when I watch something in plex I don't have to wait for my drives to spool up because they are sitting on my SSD. Anything older than 15days will spool up the drives, but I'm taking less of a risk that my files could be lost if my SSD dies. However, I do have a backup unraid machine that I backup all my media to and it just happens to look at my SSD to so even if I lost my SSD I'd have it all backed up anyways. 

Okay cool, I tried looking at the settings page, but I am not seeing an obvious way to have both of those conditions. Do you mind sharing how exactly you achieved that?

[kizer]

14 hours ago, Scheev said:

Okay cool, I tried looking at the settings page, but I am not seeing an obvious way to have both of those conditions. Do you mind sharing how exactly you achieved that?

 

Sure its a bit old screen shot, but here are my settings that I still use today. Keep in mind there might be an option or two not shown on this one, but I think its pretty self explanatory when you read the new ones. 

Just to clarify a thing or two. Every day at 5:45AM my system fires off the Mover script based off the unraid default settings. It looks to make sure that at least 5% is used and only moves files that are 20days old. The reason I did this was simple. I didn't want the script firing off every minute because in the past I noticed that when the script ran it was spinning up all my drives. No idea if that has been addressed, but doing it my way it doesn't matter because I have other things running at 6AM and these runs prior to. 

 

So after it fires of the script and verifies that at least 5% is used because there needs to be some kind of threshold set because its my understanding that it uses the threshold as the basis for triggering other things. If that was changed I'm covered either way. 😃

 

So now that I've set it to run everyday at 5:45 and at least 5% is used it looks for files that are Older than 20 days and moves them. 

 

[hugenbdd]

New Release thanks to @Swarles

 

###2023.12.19
- Change "while read" lines in age_mover to "while IFS= read -r" to fix trailing white spaces (Swarles)
- Fix where sometimes mover would not run to mover.old scrip (Swarles)
- Log if "share.cfg" doesn't exists to help trouble shooting (Swarles)
- Check for ca.mover.tuning.cfg file and additional logging. (Swarles)

[cinema-plug]

Howdy,

 

I've installed mover tuning and configured it with the following information:

Disable Mover running on a schedule: No

Only move at this threshold of used cache space: 10%
Script to run before mover (No checks, always runs): /mnt/user/appdata/mover-tuning/stop_qbit.sh

Script to run after mover (No checks, always runs): /mnt/user/appdata/mover-tuning/start_qbit.sh
Move Now button follows plug-in filters: Yes

The share currently has 197GB used of 1TB (19.7%, which is > 10%)

Where stop_qbit.sh contains:
#!/bin/bash
docker stop qbittorrent-vpn

Where start_qbit.sh contains:
#!/bin/bash
sleep 3
docker start qbittorrent-vpn
sleep 90
docker restart radarr
sleep 3
docker restart sonarr

And finally, a screenshot of my docker containers:


For some reason, whenever I manually initiate the mover it never stops qbittorrent-vpn. Am I misunderstanding a setting or where did I go wrong? Checking the system log I see the following output which indicates it's initiating the move with the appropriate settings, but still qbit doesn't shutdown.
 

Dec 20 13:10:34 Cinema-Plug root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start 0 0 0 "/mnt/user/appdata/mover-tuning/ignorelist.txt" ".part,.!qB" "/mnt/user/appdata/mover-tuning/stop_qbit.sh" "/mnt/user/appdata/mover-tuning/start_qbit.sh" '' '' '' '' 10

 

[trurl]

Why do you want to stop any containers to run mover? Your appdata should be all on fast pool such as cache and configured to stay there untouched by mover.

[cinema-plug]

6 minutes ago, trurl said:

Why do you want to stop any containers to run mover? Your appdata should be all on fast pool such as cache and configured to stay there untouched by mover.

 

I was attempting to follow the information I found here:

https://www.reddit.com/r/unRAID/comments/10a1u6h/comment/j4200rb/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

 

The idea is that I've ignored the active downloads folder /mnt/user/data/torrents/, however, if qbittorrent remains active and continues downloading (and finishing downloads) then I'll have an edge case where mover starts, qbittorrent finishes a download and moves it from /mnt/user/data/torrents/movies to /mnt/user/data/media/movies (which is actively being moved).

I'd prefer for qbit to be stopped, mover to be initiated, then once mover is finished qbit is started, and apps that interact with qbit to restart (ensuring they maintain a proper connection to qbit).

Edited December 20, 2023 by cinema-plug

[trurl]

I don't use that container, but I do use transmission, downloads to a cached share, don't even bother with mover tuning plugin, and have never had an issue with my torrents.

[cinema-plug]

21 minutes ago, trurl said:

I don't use that container, but I do use transmission, downloads to a cached share, don't even bother with mover tuning plugin, and have never had an issue with my torrents.

I’ll have to look into that but I’d still much rather find out why mover tuner doesn’t seem to be running my start/stop scripts. I’ve verified that the scripts work when called manually in the console.

[cinema-plug]

2 hours ago, cinema-plug said:

I’ll have to look into that but I’d still much rather find out why mover tuner doesn’t seem to be running my start/stop scripts. I’ve verified that the scripts work when called manually in the console.

Fixed it. It was a permissions issue.

[DevanteWeary]

Guys still trying to get a handle on this awesome plugin which seems to have a lot of work put into it so thank you!

 

So am I understanding this correctly?  

This plugin (and Mover itself?) only moves OFF of the cache drive?

 

Or does it follow the Array > Cache preference too?

 

Basically I have some shares set as Array > Cache and at the end of the day, if the space becomes available on the cache, I want the files moved FROM the array (HDD ) to the cache (SSD).

 

Basically I want the drives to be used as overflow but those files to eventually end up on the cache automatically.

 

Is this possible?

[itimpi]

30 minutes ago, DevanteWeary said:

Or does it follow the Array > Cache preference too?

This.