[Plugin] Mover Tuning

[b0mb]

On 5/12/2020 at 3:16 PM, IronBeardKnight said:

A Small Guide

 

For those that want to use the new features but are a little lost or perhaps this will save you some time.
 

 

 

Open Unraid CMD.

    cd /boot/config/plugins/ca.mover.tuning

    sudo nano skiplist.txt

 

In the nano editor you can list your locations to skip like the following:

 

 

   Ctl + o 

   Ctl +x

 

Note: The list of locations may include or not include /mnt/cache/ as this is already catered for withing the mover script

 

Find Specific files of a name or kind you may find for example all .png in a location  and then put them in your exclusion list incase they are mixed in with other files and such example below.

  find "/mnt/cache" -type f \( -name "*.png"\) -print

Open CA Mover Tuning GUI

In CA Mover Tuning Plugin Set the location of your skiplist.txt as below

    File list path: /boot/config/plugins/ca.mover.tuning/skiplist.txt

 

So can i put whole folders in to that textfile for the skipping list?

[IronBeardKnight]

2 hours ago, b0mb said:

 

So can i put whole folders in to that textfile for the skipping list?

Yes

[sasbro97]

Hey guys. I guess somebody has probably answered already but the search function does not show me anything useful. I have the same error as this user: 

mv: cannot stat '/usr/local/bin/mover': No such file or directory

I see it in the boot logs. It still boots but why is this error appearing? How to fix it?

 

[CS01-HS]

I noticed folder exclusion using the File list path option stopped working.

 

Enabling test mode I see this in the find command (where my exclude file is exclude.txt)

grep -vFx -f <(sed 's/\/*$//' '/boot/extras/mover_tuner/exclude.txt') 

 

I don't see how this can work with the -x option:

-x, --line-regexp 

Select only those matches that exactly match 
the whole line. For a regular expression 
pattern, this is like parenthesizing the pattern 
and then surrounding it with ^ and $.

 

Even files in exclude.txt won't satisfy the exact match unless they're prefaced with the cache path, e.g. /mnt/cache/sharename/file.txt vs simply sharename/file.txt, which used to work.

 

Am I missing something?

Edited December 10, 2023 by CS01-HS

[hugenbdd]

20 hours ago, sasbro97 said:

Hey guys. I guess somebody has probably answered already but the search function does not show me anything useful. I have the same error as this user: 

mv: cannot stat '/usr/local/bin/mover': No such file or directory

I see it in the boot logs. It still boots but why is this error appearing? How to fix it?

 

Mover location was changed somewhere between 6.9 and 6.11.  So older versions still need an older path, hence the "missing".

 

Once you run mover it creates a soft link.  If you search back several pages I think I posted how to create the link.  Will still go missing on bootup though.  DM me if you can't find how to create it.

[hugenbdd]

24 minutes ago, CS01-HS said:

I noticed folder exclusion using the File list path option stopped working.

 

Enabling test mode I see this in the find command (where my exclude file is exclude.txt)

grep -vFx -f <(sed 's/\/*$//' '/boot/extras/mover_tuner/exclude.txt') 

 

I don't see how this can work with the -x option:

-x, --line-regexp 

Select only those matches that exactly match 
the whole line. For a regular expression 
pattern, this is like parenthesizing the pattern 
and then surrounding it with ^ and $.

 

Even files in exclude.txt won't satisfy the exact match unless they're prefaced with the cache path, e.g. /mnt/cache/sharename/file.txt vs simply sharename/file.txt, which used to work.

 

Am I missing something?

I believe it should have always been /mnt/cache and not the sharename.  At least that is how I coded and tested it.  If sharename/file.txt was working, I don't think it was intentional.

[CS01-HS]

Just now, hugenbdd said:

I believe it should have always been /mnt/cache and not the sharename.  At least that is how I coded and tested it.  If sharename/file.txt was working, I don't think it was intentional.

 

Okay let's set that aside.

Exact match (-x) means folder exclusion can't work because while the folder will match none of the files within it will.

[Swarles]

21 hours ago, CS01-HS said:

 

Okay let's set that aside.

Exact match (-x) means folder exclusion can't work because while the folder will match none of the files within it will.

What issue are you noticing?

Inside the "exclusion.txt" file you will need to make sure that each path for exclusion is going through the "/mnt/cache" pool name share and not the "/mnt/user" share. You also need to make sure that if you are using the override settings for a particular share, you should include the exclude list inside those override settings too because it won't use the global exclude list.

[CS01-HS]

1 hour ago, Swarles said:

What issue are you noticing?

 

File-exclusion works as expected but folder-exclusion only excludes the folder itself, nothing under it.

 

e.g. given the folder structure:

MyShare/KeepOnCache/
		File1.txt
		File2.txt

 

And this entry in the exclude file:

/mnt/cache/MyShare/KeepOnCache

 

the find command (with sed evaluated) look like this:

> find "/mnt/cache/MyShare" -depth | grep -vFx "/mnt/cache/MyShare/KeepOnCache"

/mnt/cache/MyShare/KeepOnCache/File1.txt
/mnt/cache/MyShare/KeepOnCache/File2.txt

So File1.txt and File2.txt are not excluded from mover.

 

Is that intended and if so what's the point of folder-exclusion?

[Swarles]

5 hours ago, CS01-HS said:

 

File-exclusion works as expected but folder-exclusion only excludes the folder itself, nothing under it.

 

e.g. given the folder structure:

MyShare/KeepOnCache/
		File1.txt
		File2.txt

 

And this entry in the exclude file:

/mnt/cache/MyShare/KeepOnCache

 

the find command (with sed evaluated) look like this:

> find "/mnt/cache/MyShare" -depth | grep -vFx "/mnt/cache/MyShare/KeepOnCache"

/mnt/cache/MyShare/KeepOnCache/File1.txt
/mnt/cache/MyShare/KeepOnCache/File2.txt

So File1.txt and File2.txt are not excluded from mover.

 

Is that intended and if so what's the point of folder-exclusion?

Have you tried testing it or are you just going off the code?

 

That part of the code is only meant to exclude the top level folder path or individual files if they have been specified in the exclude list, it doesn't exclude anything underneath a folder. The find string will then use a -not -path "/mnt/cache/MyShare/KeepOnCache/*" to exclude the files underneath the folder.

[CS01-HS]

1 hour ago, Swarles said:

The find string will then use a -not -path "/mnt/cache/MyShare/KeepOnCache/*" to exclude the files underneath the folder.

 

Oh, I didn't realize.

I was going off the debug output in test mode.

 

I tried a real run (now with full paths) and it works. 

Thanks!

 

[Scheev]

Is there a way to use this tuner to:

 

1. Move only when the cache fills to a certain amount

2. Move only the oldest files

 

any help is appreciated!

[kizer]

11 hours ago, Scheev said:

Is there a way to use this tuner to:

 

1. Move only when the cache fills to a certain amount

2. Move only the oldest files

 

any help is appreciated!

 

That's exactly what the script does. I have mine set to only move items if the drive is 5% used. Meaning it will try and run every day. Then I have it set to only move files that are 15days or older. 

 

So with my settings it'll try and run every day because I have more than 5% on my drive, but hold all files until they are at least 15 days. Files that are 0-14days old will still be on the drive. Obviously I could set the thresh hold to 15% or more, but I'm more concerned with the how many days vs how full my drive is. 

 

I want files to sit on my drive for quick access and so when I watch something in plex I don't have to wait for my drives to spool up because they are sitting on my SSD. Anything older than 15days will spool up the drives, but I'm taking less of a risk that my files could be lost if my SSD dies. However, I do have a backup unraid machine that I backup all my media to and it just happens to look at my SSD to so even if I lost my SSD I'd have it all backed up anyways. 

[martinkeat]

Steps Tried: 1-6

System Log: Put in a spoiler. 
Mover Tuning Log: 

 

Quote

mvlogger: *********************************MOVER START*******************************
mvlogger: Age supplied: 0
mvlogger: Size supplied: 0
mvlogger: Sparness supplied: 0
mvlogger: No Skipfiles Argument Supplied
mvlogger: No Skip File Types Argument Supplied
mvlogger: No Before Script Argument Supplied
mvlogger: No After Script Argument Supplied
mvlogger: No CTIME Argument Supplied
mvlogger: CACHE THRESH to Move all Cache-Yes shares to array: 25
mvlogger: No Test Mode Argument Supplied
mvlogger: No Ignore Hidden Files Argument Supplied
mvlogger: Threshold Percent: 20
mvlogger: No Script to Run.
mvlogger: CACHETHRESH: 25
mvlogger: Checking to see if we should move all files from share.
mvlogger: Move All Cache Threshold: 25
mvlogger: The fstype of /mnt/cache is zfs.
mvlogger: /mnt/cache PCT USED: 36
mvlogger: Pool /mnt/cache is above 'move all' percentage, moving all cache-yes shares to array
mvlogger: Done checking Move All from Share
mvlogger: Share Name Only: Ripped
mvlogger: Cache Pool Name: cache 
mvlogger: cache Threshold Pct: 
mvlogger: OVERALL Threshold: 20
mvlogger: Share Path: /mnt/cache/Ripped
mvlogger: cache is zfs.
mvlogger: Pool Pct Used: 36 %
mvlogger: DFTPCT LIMIT USED FOR SETTING: 20
mvlogger: Threshold Used: 20
mvlogger: Share Name Only: system
mvlogger: Cache Pool Name: cache 
mvlogger: cache Threshold Pct: 
mvlogger: OVERALL Threshold: 20
mvlogger: Share Path: /mnt/cache/system
mvlogger: cache is zfs.
mvlogger: Pool Pct Used: 36 %
mvlogger: DFTPCT LIMIT USED FOR SETTING: 20
mvlogger: Threshold Used: 20
mvlogger: after customFilelist
mvlogger: After totalsizeFilelist
mvlogger: -----------------------------------------Running Array Files to Cache drives----------------------------------------------
mvlogger: Running to cache drive
mvlogger: File to Cache filelist created
mvlogger: Cache File List: /tmp/Mover/Cache_Tuning_2023-12-12T234731.list
mvlogger: make customArray File List
mvlogger: write first status file after to array
mvlogger: Hard Link Status: false
mvlogger: No /tmp/Mover/Custom_Mover_Directory_2023-12-12T234731.list created as no empty directories
mvlogger: No After Script to Run.
 

Extra Details: eg. Only stopped working after latest update.

[Scheev]

8 hours ago, kizer said:

 

That's exactly what the script does. I have mine set to only move items if the drive is 5% used. Meaning it will try and run every day. Then I have it set to only move files that are 15days or older. 

 

So with my settings it'll try and run every day because I have more than 5% on my drive, but hold all files until they are at least 15 days. Files that are 0-14days old will still be on the drive. Obviously I could set the thresh hold to 15% or more, but I'm more concerned with the how many days vs how full my drive is. 

 

I want files to sit on my drive for quick access and so when I watch something in plex I don't have to wait for my drives to spool up because they are sitting on my SSD. Anything older than 15days will spool up the drives, but I'm taking less of a risk that my files could be lost if my SSD dies. However, I do have a backup unraid machine that I backup all my media to and it just happens to look at my SSD to so even if I lost my SSD I'd have it all backed up anyways. 

Okay cool, I tried looking at the settings page, but I am not seeing an obvious way to have both of those conditions. Do you mind sharing how exactly you achieved that?

[kizer]

14 hours ago, Scheev said:

Okay cool, I tried looking at the settings page, but I am not seeing an obvious way to have both of those conditions. Do you mind sharing how exactly you achieved that?

 

Sure its a bit old screen shot, but here are my settings that I still use today. Keep in mind there might be an option or two not shown on this one, but I think its pretty self explanatory when you read the new ones. 

Just to clarify a thing or two. Every day at 5:45AM my system fires off the Mover script based off the unraid default settings. It looks to make sure that at least 5% is used and only moves files that are 20days old. The reason I did this was simple. I didn't want the script firing off every minute because in the past I noticed that when the script ran it was spinning up all my drives. No idea if that has been addressed, but doing it my way it doesn't matter because I have other things running at 6AM and these runs prior to. 

 

So after it fires of the script and verifies that at least 5% is used because there needs to be some kind of threshold set because its my understanding that it uses the threshold as the basis for triggering other things. If that was changed I'm covered either way. 😃

 

So now that I've set it to run everyday at 5:45 and at least 5% is used it looks for files that are Older than 20 days and moves them. 

 

[hugenbdd]

New Release thanks to @Swarles

 

###2023.12.19
- Change "while read" lines in age_mover to "while IFS= read -r" to fix trailing white spaces (Swarles)
- Fix where sometimes mover would not run to mover.old scrip (Swarles)
- Log if "share.cfg" doesn't exists to help trouble shooting (Swarles)
- Check for ca.mover.tuning.cfg file and additional logging. (Swarles)

[trurl]

Why do you want to stop any containers to run mover? Your appdata should be all on fast pool such as cache and configured to stay there untouched by mover.

[trurl]

I don't use that container, but I do use transmission, downloads to a cached share, don't even bother with mover tuning plugin, and have never had an issue with my torrents.

[DevanteWeary]

Guys still trying to get a handle on this awesome plugin which seems to have a lot of work put into it so thank you!

 

So am I understanding this correctly?  

This plugin (and Mover itself?) only moves OFF of the cache drive?

 

Or does it follow the Array > Cache preference too?

 

Basically I have some shares set as Array > Cache and at the end of the day, if the space becomes available on the cache, I want the files moved FROM the array (HDD ) to the cache (SSD).

 

Basically I want the drives to be used as overflow but those files to eventually end up on the cache automatically.

 

Is this possible?

[itimpi]

30 minutes ago, DevanteWeary said:

Or does it follow the Array > Cache preference too?

This.

[DevanteWeary]

2 hours ago, itimpi said:

This.

 

Thank you!

 

I guess the main thing I get is how does it work with the percentage full filter?

 

The settings is called "Only move at this threshold of used cache space:"

 

If I set that to 75%, and have the preference to Array > Cache, it seems to me that if the CACHE drive gets fill up to 75%, it'll move files from the array TO the cache drive thereby filling the cache even more.

[Swarles]

15 hours ago, DevanteWeary said:

 

So am I understanding this correctly?  

This plugin (and Mover itself?) only moves OFF of the cache drive?

 

Or does it follow the Array > Cache preference too?

 

Basically I have some shares set as Array > Cache and at the end of the day, if the space becomes available on the cache, I want the files moved FROM the array (HDD ) to the cache (SSD).

 

12 hours ago, DevanteWeary said:

If I set that to 75%, and have the preference to Array > Cache, it seems to me that if the CACHE drive gets fill up to 75%, it'll move files from the array TO the cache drive thereby filling the cache even more.

Just to clarify, the mover tuning plug-in settings won’t affect “Array > Cache” shares, but the mover itself still affects them. They will be treated as if it was the normal mover and an attempt to move all files in that share from the Array to the Cache will be made regardless of the settings in the mover tuning plug-in. The mover tuning plug-in settings only affect moves made from the cache to the array. You’ll notice that if you click into a share, the option to override the global mover tuning settings is only available for “Cache > Array” shares.

 

This is not to say that it can’t be supported in the future, it just hasn’t been so far and is probably a pretty niche use case.

 

 

Edited January 2 by Swarles

[BM118]

Wondering if anyone can assist.

 

Tl;DR - Mover tuning was (maybe still) not moving all files as expected, however it has been running and moving others. Invoking mover manually moved all the files as expected. This same mover tuning setup has been working since 6.10, during 6.11 and AFAIK 6.12.4. Currently running 6.12.6.

 

Over the Xmas break decided to upgrade from 6.12.4 to 6.12.6, but i doubt thats related. Upgraded from 6.11.5 to 6.12.4 months and months ago, this issue could have been around since then for all i know. It should be noted that some of the affected files were created well before December. I did change my cache setup straight after the 6.11.5 to 6.12 update to be ZFS, but confirmed files were moving at that time (as it is required to change the cache setup).

 

While investigating and making assumptions about files being locked i noticed this was affecting both old (well before the 6.12.6 update) and new files (this week, even some from yesterday). Using the "compute all" button in the shares, i noticed instances of cache still being listed when it should not have been, and other instances of drives being listed when they should not have been (note that all files were in "allowed"/"expected" locations before a move has run, but not after a move has run). confirmed through CLI, looking at the individual drive mounts, that the files were indeed on the wrong drives. I did some checks, and made sure there werent any file locks etc. I even made some edits to some to confirm it was writeable and not locked.

 

This morning i again checked and noticed they were not moved as expected (same files as before) but other files had been moved in other shares. In an effort to monitor it live, i went to the Main tab and scrolled down, clicked on the Invoke Mover now button, and all files were moved as expected. Unfortunately i dont have screenshots of it, but i do have screenshots of the pertinent mover log from early AM this morning, vs the manual mover. Mover logging was not enabled for any of this, but "log when not moving due to rules" is enabled.

 

Mover Tuning (the nextcloud content on the cache should have been moved at this point (1.4GB images), along with various other files in various shares):

 

Manual Mover (what you dont see here is the files in other shares that were also moved):

[MadMatt337]

I noticed today for the first time that I had one of my cache drives fill up before my nightly move, and although I had the setting in mover tuning setup to "Move All from Cache-yes shares pool percentage:" at 80% it did not enable the move function automatically. Is there a current glitch with this? Maybe since the change of the cache setup since they no longer use the terms cache-yes as such?