[Plugin] Mover Tuning

[DevanteWeary]

2 hours ago, itimpi said:

This.

 

Thank you!

 

I guess the main thing I get is how does it work with the percentage full filter?

 

The settings is called "Only move at this threshold of used cache space:"

 

If I set that to 75%, and have the preference to Array > Cache, it seems to me that if the CACHE drive gets fill up to 75%, it'll move files from the array TO the cache drive thereby filling the cache even more.

[Swarles]

15 hours ago, DevanteWeary said:

 

So am I understanding this correctly?  

This plugin (and Mover itself?) only moves OFF of the cache drive?

 

Or does it follow the Array > Cache preference too?

 

Basically I have some shares set as Array > Cache and at the end of the day, if the space becomes available on the cache, I want the files moved FROM the array (HDD ) to the cache (SSD).

 

12 hours ago, DevanteWeary said:

If I set that to 75%, and have the preference to Array > Cache, it seems to me that if the CACHE drive gets fill up to 75%, it'll move files from the array TO the cache drive thereby filling the cache even more.

Just to clarify, the mover tuning plug-in settings won’t affect “Array > Cache” shares, but the mover itself still affects them. They will be treated as if it was the normal mover and an attempt to move all files in that share from the Array to the Cache will be made regardless of the settings in the mover tuning plug-in. The mover tuning plug-in settings only affect moves made from the cache to the array. You’ll notice that if you click into a share, the option to override the global mover tuning settings is only available for “Cache > Array” shares.

 

This is not to say that it can’t be supported in the future, it just hasn’t been so far and is probably a pretty niche use case.

 

 

Edited January 2 by Swarles

[BM118]

Wondering if anyone can assist.

 

Tl;DR - Mover tuning was (maybe still) not moving all files as expected, however it has been running and moving others. Invoking mover manually moved all the files as expected. This same mover tuning setup has been working since 6.10, during 6.11 and AFAIK 6.12.4. Currently running 6.12.6.

 

Over the Xmas break decided to upgrade from 6.12.4 to 6.12.6, but i doubt thats related. Upgraded from 6.11.5 to 6.12.4 months and months ago, this issue could have been around since then for all i know. It should be noted that some of the affected files were created well before December. I did change my cache setup straight after the 6.11.5 to 6.12 update to be ZFS, but confirmed files were moving at that time (as it is required to change the cache setup).

 

While investigating and making assumptions about files being locked i noticed this was affecting both old (well before the 6.12.6 update) and new files (this week, even some from yesterday). Using the "compute all" button in the shares, i noticed instances of cache still being listed when it should not have been, and other instances of drives being listed when they should not have been (note that all files were in "allowed"/"expected" locations before a move has run, but not after a move has run). confirmed through CLI, looking at the individual drive mounts, that the files were indeed on the wrong drives. I did some checks, and made sure there werent any file locks etc. I even made some edits to some to confirm it was writeable and not locked.

 

This morning i again checked and noticed they were not moved as expected (same files as before) but other files had been moved in other shares. In an effort to monitor it live, i went to the Main tab and scrolled down, clicked on the Invoke Mover now button, and all files were moved as expected. Unfortunately i dont have screenshots of it, but i do have screenshots of the pertinent mover log from early AM this morning, vs the manual mover. Mover logging was not enabled for any of this, but "log when not moving due to rules" is enabled.

 

Mover Tuning (the nextcloud content on the cache should have been moved at this point (1.4GB images), along with various other files in various shares):

 

Manual Mover (what you dont see here is the files in other shares that were also moved):

[MadMatt337]

I noticed today for the first time that I had one of my cache drives fill up before my nightly move, and although I had the setting in mover tuning setup to "Move All from Cache-yes shares pool percentage:" at 80% it did not enable the move function automatically. Is there a current glitch with this? Maybe since the change of the cache setup since they no longer use the terms cache-yes as such?

[alturismo]

21 minutes ago, MadMatt337 said:

I had the setting in mover tuning setup to "Move All from Cache-yes shares pool percentage:" at 80% it did not enable the move function automatically.

the mover is scheduled, mover tuning is a "filter" WHEN the mover is running ...

 

so when you use a 80 % filter ... this will jump in ONLY when the mover is triggered ...

 

it WONT watch the fillrate in real time and trigger the mover ... so rather increase your schedule triggers (like hourly) and the mover will start each hour, IF the fillrate is 80+ it ll move, otherwise not ...

[huggie]

Just wanted to say what a great plugin this is

I see you can run scripts before and after the mover is enabled, does anyone have a decent script that stops torrents or the container before the move so they can be moved and then restart them after?

 

Thankyou

[huggie]

me again, i set up Mover Tuning to move files that are over 20days old. That way I can keep new stuff on the cache and move it off after its been watched and seeded.

 

Problem is it doesn't seem to be working. I'm getting this error in the logs.

 

 root: Log Level: 1
 root: /tmp/Mover Directory does not exist. Creating it
 root: mover: started
 root: df: /mnt/cache: No such file or directory
 root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 687: [: -gt: unary operator expected
 root: mover: finished
 emhttpd: spinning down /dev/sdb
 emhttpd: read SMART /dev/sdb

 

Any idea what is going on here? I've tried deleting this plugin and installing again but it still does it.

[brandonhowlett]

Haven't made any changes recently. Noticed this spamming the syslog today and my cache drives were full.

 

Move files off cache based on age has always been set to no.

 

move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 372: ((: TOTALARRAYSIZE += : syntax error: operand expected (error token is "+= ")

Edited January 16 by brandonhowlett

[Swarles]

14 hours ago, huggie said:

 root: df: /mnt/cache: No such file or directory
 root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 687: [: -gt: unary operator expected

 

Any idea what is going on here? I've tried deleting this plugin and installing again but it still does it.

Can you confirm if you have a cache pool called "cache" and if the location "/mnt/cache" exists?

My first thoughts are that your cache pool is not called that but you may have some left over cache pool config files located at "/boot/config/pools". Check that location and see if all your pools are correct.

[Swarles]

2 hours ago, brandonhowlett said:

Haven't made any changes recently. Noticed this spamming the syslog today and my cache drives were full.

 

Move files off cache based on age has always been set to no.

 

move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 372: ((: TOTALARRAYSIZE += : syntax error: operand expected (error token is "+= ")

Not 100% sure but it seems like you might be getting an invalid file size for some of the files that are performing an array>cache move. Could you check the contents of your "/tmp/Mover/Custom_Cache_Tuning_$NOW.list" file, you may have some entries that are missing a number at the end.

[huggie]

16 hours ago, Swarles said:

Can you confirm if you have a cache pool called "cache" and if the location "/mnt/cache" exists?

My first thoughts are that your cache pool is not called that but you may have some left over cache pool config files located at "/boot/config/pools". Check that location and see if all your pools are correct.

 

My cache pool was renamed to cache_nvme. would this mess it up?

[Swarles]

4 hours ago, huggie said:

 

My cache pool was renamed to cache_nvme. would this mess it up?

The mover tuning plugin uses the cache pool configuration files located in "/boot/config/pools" to determine what pools exist and their settings, check to see if you have a file called "cache.cfg" in there in addition to your "cache_nvme.cfg" file.

 

Let me know if you don't have one, or if you only have "cache.cfg" check inside it to see what the cache name reports back.

[huggie]

11 hours ago, Swarles said:

The mover tuning plugin uses the cache pool configuration files located in "/boot/config/pools" to determine what pools exist and their settings, check to see if you have a file called "cache.cfg" in there in addition to your "cache_nvme.cfg" file.

 

Let me know if you don't have one, or if you only have "cache.cfg" check inside it to see what the cache name reports back.

 

 

Right theres a cache.nvme.cfg file but no cache.cfg file.

[Swarles]

2 hours ago, huggie said:

 

 

Right theres a cache.nvme.cfg file but no cache.cfg file.

Can you confirm if the path that exists is,

/mnt/cache.nvme
or
/mnt/cache_nvme

It could be that perhaps the code is treating the ".nvme.cfg" as the full extension and removing it, but I have checked this and it should not be the case. But for a better understanding you should provide some logs.

1. Go to Settings>Scheduler>Mover Settings
2. Set "Mover Logging" to Enabled.
3. Go to the Mover Tunning settings and set "Test Mode" to "Yes"
4. Set "Move Now button follows plug-in filters" to "Yes".
5. On the Main tab press the "Move" button to run the mover. Running in test mode will generate logs but not make moves.
6. Go to "/tmp/Mover" and get the most recent file ending in ".log"
7. You can revert test mode and logging now.

You can supply the .log contents here in a spoiler or send them to me privately. This might help me understand why you are experiencing behaviour that is unexpected.

[huggie]

23 minutes ago, Swarles said:

 

 

its cache_nvme.cfg

 

The logs are exactly the same that I posted a few posts up. But I'll run it again and send it to you in a PM

[ApfelBirneKreis]

I want to shutdown docker services with a script and enable it after the mover ran again.

how do i run the script? Is it in bash? whats the filename? Is there any documentation?

[Texot]

On 1/17/2024 at 8:25 AM, Swarles said:

Not 100% sure but it seems like you might be getting an invalid file size for some of the files that are performing an array>cache move. Could you check the contents of your "/tmp/Mover/Custom_Cache_Tuning_$NOW.list" file, you may have some entries that are missing a number at the end.

There is a bug in age_mover line 368 reading CUSTOMLINE while in 370 it's referring to CUSTOMFILE

[Swarles]

9 hours ago, Texot said:

There is a bug in age_mover line 368 reading CUSTOMLINE while in 370 it's referring to CUSTOMFILE

Oh yep, that'll do it, fixed now. Should be fixed in the next update. Thank you for spotting that

[Autchirion]

Somehow it seems to me, that moving when the cache is full isn't working.

• I've got a share with the "Move All from Cache-Yes shares when disk is above a certain percentage" yes.
• Two caches (Naming Cache and Datacache), where for this share the Datacache (~120GB) is used
• I'm downloading ~100 files with ~2GB each, files are beeing pre generated (~1KB) and then after this filled with data. -> Mover isn't being activated when Datacache is 80% full.

I'm not sure if I set up something wrong or if this usecase isn't covered by mover tuning.

 

[EDIT] since it sounded similar, cache.cfg and datacache.cfg exist, I'm not sure if it is case sensitive, because the filename is lowercase, but the first letter of the Cache Name in the UI is uppercase.

 

Mover Tuning - Share settings:

 

Mover Tuning Settings:

Edited January 25 by Autchirion

[alturismo]

34 minutes ago, Autchirion said:

Mover isn't being activated when Datacache is 80% full.

the mover is running scheduled ... so the trigger is ONLY time based ...

 

WHEN its triggered it ll follow the rules

 

so, did you set the schedule and waited for it to run (supposed behaviour) ?

or did you expect the mover kicks in automatically while reaching the fill rate (wont happen) ?

[Autchirion]

3 minutes ago, alturismo said:

or did you expect the mover kicks in automatically while reaching the fill rate (wont happen) ?

yes, I expected this, damn it. 🙂 

[alturismo]

2 minutes ago, Autchirion said:

yes, I expected this, damn it. 🙂 

then may read a littl ebackwards or overall how it works.

 

sample (same page ... here)

 

 

so yes, you where not alone ... but may also read the manual again

 

in my case, i just inreased the scheduler to run hourly ... as i control all by mover tuning.

[Autchirion]

Yeah, my Problem is, cache is only 120GB, if I set it to e.g. move only at 80% I'd need to run it every 3.4 Minutes (assuming 1Gbit/s upload rate which is my home network speed) to make sure it never fills up.

[Kilrah]

14 minutes ago, Autchirion said:

I'd need to run it every 3.4 Minutes (assuming 1Gbit/s upload rate which is my home network speed) to make sure it never fills up.

Then disable cache for those shares, it's completely pointless to use it.

It's useful if you're using it to absorb bursts of data that are coming in faster than the array can take or to store things on for a few days, but if you have to empty it continuously you don't gain anything.

Edited January 25 by Kilrah

[itimpi]

3 hours ago, Autchirion said:

Yeah, my Problem is, cache is only 120GB, if I set it to e.g. move only at 80% I'd need to run it every 3.4 Minutes (assuming 1Gbit/s upload rate which is my home network speed) to make sure it never fills up.

It is worth pointing out that if you have Turbo Write enabled you can write directly to the array at speeds of that order of magnitude.   Probable a better way to handle your requirements anyway.