[Plugin] Mover Tuning

[bukunraid]

8 hours ago, alturismo said:

 

what doesnt work (would be sweet), fillrate 90+ reached, now move oldest files UNTIL (sample) 70 % is reached ...

better speaking, always keeping xy % alive on cache ... in the end, im really fine as is with the age setup only, but you probably looking for another option to specify the "keep on cache rate" which doesnt exist (yet).

 

Perhaps I didn't word it in the best way but what you describe is basically what I want, either/or when cache is over 90% then move 'x' amount of oldest files, if that's up to 'x' cache percentage or whatever would be good. 

 

I've used the 'move files older than'  but it's difficult to get right and have had a couple of instances where cache fills up before the oldest files have been moved.

[jit-010101]

Is there a way to stop/disable the mover completely just for a specific share - similiar to how you disable it globally?

 

or even better to execute a custom script instead, or both combined ?

Edited November 20, 2023 by jit-010101

[Kilrah]

1 hour ago, jit-010101 said:

Is there a way to stop/disable the mover completely just for a specific share

You'd just change that share's storage and mover settings.

[jit-010101]

4 hours ago, Kilrah said:

You'd just change that share's storage and mover settings.

 

Not possible, obviously thats the first thing i tried and looked for - share storage settings should be left untouched and makes no sense to disable it because thats serving as the very base layer of movers.

Edited November 20, 2023 by jit-010101

[wgstarks]

1 hour ago, jit-010101 said:

 

Not possible, obviously thats the first thing i tried and looked for - share storage settings should be left untouched and makes no sense to disable it because thats serving as the very base layer of movers.

I think perhaps you misunderstood. You would set the shares primary storage to “cache” and mover action to “array>cache” and mover will not move it off the cache disk. This setting will be in the share settings page.

 

[Kilrah]

1 hour ago, jit-010101 said:

 

Not possible, obviously thats the first thing i tried and looked for - share storage settings should be left untouched and makes no sense to disable it because thats serving as the very base layer of movers.

Mover behavior is configured individually for each share in the share's settings, so if you have a share where you don't want the mover to do anything you just set that share as such, e.g. disable "secondary storage". 

Edited November 20, 2023 by Kilrah

[jit-010101]

13 hours ago, wgstarks said:

I think perhaps you misunderstood. You would set the shares primary storage to “cache” and mover action to “array>cache” and mover will not move it off the cache disk. This setting will be in the share settings page.

 

 

No I understood that correctly - but this would automatically move every single file from the Array to the Cache. Which is not what I would like to do, since it doesn't fit my use case.

 

Same goes for disabling secondary storage - because that's actually really usefull if you want to manually move some files but not everything.

 

This is also why neither of your approaches would really work in this case.

 

Right now you can disable the Mover globally, doing so on a per Share level too would be massively helpfull is all I'm saying. Pair that with an script option that is run when the Mover is triggered when enabled would be even more usefull.

 

 

Edited November 21, 2023 by jit-010101

[itimpi]

9 minutes ago, jit-010101 said:

Same goes for disabling secondary storage - because that's actually really usefull if you want to manually move some files but not everything.

I still do not understand why this does not suit your Use Case - disabling Secondary storage by definition stops mover running on that share.  What am I missing?

[Kilrah]

1 hour ago, jit-010101 said:

Same goes for disabling secondary storage - because that's actually really usefull if you want to manually move some files but not everything.

Disabling secondary storage doesn't prevent you from having/manually putting files on multiple drives, they'll still be consolidated when reading. Just means that everything new will go to primary storage and no automated action will happen, which is what you apparently want.

Edited November 21, 2023 by Kilrah

[jit-010101]

1 hour ago, itimpi said:

I still do not understand why this does not suit your Use Case - disabling Secondary storage by definition stops mover running on that share.  What am I missing?

 

Because that disables the shfs layering too (/mnt/user/[share]) so you'll have to choose which disk to write on manually e.g. and completely disabling read access by that file path - e.g. in docker containers.

 

(at least that's how I understand how unraid works, correct me when I'm wrong, but please read till the end even if that part is not completely accurate - I've been using unraid just for a few months at best so my insight is a little bit limited)

 

Well better said it does not disable the path completely but its just pointing at the primary storage only, not giving you the option to build on the concept that unraid invented, just with your own logic (with your proposale you'd basically have to write to the specific disks manually - and they're not part of /mnt/user/[share] so if you mount that into a docker container you could not access all movies f.e.).

 

That's where the second part comes into play -> custom mover logic -> say not completely disabling it but basically enabling us to run a custom shellscript on the move trigger, similiar to how you likely do it already for "files older then" mover, which - as I understand it - triggers a custom script you shipped already (?).

 

Use cases could be - write to SSD as primary storage, trigger custom mover script which will just move some files to HDD

 

Same can be inversed - like primary storage HDD, trigger custom mover script which will just move some files to SSD

 

For example for movies that get frequently streamed often (e.g. kids watching a specific movies x times a week)

 

This could be used for example for frequently access files, since atime is not enabled by default so people can implement their own custom move-logic, be it enabling atime for their zfs datasets, or having their own toolings around how/which files are frequently accessed.

 

I proposed it like this, specifically because atime is not enabled by default in unraid.

 

 

Edited November 21, 2023 by jit-010101

[itimpi]

Primary storage is always where Unraid writes NEW files, but it can find files that belong to a share for read purposes regardless of whether they are on primary storage or not so the SHFS layer is not completely disabled.

 

If you later want to temporarily enable mover just assign secondary storage and set the mover direction appropriately.

 

[jit-010101]

4 hours ago, itimpi said:

Primary storage is always where Unraid writes NEW files, but it can find files that belong to a share for read purposes regardless of whether they are on primary storage or not so the SHFS layer is not completely disabled.

 

I'm a little bit confused - wasn't the proposal to disable secondary storage? 

 

Or otherwise just swap them - so that would mean as soon as the mover is triggered it would still move files it isn't supposed to do (just in the other direction).

 

Does the ignore file-list support RegExp/Wildcards? The extension-list doesn't ... already tried that.

 

4 hours ago, itimpi said:

If you later want to temporarily enable mover just assign secondary storage and set the mover direction appropriately.

 

Isn't that unfesible if you want to always read from both primary and secondary storage?

 

Say you have some movies stored on SSD and some on HDD - and you want to keep it there

 

For that disabling the mover per share might be usefull is all I'm saying to begin with ... because with the current options of this plugin you can only go with so many parameters and options as far as I can see it.

 

But not that very simple use-case that simply goes by disabling the move for everything for this share alone (no matter how you call it to be honest)

Edited November 21, 2023 by jit-010101

[itimpi]

3 hours ago, jit-010101 said:

But not that very simple use-case that simply goes by disabling the move for everything for this share alone (no matter how you call it to be honest)

That was what I was saying you should do.   It seems to satisfy the Use Case you mention.

[Kilrah]

12 hours ago, jit-010101 said:

Because that disables the shfs layering too

No it doesn't. It's always there for reads whatever the share settings.

Edited November 21, 2023 by Kilrah

[xokia]

at start up I get
mv: cannot stat '/user/local/bin/mover' : No such file or directory

 

Everything operates fine I just get this message at startup. I have tried uninstalling and reinstalling mover tuning no luck.

How do I fix? create a soft link?

Edited November 22, 2023 by xokia

[Andiroo2]

Odd issue today...trying to move files off my ZFS cache and...nothing is happening.  I've had Mover Tuning for years without issue.  I have tried changing a LOT of these settings to see if anything changes and nothing has worked so far. I haven't tried disabling Mover Tuning yet because I don't want to clear the cache pool out at this point. 

 

My Mover Tuning settings:

 

My share overview is attached as well.

 

If I run the dedicated "move all from this share" in a shares settings, Mover works.  When I run mover via Move Now (or via the schedule), the log shows this:

 

Nov 22 06:43:37 Tower emhttpd: shcmd (206793): /usr/local/sbin/mover |& logger -t move &
Nov 22 06:43:37 Tower root: Starting Mover
Nov 22 06:43:37 Tower root: Forcing turbo write on
Nov 22 06:43:37 Tower root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start 45 5 0 "/mnt/user/system/Mover_Exclude.txt" "ini" '' '' no 100 '' '' 95
Nov 22 06:43:37 Tower kernel: mdcmd (46): set md_write_method 1

  

I don't get the ****staring mover*** line in the log.  

 

When I run mover manually from a share's settings page, I get a much nicer looking set of logs:

mvlogger: Log Level: 1
mvlogger: *********************************MOVER -SHARE- START*******************************
mvlogger: Wed Nov 22 05:52:42 EST 2023
mvlogger: Share supplied CommunityApplicationsAppdataBackup
mvlogger: Cache Pool Name: cache 
mvlogger: Share Path: /mnt/cache/CommunityApplicationsAppdataBackup
mvlogger: Complete Mover Command: find "/mnt/cache/CommunityApplicationsAppdataBackup" -depth | /usr/local/sbin/move -d 1
file: /mnt/cache/CommunityApplicationsAppdataBackup/ab_20231120_020002-failed/my-telegraf.xml
file: /mnt/cache/CommunityApplicationsAppdataBackup/ab_20231120_020002-failed/my-homebridge.xml
...

 

Running a find command shows plenty of files older than the specified age that should be caught by the mover:

root@Tower:~# find /mnt/cache/Movies -mtime +45
/mnt/cache/Movies/Movies/BlackBerry (2023)/BlackBerry (2023).eng - 1080p.srt
/mnt/cache/Movies/Movies/BlackBerry (2023)/BlackBerry (2023).eng -.srt
/mnt/cache/Movies/Movies/BlackBerry (2023)/BlackBerry (2023) - 1080p.fr.srt
/mnt/cache/Movies/Movies/BlackBerry (2023)/BlackBerry (2023).eng.HI -.srt
/mnt/cache/Movies/Movies/BlackBerry (2023)/BlackBerry (2023) - 1080p.es.srt
/mnt/cache/Movies/Movies/BlackBerry (2023)/BlackBerry (2023) - 1080p.en.srt
/mnt/cache/Movies/Movies/BlackBerry (2023)/BlackBerry (2023) - 1080p.mp4
/mnt/cache/Movies/Movies/80 for Brady (2023)
/mnt/cache/Movies/Movies/80 for Brady (2023)/80 for Brady (2023) WEBDL-2160p.es.srt
/mnt/cache/Movies/Movies/Toy Story 4 (2019)/Toy Story 4 (2019).mkv
/mnt/cache/Movies/Movies/Toy Story 4 (2019)/Toy Story 4 (2019).en.srt
/mnt/cache/Movies/Movies/Toy Story 4 (2019)/Toy Story 4 (2019).en.forced.srt
/mnt/cache/Movies/Movies/Toy Story 4 (2019)/Toy Story 4 (2019).es.srt
/mnt/cache/Movies/Movies/Toy Story 4 (2019)/Toy Story 4 (2019).fr.srt

 

Diagnostics attached in case it helps.  Thanks!

tower-diagnostics-20231122-0656.zip

[Sebby]

Hi - I know its an issue that has been mentioned previously but i also got the error:

/usr/local/sbin/mover &> /dev/null &

if trying to invoke mover or any use of mover with mover tuning installed.

Mover was invoked fine once mover tuning was uninstalled.

 

Changing the cache threshold did not work for me.

 

Unsure if there is another fix possible or it's my setup that caused the issue.

sebbytower-diagnostics-20231124-1317.zip

[NEURO]

Until a few days ago I used to manually move very well but now it doesn't work for me the disk has been filled to 100% and I am manually copying the files from the cache I put my settings and I don't know how to download my logs if someone guides me on what could have happened I am a newbie

[Indi]

"Move All from Cache-yes shares pool percentage" isn't working anymore  is this a known issue with ZFS pools by chance? 

My cache pool is 92% full, I have it set to Yes if it's above 90%, but it's not moving it. 

[Revan335]

Is this the same like Share Mover/Cache Settings? For example system/appdata only cache and other Shares Cache First an by exceeded the Limit min free Space in the Cache and Share Settings move to Array.

 

Or Override this Mover Plugin Setting the Share/Cache Settings?

[PedroPutenstripe]

i am new to unraid, so sorry if its a silly question, but im not sure if this is the expected behaviour of the mover tuner.

problem:
i want to use the "move files based on age" setting, but the mover is moving all files, and ignores the age.

tested case:
my cache space was at 88%, when the scheduled run started.
all the files were created on the day before, so age is 1day for all files.

my movertuner settings are:
move above threshold: 80%
move files that are greater than this many days old: 40

result:
it moved all files onto the array.

expected result.
it should've moved no file onto the array?



any ideas?

 

[Dviance]

After months of neglecting my system, I finally booted it up and found that I cannot invoke mover from the front page or any other pages. (this was after I ran all my updates etc)

After some tinkering, I did get mover to work from the main page by changing my shares SMB Security Settings from Private to Public.

I'm not sure if it relates to the issues multiple members have been experiencing but is worth a shot and could hopefully result in a fix if it was indeed the issue. 

 

[dyedfire]

On 9/5/2023 at 10:51 PM, Swarles said:

Tagging some people this is relevant for (don't have to do it if you don't want): @Robbie G78, @Kloudz, @sourCream, @wibble, @WWGT, @knights_of_pine, @Chunks, @Taddeusz, and @Magic815

Okay so for everyone where the Mover Tuning Plugin is no longer working and the mover wont run please follow these steps to help me figure this out

 

Fill out this quick template:

 

Troubleshooting - Try each step one at a time, if it does not fix your issue try the next one:

1. Enable mover logging! "Settings>Scheduler>Mover Settings" 
2. (Manual Moves Only): Is "Move Now button follows plug-in filters:" (In "Settings>Scheduler>Mover Tuning") set to Yes? If not, make this change and run the mover again.
   If this now works, PLEASE LET ME KNOW.

3. Check to see if "share.cfg" exists.

     Reveal hidden contents

    

   ls "/boot/config"

   If it does exist:

   cat "/boot/config/share.cfg"

   Check to see if the following line is in the file 'shareCacheEnabled="yes"'. If not add it:

   echo 'shareCacheEnabled="yes"' >> /boot/config/share.cfg

   Run the mover again. If it works now, please let me know.

4. Check to see if "ca.mover.tuning.cfg" exists.
     Reveal hidden contents

     

   ls "/boot/config/plugins/ca.mover.tuning"

    

   If it does not exist, go to "Settings>Scheduler>Mover Tuning", make some changes and save and see if it exists now.
   If it STILL doesn't exist make a note to tell me and do the following:

   echo -e "moverDisabled=\"no\"\nmoverNice=\"0\"\nmoverIO=\"-c 2 -n 0\"\nthreshold=\"0\"\nage=\"no\"\nsizef=\"no\"\nsparsnessf=\"no\"\nfilelistf=\"no\"\nfiletypesf=\"no\"\nparity=\"yes\"\nenableTurbo=\"no\"\nlogging=\"no\"\nforce=\"no\"\nignoreHidden=\"no\"\nbeforeScript=\"\"\nafterScript=\"\"\nomovercfg=\"no\"\nmovenow=\"yes\"\ntestmode=\"no\"" >> /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg

   Feel free to change the settings back to what you want and now run the mover again.
   If it works now, please let me know.

5. Check to see if all your user shares exist in "/boot/config/shares"

     Reveal hidden contents

     

   ls "/boot/config/shares"

   All your shares should exist here with the exact same name they have in your "Shares" tab (including capital letters). If the name isn't there or isn't the same, try renaming completely and saving, then renaming back.
   Run the mover. If it works now, please let me know.

6. If all else hasn't worked please try make changes to the "mover.php" file.

     Reveal hidden contents

     

   nano "/usr/local/emhttp/plugins/ca.mover.tuning/mover.php"

   Find the following lines (line 162/163): 

   logger("ionice $ioLevel nice -n $niceLevel /usr/local/emhttp/plugins/ca.mover.tuning/age_mover $options");
   passthru("ionice $ioLevel nice -n $niceLevel /usr/local/emhttp/plugins/ca.mover.tuning/age_mover $options");

   and replace with the following:

   logger("ionice $ioLevel nice -n $niceLevel /usr/local/sbin/mover.old $options");
   passthru("ionice $ioLevel nice -n $niceLevel /usr/local/sbin/mover.old $options");

   Once done, save the file with "ctrl+o" "enter" and then "ctrl+x" to exit.
   Try running the mover again. In this case, I would like for you to try run the mover twice, once with "step 2" setting disabled and once with it enabled.
   It is important to know that these changes will revert when your server restarts.
   If this now works, please let me know and which one or both it worked for.

If you run through these steps and it is WORKING please add the following to your starting template:

If you run through all the steps and the is is still NOT WORKING, please add the following to your starting template:

 

Thanks guys, doing these steps and reporting back to us will really help us figure out what exactly is wrong. None of these issues are occurring for me personally so it is very hard to troubleshoot.

I just wanted to say that #5 on this list was the culprit for me. There were a lot of old share configurations that the mover was getting hung up on, therefore never running. After a clean up, the move now worked with the proper filters in place.

Thank you for this checklist.

Edited December 6, 2023 by dyedfire

[xokia]

Issue I ran into with mover tuning.

 

When all my disks are spinning and nothing is really going on I will click "move now" to manually move files off my cache drive.

 

I have "move now follows plug-in filter" set to no

 

normally if I do not move thing manually I have it set to move things when the cache drive reaches 60%

 

However when I clicked "move now" nothing was happening.

I had to move the threshold to 0% then click "move now" to get things moving

 

So it would appear the tool isn't following the rule move now follows plug-in filter = no

[Gunny]

Something I ran into today was that "Comma seperated list of file types:" doesn't match just the end of the filepath (i.e. just the extension/type) it also matches anywhere in the file path.  So if you are trying to not move a file that ends in .part, that will also match files with .part elsewhere in the path.  e.g. /mnt/user/tv/show1/s1/episode1.part1.mkv and /mnt/user/tv/the.partition.showname/s1/episode1.mkv will match.

Edited December 7, 2023 by Gunny