[Plugin] Mover Tuning


Recommended Posts

Not sure if this has been reported before but I'm seeing these errors whenever my mover runs
 

Apr 12 06:00:01 Blackbox root: mvlogger: Pool /mnt/cache is above 'move all' percentage, moving all cache-yes shares to array
Apr 12 06:00:01 Blackbox root: mvlogger: Not in Test mode
Apr 12 06:00:01 Blackbox root: mvlogger: Complete Mover Command: find /mnt/cache/TV Shows -depth | /usr/local/sbin/move -d 1
Apr 12 06:00:01 Blackbox root: find: '/mnt/cache/TV': No such file or directory
Apr 12 06:00:01 Blackbox root: find: 'Shows': No such file or directory
Apr 12 06:00:01 Blackbox root: mvlogger: Done checking Move All from Share
Apr 12 06:00:01 Blackbox root: mvlogger: Share Name Only: TV Shows
Apr 12 06:00:01 Blackbox root: mvlogger: Cache Pool Name:
Apr 12 06:00:01 Blackbox root: mvlogger: No shareCachePool entry found in config file, defaulting to cache
Apr 12 06:00:01 Blackbox root: mvlogger: cache Threshold Pct:
Apr 12 06:00:01 Blackbox root: mvlogger: OVERALL Threshold: 0
Apr 12 06:00:01 Blackbox root: mvlogger: Share Path: /mnt/cache/TV Shows
Apr 12 06:00:01 Blackbox root: mvlogger: Pool Pct Used: 79 %
Apr 12 06:00:01 Blackbox root: mvlogger: DFTPCT LIMIT USED FOR SETTING: 0
Apr 12 06:00:01 Blackbox root: mvlogger: Threshold Used: 0
Apr 12 06:00:01 Blackbox root: mvlogger: Adding Age
Apr 12 06:00:01 Blackbox root: mvlogger: Age 7
Apr 12 06:00:01 Blackbox root: mvlogger: Skipfiletypes string: find "/mnt/cache/TV Shows" -depth -mtime +6
Apr 12 06:00:01 Blackbox root: mvlogger: Not in Test mode
Apr 12 06:00:01 Blackbox root: mvlogger: Complete Mover Command: find "/mnt/cache/TV Shows" -depth -mtime +6 | /usr/local/sbin/move -d 1

 

It looks like the space in my `TV Shows` share is causing issues with the path. I'm not sure if that's a bug in the original mover script or this plugin.

Link to comment
On 4/12/2022 at 9:20 AM, Moussa said:

Not sure if this has been reported before but I'm seeing these errors whenever my mover runs
 

Apr 12 06:00:01 Blackbox root: mvlogger: Pool /mnt/cache is above 'move all' percentage, moving all cache-yes shares to array
Apr 12 06:00:01 Blackbox root: mvlogger: Not in Test mode
Apr 12 06:00:01 Blackbox root: mvlogger: Complete Mover Command: find /mnt/cache/TV Shows -depth | /usr/local/sbin/move -d 1
Apr 12 06:00:01 Blackbox root: find: '/mnt/cache/TV': No such file or directory
Apr 12 06:00:01 Blackbox root: find: 'Shows': No such file or directory
Apr 12 06:00:01 Blackbox root: mvlogger: Done checking Move All from Share
Apr 12 06:00:01 Blackbox root: mvlogger: Share Name Only: TV Shows
Apr 12 06:00:01 Blackbox root: mvlogger: Cache Pool Name:
Apr 12 06:00:01 Blackbox root: mvlogger: No shareCachePool entry found in config file, defaulting to cache
Apr 12 06:00:01 Blackbox root: mvlogger: cache Threshold Pct:
Apr 12 06:00:01 Blackbox root: mvlogger: OVERALL Threshold: 0
Apr 12 06:00:01 Blackbox root: mvlogger: Share Path: /mnt/cache/TV Shows
Apr 12 06:00:01 Blackbox root: mvlogger: Pool Pct Used: 79 %
Apr 12 06:00:01 Blackbox root: mvlogger: DFTPCT LIMIT USED FOR SETTING: 0
Apr 12 06:00:01 Blackbox root: mvlogger: Threshold Used: 0
Apr 12 06:00:01 Blackbox root: mvlogger: Adding Age
Apr 12 06:00:01 Blackbox root: mvlogger: Age 7
Apr 12 06:00:01 Blackbox root: mvlogger: Skipfiletypes string: find "/mnt/cache/TV Shows" -depth -mtime +6
Apr 12 06:00:01 Blackbox root: mvlogger: Not in Test mode
Apr 12 06:00:01 Blackbox root: mvlogger: Complete Mover Command: find "/mnt/cache/TV Shows" -depth -mtime +6 | /usr/local/sbin/move -d 1

 

It looks like the space in my `TV Shows` share is causing issues with the path. I'm not sure if that's a bug in the original mover script or this plugin.

Fixed with new release.  It now quotes the share name in the "Move All" code.

Link to comment
  • 2 weeks later...

Running in a bit of a challenge here, where mover only seems to move a certain percentage of files.

Expected operation -> cache share hits 95% usage, mover then moves all files to array. In stead it only moves roughly 20%.

on the 23rd i did a manual move(Main Menu, the Move button) which took it down to +-10%.

24th and 25th was the automaic moves once it hits 95%, but it only moved about 25%

image.png.43605762a08d88fd71b6a840c39eedbb.png

 

configuration as follows :

image.png.3f5de84df9853634abe5f1a6a8201d52.png

image.png.6cdf87ea371ffdff5253c445d2c512dc.png

 

Any ideas? should i enable mover logging and/or test mode for further troubleshooting?

 

Thx!

Link to comment
  • 2 weeks later...

It goes without saying, but I'll say it anyway: Fantastic plug-in, on my list of must-haves.

 

Feature request: Ability to exclude/include specified shares in mover tuning, vs just moving on whatever the set schedule is (as if tuning wasn't installed).

 

Use case: Newly added files go to cache and are added to Plex, and usually watched over the next few days (they're popular at the time, etc) so I like them to hang around for a few days to avoid issues around hard drive thrashing.

I have several things backing up to cache (perf reasons), which I'd like moved to array every night as they don't need to live on cache for the 5 days like everything else.

 

Unless of course I'm missing a way to do this as-is!

Edited by -Daedalus
Link to comment
3 hours ago, -Daedalus said:

It goes without saying, but I'll say it anyway: Fantastic plug-in, on my list of must-haves.

 

Feature request: Ability to exclude/include specified shares in mover tuning, vs just moving on whatever the set schedule is (as if tuning wasn't installed).

 

Use case: Newly added files go to cache and are added to Plex, and usually watched over the next few days (they're popular at the time, etc) so I like them to hang around for a few days to avoid issues around hard drive thrashing.

I have several things backing up to cache (perf reasons), which I'd like moved to array every night as they don't need to live on cache for the 5 days like everything else.

 

Unless of course I'm missing a way to do this as-is!


This is already a feature via exclude location list.

Link to comment
On 4/25/2022 at 6:52 PM, Neo_x said:

Running in a bit of a challenge here, where mover only seems to move a certain percentage of files.

Expected operation -> cache share hits 95% usage, mover then moves all files to array. In stead it only moves roughly 20%.

on the 23rd i did a manual move(Main Menu, the Move button) which took it down to +-10%.

24th and 25th was the automaic moves once it hits 95%, but it only moved about 25%

image.png.43605762a08d88fd71b6a840c39eedbb.png

 

configuration as follows :

image.png.3f5de84df9853634abe5f1a6a8201d52.png

image.png.6cdf87ea371ffdff5253c445d2c512dc.png

 

Any ideas? should i enable mover logging and/or test mode for further troubleshooting?

 

Thx!

check your share settings as you may have had something going to cache that has now been set to only use the array thus the files get left on the cache and never moved.

Correct procedure if changing your caching of shares to not use the cache anymore is always to stop what ever is feeding that share, run a full mover run  then change the share setting back to array.

 

I hope this helps

 

Link to comment
On 5/9/2022 at 11:23 PM, -Daedalus said:

 

I saw that, but the wording of the help section makes it sound like it'll just never get moved off cache:

 

image.png.076392c1050b0fd62ffdfc406ce2597f.png

this is correct it is used for keeping things on the cache (excluding from the move)

Situation example:

You have a share set to cache yes  data is read written to the cache until criteria is met for the mover to run, the mover runs and normally every bit of that data currently under that share that is on the cache is moved to the array.

Let;s say you have a bunch of sub files or folders in that share that you would like to have stay on the array when the mover runs so that applications that depend on that data can run faster using the cache.

having this option allows you to have less shares created and increase speed of some application you have used it for.

e.g
Nextcloud requires a share for user data which includes docs. thumbnails, photos etc, if you set that share to cache_yes the all the data that was once on the cache becomes very slow now especially small files after the mover runs and it gets transfered/moved to the array as things like thumbnails etc have to be then read from the array instead of the cache.

Enter this mover feature! 
Allowing you to find the thumbnail sub sub sub folder or what ever else you want and set it to stay on the cache regardless of mover run, however, all the actual pictures docs etc not specified get moved to the array still, keeping your end user experience nice and fast in the gui/webpage of nextcloud as you cached your thumbnails but allowing you to optimize your used storage of your cache by having the huge files sit in slower storage as they are not regularly accessed,

Summery is that this feature of mover allows for more:
Granular cache control
cache Space Saving
Application/docker performance
Less Mover run time
faster load times of games: if you set assets or .exe files etc to stay on cache
etc
etc



 

Link to comment

Has anyone actually done a perf test of different size files using xfs vs btrfs with mover to help speed things up.

I know that xfs yes less features has better io in general on linux, however, with the mover choking on larger amounts of small files eg kb's using btrfs I wonder has anyone actually tested move time / perf of the two file systems for the purpose of the the mover?

Link to comment
3 hours ago, IronBeardKnight said:

this is correct it is used for keeping things on the cache (excluding from the move)
[ snip ]
Summery is that this feature of mover allows for more:
Granular cache control
cache Space Saving
Application/docker performance
Less Mover run time
faster load times of games: if you set assets or .exe files etc to stay on cache
etc
etc

 

 

Got'cha, thanks for the explanation.

 

Though what I was requesting was different:

I'd like the ability to say "Move according to tuning rules for all shares except these ones. For these ones, just follow stock mover rules (daily, etc.)"

Link to comment

mmm

On 5/11/2022 at 6:50 PM, -Daedalus said:

 

Got'cha, thanks for the explanation.

 

Though what I was requesting was different:

I'd like the ability to say "Move according to tuning rules for all shares except these ones. For these ones, just follow stock mover rules (daily, etc.)"

Everything is possible with time, however, I don't believe there has been significant  need or request for something like this yet as what we have currently caters for 95% of situations, however I have come across a couple of situations where having temporary delayed moving on a file/folder level would have been good.

So to state your request another way, basically wanting a timer option that can be set per file/folder/share?

As for getting this done, the best point to start would be then to modify this current "File list path" image.png.b267cfa607a664d43bf16de0c8bf0401.png option to do what you want E.g you would just add as normal your file/ folder locations  with a (space at the end followed by 5d, 7h, 9m, 70s) this would be the time it stays on cache for when using the parent share folder with cache Yes.
 

Code changes would need to change from:
looped if (exists)

to something like: 

looped if (exists && current date < $variableFromLine).

 

Not actual code ^ but you get the hint :)


The problem with this is that the script now has to store dates and times of arrival on cache for said folder / file per line but on every subsequent child file/folder and give them individual id's and times to reference you can imagine how this now grows very fast in both compute, io, that the mover would need not to mention now a database need, etc etc, a mover database which won't be as easy as first through to implement with a bash script which is what is run for the mover is a-lot of extra coding to be required and potentially this edges on complete mover redesign.
 

I cannot see this being implemented from my point of view.

 @-Daedalus thank you for raising it here though.

New Idea!
What could potentially solve your issue and would be very cool in nearly every single case is if we where to use for example the code from the " Active-IO-Streams" and/or "Open-Files" plugins  then modify a little to then advise the mover  what files where frequently most accessed  by way of access count and time open and have the mover be bi-directional, as in taking those frequent files and moving them to the cache.
Having the option to auto add said files to the "exclude file list option" of the mover would also be great as this stops the files from being moved back so soon if your like me and have your mover run hourly or so, at this point of adding to the exclude list you could have each and every file and or folder added to the list automatically ( basically using the txt file as a db ) which would allow you to then add a blanket timeframe to each entry as well if you want for your original needs OR instead of a blanket have the timeframe auto configure based on the usage of the file or folder e.g accessed > 9 times etc 

 

The mover then becomes essentially a proper bidirectional cache and your system gets a little smarter by making what is accessed frequently available on the faster drives but again basically a mover plugin redesign.

I would be happy to help out getting something like this out the door but as this is not my plugin and my time is limited its a decision that is not up to me.

@hugenbdd not sure if your down to go this far but its all future potential and idea's

Pardon the grammer and spelling was in a rush.

Link to comment

Quick question, do you know if the move if cache hits percentage thresholds calculates that percent full correctly if I have a 3 disk cache in raid 1? I know it does the thing where if the cache is empty it incorrectly shows free space. Then as it fills up it gradually gets closer to accurately reporting the free space. Basically when the cache is actually full then it shows the correct 0 space available out of the full available amount. 

 

Does that make sense? I can grab screen shots and try to explain further if that doesn't make sense. 

 

Thank you! 

Link to comment
6 minutes ago, Rhuarc said:

Quick question, do you know if the move if cache hits percentage thresholds calculates that percent full correctly if I have a 3 disk cache in raid 1? I know it does the thing where if the cache is empty it incorrectly shows free space. Then as it fills up it gradually gets closer to accurately reporting the free space. Basically when the cache is actually full then it shows the correct 0 space available out of the full available amount. 

 

Does that make sense? I can grab screen shots and try to explain further if that doesn't make sense. 

 

Thank you! 

Here's the line of code that is used to calculate.

POOLPCTUSED=`df --output=pcent /mnt/$CACHEPOOLNAME | tail -n 1 | tr -d '%'`

 

So if the df command is wrong, then so it the script.  not much else I can go on.

 

There is a scenario where later shares might not get moved as the percentage used drops, that I plan to fix after the new release of unRAID.

Link to comment
16 hours ago, IronBeardKnight said:

mmm

Everything is possible with time, however, I don't believe there has been significant  need or request for something like this yet as what we have currently caters for 95% of situations, however I have come across a couple of situations where having temporary delayed moving on a file/folder level would have been good.

So to state your request another way, basically wanting a timer option that can be set per file/folder/share?

As for getting this done, the best point to start would be then to modify this current "File list path" image.png.b267cfa607a664d43bf16de0c8bf0401.png option to do what you want E.g you would just add as normal your file/ folder locations  with a (space at the end followed by 5d, 7h, 9m, 70s) this would be the time it stays on cache for when using the parent share folder with cache Yes.
 

Code changes would need to change from:
looped if (exists)

to something like: 

looped if (exists && current date < $variableFromLine).

 

Not actual code ^ but you get the hint :)


The problem with this is that the script now has to store dates and times of arrival on cache for said folder / file per line but on every subsequent child file/folder and give them individual id's and times to reference you can imagine how this now grows very fast in both compute, io, that the mover would need not to mention now a database need, etc etc, a mover database which won't be as easy as first through to implement with a bash script which is what is run for the mover is a-lot of extra coding to be required and potentially this edges on complete mover redesign.
 

I cannot see this being implemented from my point of view.

 @-Daedalus thank you for raising it here though.

New Idea!
What could potentially solve your issue and would be very cool in nearly every single case is if we where to use for example the code from the " Active-IO-Streams" and/or "Open-Files" plugins  then modify a little to then advise the mover  what files where frequently most accessed  by way of access count and time open and have the mover be bi-directional, as in taking those frequent files and moving them to the cache.
Having the option to auto add said files to the "exclude file list option" of the mover would also be great as this stops the files from being moved back so soon if your like me and have your mover run hourly or so, at this point of adding to the exclude list you could have each and every file and or folder added to the list automatically ( basically using the txt file as a db ) which would allow you to then add a blanket timeframe to each entry as well if you want for your original needs OR instead of a blanket have the timeframe auto configure based on the usage of the file or folder e.g accessed > 9 times etc 

 

The mover then becomes essentially a proper bidirectional cache and your system gets a little smarter by making what is accessed frequently available on the faster drives but again basically a mover plugin redesign.

I would be happy to help out getting something like this out the door but as this is not my plugin and my time is limited its a decision that is not up to me.

@hugenbdd not sure if your down to go this far but its all future potential and idea's

Pardon the grammer and spelling was in a rush.

Few things.  If you scroll back way in the thread we talk about a-time etc.  I can't remember the specific's but the way unRAID is implemented it doesn't allow us to see the last "accessed" time of files.  Would have made it very easy to move files over.

 

If, someone comes up with code that provides a filelist, I could call that and send it to mover.  i.e. off cache or onto cache.  i.e. Essentially just using a cat filelist>mover

Link to comment
6 hours ago, hugenbdd said:

Few things.  If you scroll back way in the thread we talk about a-time etc.  I can't remember the specific's but the way unRAID is implemented it doesn't allow us to see the last "accessed" time of files.  Would have made it very easy to move files over.

 

If, someone comes up with code that provides a filelist, I could call that and send it to mover.  i.e. off cache or onto cache.  i.e. Essentially just using a cat filelist>mover

If I get some time over the next few days ill give it a shot otherwise our own or derived version of (active io streams) or (open files) plugin would need to be created to keep track of accessed / last accessed files which would give you your list.

Link to comment
On 5/18/2022 at 7:38 PM, Squid said:

First thing is to check the date & time on your server

One of my friend's Unraid server showed the same error “ssl verification failure" when installing a plugin, and the server's ntp servers / date and time settings are all correct. Could you help with this? 

Version: 6.10

Thank you very much! 

Link to comment
  • 3 weeks later...

Is there a way for MOVER to move files by oldest first until x % of space is free?

 

Not by age, like any file over 9 days old, but by moving the oldest file first out of all the files in the cache pool until the cache pool has  x % of space free.

  • Upvote 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.