[Plugin] Mover Tuning

[hugenbdd]

New Release

 

2020.06.03

- Added size of all files from find string per share to test mode output.

- Added "Ignore Hidden Files" option. This adds -not -path '/.' to the end of the find command. Requested by slimjim181 to help with video editing folders.

[stev067]

Hey quick question. Am I understanding the difference between these two values correctly?

"Only move at this threshold of used cache space" - I have this set to 70%, so that when the mover is pinged nightly, it will only act if the cache is over 70% full.

"Move All from Cache-yes shares pool percentage" - I have this set to 90%. I thought this was more of an immediate move. If at any time during the day, the cache passes 90%, the mover is invoked.

 

I think I'm misunderstanding the second one, or the difference between the two, because I just passed 90% and nothing was happening.

[hugenbdd]

24 minutes ago, stev067 said:

Hey quick question. Am I understanding the difference between these two values correctly?

"Only move at this threshold of used cache space" - I have this set to 70%, so that when the mover is pinged nightly, it will only act if the cache is over 70% full.

"Move All from Cache-yes shares pool percentage" - I have this set to 90%. I thought this was more of an immediate move. If at any time during the day, the cache passes 90%, the mover is invoked.

 

I think I'm misunderstanding the second one, or the difference between the two, because I just passed 90% and nothing was happening.

 

1.) Correct, the first thing mover does is check the percentage of the cache used.  When it's run, if used space is >70% (your setting) it will then run through the rest of the filters and send the "find" string to the mover binary.

2.)Mover is only kicked off based on the schedule.

Say you have Mover set to hourly, and the cache hit's 95% sometime in that hour, it won't move until the set schedule time. (i.e. hourly, or daily, etc) 

[stev067]

5 minutes ago, hugenbdd said:

 

1.) Correct, the first thing mover does is check the percentage of the cache used.  When it's run, if used space is >70% (your setting) it will then run through the rest of the filters and send the "find" string to the mover binary.

2.)Mover is only kicked off based on the schedule.

Say you have Mover set to hourly, and the cache hit's 95% sometime in that hour, it won't move until the set schedule time. (i.e. hourly, or daily, etc) 

Ok so they are the same, except that #2 does not include filters? And if I'm not using any filters, then they're just identical?

[hugenbdd]

3 hours ago, stev067 said:

Ok so they are the same, except that #2 does not include filters? And if I'm not using any filters, then they're just identical?

Pretty much.

I'm away from my server to check the code... but... if you have multiple cache pools, then it will also move those even if that other cache pool is not at 90% (Your setting).  i.e. I call the original Mover bash script.

[crazygambit]

I've been racking my brain to come up with a way to have the plugin automatically start the mover if the cache is filled up over a certain threshold (say 90%), but also run once a day normally at a certain time.

 

I don't want to take the performance hit to run it every hour if it's not needed, but I'm willing to take the hit if it's getting full. I know the plugin won't automatically detect if it's over 90% full, so I'm running it hourly if it's over a certain threshold and have the option to Cron Schedule to force move all of files at 6 am. The problem is that the hourly move never activates, even if the cache is full. Is it possible to do what I want or am I completely misunderstanding what this plugin does?

[hugenbdd]

16 minutes ago, crazygambit said:

I've been racking my brain to come up with a way to have the plugin automatically start the mover if the cache is filled up over a certain threshold (say 90%), but also run once a day normally at a certain time.

 

I don't want to take the performance hit to run it every hour if it's not needed, but I'm willing to take the hit if it's getting full. I know the plugin won't automatically detect if it's over 90% full, so I'm running it hourly if it's over a certain threshold and have the option to Cron Schedule to force move all of files at 6 am. The problem is that the hourly move never activates, even if the cache is full. Is it possible to do what I want or am I completely misunderstanding what this plugin does?

The hourly schedule should be going off.  Can you enable logging, post a screen shot of your settings and the mover entries in the syslog?

 

For it to "automatically" move files based on percentage, you could setup a custom script that runs every minute and checks the size of the cache pool and then kicks off mover.  you could take look at some of my code for examples of how to find the size.

[metabubble]

Is there any way you can include a "run script after every moved file" or "run command after every moved file"?

 

ideally it would also send the last moved file as first argument to the script.

[hugenbdd]

24 minutes ago, metabubble said:

Is there any way you can include a "run script after every moved file" or "run command after every moved file"?

 

ideally it would also send the last moved file as first argument to the script.

The way it's currently scripted, no.  It pipe's in the find command to Unraid's binary mover.

 

In the future?  Possibly, there are some changes to Unraid that I'm waiting to be implemented that will track the mover's progress.  Once that is implemented I may be able to do something like this.  It's also possible a "tee" command could be used, but I don't think that will satisfy the "after moved" requirement you mentioned.

 

Curios, what is the use case to run a script after every file?

[metabubble]

On 6/26/2021 at 1:37 AM, hugenbdd said:

The way it's currently scripted, no.  It pipe's in the find command to Unraid's binary mover.

 

In the future?  Possibly, there are some changes to Unraid that I'm waiting to be implemented that will track the mover's progress.  Once that is implemented I may be able to do something like this.  It's also possible a "tee" command could be used, but I don't think that will satisfy the "after moved" requirement you mentioned.

 

Curios, what is the use case to run a script after every file?

I am trying to debug IO starvation whenever the mover is running. Even with nice and ionice to the lowest prio, as soon as the mover runs, smb craps out, plex will not serve any media, basically what everyone is complaining. I was wanting to see if running "sync" (or sync; sleep 1) after every file (causing it to pause till the cache is flushed) will improve the situation by giving plex a little bit of IO time.

 

I have narrowed the problem down to the mover just writing to write cache, which becomes saturated and then it is written out as a blocking operation. A hook into the mover would give me control to pause it briefly whenever reads occur, so I could readahead into the read buffer...

[hugenbdd]

51 minutes ago, metabubble said:

I am trying to debug IO starvation whenever the mover is running. Even with nice and ionice to the lowest prio, as soon as the mover runs, smb craps out, plex will not serve any media, basically what everyone is complaining. I was wanting to see if running "sync" (or sync; sleep 1) after every file (causing it to pause till the cache is flushed) will improve the situation by giving plex a little bit of IO time.

 

I have narrowed the problem down to the mover just writing to write cache, which becomes saturated and then it is written out as a blocking operation. A hook into the mover would give me control to pause it briefly whenever reads occur, so I could readahead into the read buffer...

okay, that makes sense.

 

You could write a custom script which loops through a list of files and sends each file to the mover to see if it solves the issue.

 

Once the updates for the gui/mover progress come out, I should be able to accomplish something like this also. (i.e. a wait between files). 

[IMI4tth3w]

First, I wanted to say thank you for this awesome plugin.

 

Next, I wanted to ask if there was a way to use the "Ignore files listed inside of a text file:" feature to ignore an entire directory instead of just a couple specific files? If not, is this possible in some other way?

 

Thank you!

Edited July 9, 2021 by IMI4tth3w

[hugenbdd]

11 minutes ago, IMI4tth3w said:

First, I wanted to say thank you for this awesome plugin.

 

Next, I wanted to ask if there was a way to use the "Ignore files listed inside of a text file:" feature to ignore an entire directory instead of just a couple specific files? If not, is this possible in some other way?

 

Thank you!

I think you can do this.

It has been a while since I programmed it and can't review the code right now.  Will look at it tomorrow.

 

However, you can play with this pretty easily. (without actually moving anything.)

Create the text file, add the path you don't want moved.  Then change two other settings.

 

Move Now button follows plug-in filters:   Yes

Test Mode: Yes

 

Hit the move now button and then review the sys log (upper right, 2nd most right button.)

 

If it does what you want, change the test mode back to "no".

[IMI4tth3w]

8 minutes ago, hugenbdd said:

I think you can do this.

It has been a while since I programmed it and can't review the code right now.  Will look at it tomorrow.

 

However, you can play with this pretty easily. (without actually moving anything.)

Create the text file, add the path you don't want moved.  Then change two other settings.

 

Move Now button follows plug-in filters:   Yes

Test Mode: Yes

 

Hit the move now button and then review the sys log (upper right, 2nd most right button.)

 

If it does what you want, change the test mode back to "no".

 

Thanks for the quick reply.

This is exactly what I am doing currently but I don't believe it is working how I intend, but it also might not be working at all.

 

Here's a snippet of my syslog

Jul  8 20:51:00 UNRAID root: mvlogger: Adding Skip File List
Jul  8 20:51:00 UNRAID root: mvlogger: Skip File List Path: /mnt/user/data/mover-ignore.txt
Jul  8 20:51:00 UNRAID root: mvlogger: Skipfiletypes string: find "/mnt/cache/data" -depth | grep -vFf '/mnt/user/data/mover-ignore.txt'
Jul  8 20:51:00 UNRAID root: mvlogger: Expected Size of all files for this cache to be moved:
Jul  8 20:51:00 UNRAID root: mvlogger: find "/mnt/cache/data" -depth -type f -print0 | grep -vFf '/mnt/user/data/mover-ignore.txt' | du -shc --files0-from - | tail -n1
Jul  8 20:51:00 UNRAID root: grep: (standard input): binary file matches
Jul  8 20:51:00 UNRAID root: 0#011total
Jul  8 20:51:00 UNRAID root: mvlogger: Running in Test Mode: find "/mnt/cache/data" -depth | grep -vFf '/mnt/user/data/mover-ignore.txt'

 

after the last line it outputs the lists of files that it is assumingly going to move. Included in this list is all the files in the directory i specified in the mover-ignore.txt as well as a file i specified just as a test.

 

I'll keep digging and let you know what I find.
 

Thank you

[hugenbdd]

7 minutes ago, IMI4tth3w said:

 

Thanks for the quick reply.

This is exactly what I am doing currently but I don't believe it is working how I intend, but it also might not be working at all.

 

Here's a snippet of my syslog

Jul  8 20:51:00 UNRAID root: mvlogger: Adding Skip File List
Jul  8 20:51:00 UNRAID root: mvlogger: Skip File List Path: /mnt/user/data/mover-ignore.txt
Jul  8 20:51:00 UNRAID root: mvlogger: Skipfiletypes string: find "/mnt/cache/data" -depth | grep -vFf '/mnt/user/data/mover-ignore.txt'
Jul  8 20:51:00 UNRAID root: mvlogger: Expected Size of all files for this cache to be moved:
Jul  8 20:51:00 UNRAID root: mvlogger: find "/mnt/cache/data" -depth -type f -print0 | grep -vFf '/mnt/user/data/mover-ignore.txt' | du -shc --files0-from - | tail -n1
Jul  8 20:51:00 UNRAID root: grep: (standard input): binary file matches
Jul  8 20:51:00 UNRAID root: 0#011total
Jul  8 20:51:00 UNRAID root: mvlogger: Running in Test Mode: find "/mnt/cache/data" -depth | grep -vFf '/mnt/user/data/mover-ignore.txt'

 

after the last line it outputs the lists of files that it is assumingly going to move. Included in this list is all the files in the directory i specified in the mover-ignore.txt as well as a file i specified just as a test.

 

I'll keep digging and let you know what I find.
 

Thank you

Yup that’s the right find command.

 

if you look further back in the thread someone posted a good explanation of how to use it.

 

I’ll still review tomorrow.

[IMI4tth3w]

51 minutes ago, hugenbdd said:

Yup that’s the right find command.

 

if you look further back in the thread someone posted a good explanation of how to use it.

 

I’ll still review tomorrow.

 

Figured it out.

 

My mover-ignore.txt had the wrong directories in it..

 

old

/mnt/user/data/downloads/

 

fixed

/mnt/cache/data/downloads/

 

replaced user with cache...

 

dumb mistake. But i did find out that adding the * to the end of the directory is not needed. grep is effectively looking at the strings of the file name looking for matches. obviously "/mnt/user" wasn't going to match to anything that is "/mnt/cache"

 

thank you again for the help!

[DZMM]

Hi does the plugin support multiple pools yet, or still only shares that use a pool called "cache"?

 

Thanks in advance

 

 

 

 

 

 

[hugenbdd]

10 minutes ago, DZMM said:

Hi does the plugin support multiple pools yet, or still only shares that use a pool called "cache"?

 

Thanks in advance

 

 

 

 

 

 

Supports multiple pools.  However, settings are applied to all pools.

[CriticalMach]

On 4/15/2021 at 12:50 PM, hugenbdd said:

Saw a few posts on Reddit with cache dealing with locked files, or downloaders getting slowed down during move.

 

Would it be helpful if I added two new sections to the tuner?

 

- Script to run before mover starts 

 

- Script to run when mover ends

 

I'm thinking this would allow people to shut down dockers or pause downloaders.

This will probably run before checking any of the "bounds".  i.e. if a pool cache % used is hit or not.

 

Thoughts?

 

How about a way to exclude a folder from being moved? I have a usenet directory under my data share and a media directory also under data. I obviously want the mover to run on media directory but would like to be able to exclude the usenet directory.

[hugenbdd]

7 hours ago, CriticalMach said:

 

How about a way to exclude a folder from being moved? I have a usenet directory under my data share and a media directory also under data. I obviously want the mover to run on media directory but would like to be able to exclude the usenet directory.

See the post a few up.

 

Use the setting

Ignore files listed inside of a text file:

Then enter the path to your file containing the path to the directory you don't want moved.  (i.e. must be /mnt/cache/ or your cache pool path, not the share path.)

 

[CriticalMach]

1 hour ago, hugenbdd said:

See the post a few up.

 

Use the setting

Ignore files listed inside of a text file:

Then enter the path to your file containing the path to the directory you don't want moved.  (i.e. must be /mnt/cache/ or your cache pool path, not the share path.)

 

 

Gotcha thanks, I saw that right after I posted and finally got it working. The tooltip was throwing me off, maybe a slight tweak in the ui to "Full path to a file that contains a list of files/directories you want ignored from being moved off the cache pool." Thanks for the help!

[hugenbdd]

4 hours ago, CriticalMach said:

 

Gotcha thanks, I saw that right after I posted and finally got it working. The tooltip was throwing me off, maybe a slight tweak in the ui to "Full path to a file that contains a list of files/directories you want ignored from being moved off the cache pool." Thanks for the help!

noted, will include in the next update.

[GoChris]

Sorry if this has been asked/mentioned.

Can there be an option to move files for a share for which the setting "Use cache pool" is set to no?

[JonathanM]

7 minutes ago, GoChris said:

Sorry if this has been asked/mentioned.

Can there be an option to move files for a share for which the setting "Use cache pool" is set to no?

From where to where?

[GoChris]

1 minute ago, jonathanm said:

From where to where?

From a cache device to the array.