[Plugin] Mover Tuning


Recommended Posts

New Release

 

2020.06.03

- Added size of all files from find string per share to test mode output.

- Added "Ignore Hidden Files" option. This adds -not -path '/.' to the end of the find command. Requested by slimjim181 to help with video editing folders.

Link to comment
  • 2 weeks later...

Hey quick question. Am I understanding the difference between these two values correctly?

"Only move at this threshold of used cache space" - I have this set to 70%, so that when the mover is pinged nightly, it will only act if the cache is over 70% full.

"Move All from Cache-yes shares pool percentage" - I have this set to 90%. I thought this was more of an immediate move. If at any time during the day, the cache passes 90%, the mover is invoked.

 

I think I'm misunderstanding the second one, or the difference between the two, because I just passed 90% and nothing was happening.

Link to comment
24 minutes ago, stev067 said:

Hey quick question. Am I understanding the difference between these two values correctly?

"Only move at this threshold of used cache space" - I have this set to 70%, so that when the mover is pinged nightly, it will only act if the cache is over 70% full.

"Move All from Cache-yes shares pool percentage" - I have this set to 90%. I thought this was more of an immediate move. If at any time during the day, the cache passes 90%, the mover is invoked.

 

I think I'm misunderstanding the second one, or the difference between the two, because I just passed 90% and nothing was happening.

 

1.) Correct, the first thing mover does is check the percentage of the cache used.  When it's run, if used space is >70% (your setting) it will then run through the rest of the filters and send the "find" string to the mover binary.

2.)Mover is only kicked off based on the schedule.

Say you have Mover set to hourly, and the cache hit's 95% sometime in that hour, it won't move until the set schedule time. (i.e. hourly, or daily, etc) 

  • Like 1
Link to comment
5 minutes ago, hugenbdd said:

 

1.) Correct, the first thing mover does is check the percentage of the cache used.  When it's run, if used space is >70% (your setting) it will then run through the rest of the filters and send the "find" string to the mover binary.

2.)Mover is only kicked off based on the schedule.

Say you have Mover set to hourly, and the cache hit's 95% sometime in that hour, it won't move until the set schedule time. (i.e. hourly, or daily, etc) 

Ok so they are the same, except that #2 does not include filters? And if I'm not using any filters, then they're just identical?

Link to comment
3 hours ago, stev067 said:

Ok so they are the same, except that #2 does not include filters? And if I'm not using any filters, then they're just identical?

Pretty much.

I'm away from my server to check the code... but... if you have multiple cache pools, then it will also move those even if that other cache pool is not at 90% (Your setting).  i.e. I call the original Mover bash script.

  • Like 1
Link to comment

I've been racking my brain to come up with a way to have the plugin automatically start the mover if the cache is filled up over a certain threshold (say 90%), but also run once a day normally at a certain time.

 

I don't want to take the performance hit to run it every hour if it's not needed, but I'm willing to take the hit if it's getting full. I know the plugin won't automatically detect if it's over 90% full, so I'm running it hourly if it's over a certain threshold and have the option to Cron Schedule to force move all of files at 6 am. The problem is that the hourly move never activates, even if the cache is full. Is it possible to do what I want or am I completely misunderstanding what this plugin does?

Link to comment
16 minutes ago, crazygambit said:

I've been racking my brain to come up with a way to have the plugin automatically start the mover if the cache is filled up over a certain threshold (say 90%), but also run once a day normally at a certain time.

 

I don't want to take the performance hit to run it every hour if it's not needed, but I'm willing to take the hit if it's getting full. I know the plugin won't automatically detect if it's over 90% full, so I'm running it hourly if it's over a certain threshold and have the option to Cron Schedule to force move all of files at 6 am. The problem is that the hourly move never activates, even if the cache is full. Is it possible to do what I want or am I completely misunderstanding what this plugin does?

The hourly schedule should be going off.  Can you enable logging, post a screen shot of your settings and the mover entries in the syslog?

 

For it to "automatically" move files based on percentage, you could setup a custom script that runs every minute and checks the size of the cache pool and then kicks off mover.  you could take look at some of my code for examples of how to find the size.

Link to comment
  • 2 weeks later...
24 minutes ago, metabubble said:

Is there any way you can include a "run script after every moved file" or "run command after every moved file"?

 

ideally it would also send the last moved file as first argument to the script.

The way it's currently scripted, no.  It pipe's in the find command to Unraid's binary mover.

 

In the future?  Possibly, there are some changes to Unraid that I'm waiting to be implemented that will track the mover's progress.  Once that is implemented I may be able to do something like this.  It's also possible a "tee" command could be used, but I don't think that will satisfy the "after moved" requirement you mentioned.

 

Curios, what is the use case to run a script after every file?

Link to comment
On 6/26/2021 at 1:37 AM, hugenbdd said:

The way it's currently scripted, no.  It pipe's in the find command to Unraid's binary mover.

 

In the future?  Possibly, there are some changes to Unraid that I'm waiting to be implemented that will track the mover's progress.  Once that is implemented I may be able to do something like this.  It's also possible a "tee" command could be used, but I don't think that will satisfy the "after moved" requirement you mentioned.

 

Curios, what is the use case to run a script after every file?

I am trying to debug IO starvation whenever the mover is running. Even with nice and ionice to the lowest prio, as soon as the mover runs, smb craps out, plex will not serve any media, basically what everyone is complaining. I was wanting to see if running "sync" (or sync; sleep 1) after every file (causing it to pause till the cache is flushed) will improve the situation by giving plex a little bit of IO time.

 

I have narrowed the problem down to the mover just writing to write cache, which becomes saturated and then it is written out as a blocking operation. A hook into the mover would give me control to pause it briefly whenever reads occur, so I could readahead into the read buffer...

Link to comment
51 minutes ago, metabubble said:

I am trying to debug IO starvation whenever the mover is running. Even with nice and ionice to the lowest prio, as soon as the mover runs, smb craps out, plex will not serve any media, basically what everyone is complaining. I was wanting to see if running "sync" (or sync; sleep 1) after every file (causing it to pause till the cache is flushed) will improve the situation by giving plex a little bit of IO time.

 

I have narrowed the problem down to the mover just writing to write cache, which becomes saturated and then it is written out as a blocking operation. A hook into the mover would give me control to pause it briefly whenever reads occur, so I could readahead into the read buffer...

okay, that makes sense.

 

You could write a custom script which loops through a list of files and sends each file to the mover to see if it solves the issue.

 

Once the updates for the gui/mover progress come out, I should be able to accomplish something like this also. (i.e. a wait between files). 

Link to comment
  • 2 weeks later...
Posted (edited)

First, I wanted to say thank you for this awesome plugin.

 

Next, I wanted to ask if there was a way to use the "Ignore files listed inside of a text file:" feature to ignore an entire directory instead of just a couple specific files? If not, is this possible in some other way?

 

Thank you!

Edited by IMI4tth3w
Link to comment
11 minutes ago, IMI4tth3w said:

First, I wanted to say thank you for this awesome plugin.

 

Next, I wanted to ask if there was a way to use the "Ignore files listed inside of a text file:" feature to ignore an entire directory instead of just a couple specific files? If not, is this possible in some other way?

 

Thank you!

I think you can do this.

It has been a while since I programmed it and can't review the code right now.  Will look at it tomorrow.

 

However, you can play with this pretty easily. (without actually moving anything.)

Create the text file, add the path you don't want moved.  Then change two other settings.

 

Move Now button follows plug-in filters:   Yes

Test Mode: Yes

 

Hit the move now button and then review the sys log (upper right, 2nd most right button.)

 

If it does what you want, change the test mode back to "no".

Link to comment
8 minutes ago, hugenbdd said:

I think you can do this.

It has been a while since I programmed it and can't review the code right now.  Will look at it tomorrow.

 

However, you can play with this pretty easily. (without actually moving anything.)

Create the text file, add the path you don't want moved.  Then change two other settings.

 

Move Now button follows plug-in filters:   Yes

Test Mode: Yes

 

Hit the move now button and then review the sys log (upper right, 2nd most right button.)

 

If it does what you want, change the test mode back to "no".

 

Thanks for the quick reply.

This is exactly what I am doing currently but I don't believe it is working how I intend, but it also might not be working at all.

 

Here's a snippet of my syslog

Jul  8 20:51:00 UNRAID root: mvlogger: Adding Skip File List
Jul  8 20:51:00 UNRAID root: mvlogger: Skip File List Path: /mnt/user/data/mover-ignore.txt
Jul  8 20:51:00 UNRAID root: mvlogger: Skipfiletypes string: find "/mnt/cache/data" -depth | grep -vFf '/mnt/user/data/mover-ignore.txt'
Jul  8 20:51:00 UNRAID root: mvlogger: Expected Size of all files for this cache to be moved:
Jul  8 20:51:00 UNRAID root: mvlogger: find "/mnt/cache/data" -depth -type f -print0 | grep -vFf '/mnt/user/data/mover-ignore.txt' | du -shc --files0-from - | tail -n1
Jul  8 20:51:00 UNRAID root: grep: (standard input): binary file matches
Jul  8 20:51:00 UNRAID root: 0#011total
Jul  8 20:51:00 UNRAID root: mvlogger: Running in Test Mode: find "/mnt/cache/data" -depth | grep -vFf '/mnt/user/data/mover-ignore.txt'

 

after the last line it outputs the lists of files that it is assumingly going to move. Included in this list is all the files in the directory i specified in the mover-ignore.txt as well as a file i specified just as a test.

 

I'll keep digging and let you know what I find.
 

Thank you

Link to comment
7 minutes ago, IMI4tth3w said:

 

Thanks for the quick reply.

This is exactly what I am doing currently but I don't believe it is working how I intend, but it also might not be working at all.

 

Here's a snippet of my syslog


Jul  8 20:51:00 UNRAID root: mvlogger: Adding Skip File List
Jul  8 20:51:00 UNRAID root: mvlogger: Skip File List Path: /mnt/user/data/mover-ignore.txt
Jul  8 20:51:00 UNRAID root: mvlogger: Skipfiletypes string: find "/mnt/cache/data" -depth | grep -vFf '/mnt/user/data/mover-ignore.txt'
Jul  8 20:51:00 UNRAID root: mvlogger: Expected Size of all files for this cache to be moved:
Jul  8 20:51:00 UNRAID root: mvlogger: find "/mnt/cache/data" -depth -type f -print0 | grep -vFf '/mnt/user/data/mover-ignore.txt' | du -shc --files0-from - | tail -n1
Jul  8 20:51:00 UNRAID root: grep: (standard input): binary file matches
Jul  8 20:51:00 UNRAID root: 0#011total
Jul  8 20:51:00 UNRAID root: mvlogger: Running in Test Mode: find "/mnt/cache/data" -depth | grep -vFf '/mnt/user/data/mover-ignore.txt'

 

after the last line it outputs the lists of files that it is assumingly going to move. Included in this list is all the files in the directory i specified in the mover-ignore.txt as well as a file i specified just as a test.

 

I'll keep digging and let you know what I find.
 

Thank you

Yup that’s the right find command.

 

if you look further back in the thread someone posted a good explanation of how to use it.

 

I’ll still review tomorrow.

Link to comment
51 minutes ago, hugenbdd said:

Yup that’s the right find command.

 

if you look further back in the thread someone posted a good explanation of how to use it.

 

I’ll still review tomorrow.

 

Figured it out.

 

My mover-ignore.txt had the wrong directories in it..

 

old

/mnt/user/data/downloads/

 

fixed

/mnt/cache/data/downloads/

 

replaced user with cache...

 

dumb mistake. But i did find out that adding the * to the end of the directory is not needed. grep is effectively looking at the strings of the file name looking for matches. obviously "/mnt/user" wasn't going to match to anything that is "/mnt/cache"

 

thank you again for the help!

  • Like 1
Link to comment
  • 2 weeks later...
10 minutes ago, DZMM said:

Hi does the plugin support multiple pools yet, or still only shares that use a pool called "cache"?

 

Thanks in advance

 

 

 

 

 

 

Supports multiple pools.  However, settings are applied to all pools.

Link to comment
On 4/15/2021 at 12:50 PM, hugenbdd said:

Saw a few posts on Reddit with cache dealing with locked files, or downloaders getting slowed down during move.

 

Would it be helpful if I added two new sections to the tuner?

 

- Script to run before mover starts 

 

- Script to run when mover ends

 

I'm thinking this would allow people to shut down dockers or pause downloaders.

This will probably run before checking any of the "bounds".  i.e. if a pool cache % used is hit or not.

 

Thoughts?

 

How about a way to exclude a folder from being moved? I have a usenet directory under my data share and a media directory also under data. I obviously want the mover to run on media directory but would like to be able to exclude the usenet directory.

Link to comment
7 hours ago, CriticalMach said:

 

How about a way to exclude a folder from being moved? I have a usenet directory under my data share and a media directory also under data. I obviously want the mover to run on media directory but would like to be able to exclude the usenet directory.

See the post a few up.

 

Use the setting

Ignore files listed inside of a text file:

Then enter the path to your file containing the path to the directory you don't want moved.  (i.e. must be /mnt/cache/ or your cache pool path, not the share path.)

 

Link to comment
1 hour ago, hugenbdd said:

See the post a few up.

 

Use the setting

Ignore files listed inside of a text file:

Then enter the path to your file containing the path to the directory you don't want moved.  (i.e. must be /mnt/cache/ or your cache pool path, not the share path.)

 

 

Gotcha thanks, I saw that right after I posted and finally got it working. The tooltip was throwing me off, maybe a slight tweak in the ui to "Full path to a file that contains a list of files/directories you want ignored from being moved off the cache pool." Thanks for the help!

Link to comment
4 hours ago, CriticalMach said:

 

Gotcha thanks, I saw that right after I posted and finally got it working. The tooltip was throwing me off, maybe a slight tweak in the ui to "Full path to a file that contains a list of files/directories you want ignored from being moved off the cache pool." Thanks for the help!

noted, will include in the next update.

Link to comment
  • 2 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.