Jump to content
Squid

[Plugin] Mover Tuning

229 posts in this topic Last Reply

Recommended Posts

On 3/24/2020 at 11:28 AM, guythnick said:

Yes, I think should be independent of the age option, but both can be used.

Under 3MB seems to capture all the images and subtitle files that I want to keep on the cache.

 

If it is piping into the find command, I believe you really only need to add one switch to the command when the option is selected:

 

-size +x

where x would be the integer in megabytes.

Updated (2020.03.28) release.

Size can be run independently of the of the age setting.  I was not able to test the condition on it's own due to the need to keep my current files in cache, but I did test age + size, and age.  + I left some extra logging in the code for now.  Which shouldn't hurt anything at this point.

 

Let me know if you run into any issues.   

 

If I add any more commands to the "find", I think I will need a different route to program the selections.  The only other type of modification I can think of right now, is to ignore certain file types.  Such as .jpg or .gif files...  But those should be ignored based on size.....

Share this post


Link to post

Hello, I've been running this since it came out, but the last couple of days, I'm getting an email about the cron job. Not sure why. Any idea?

 

Subject: cron for user root /usr/local/sbin/mover force 2>/dev/null

 

No Size Argument Supplied
Share: /mnt/cache/Backups/
Doing Find for Age Only
Age 5
Share: /mnt/cache/Downloads/
Doing Find for Age Only
Age 5
Share: /mnt/cache/TV/
Doing Find for Age Only
Age 5
Share: /mnt/cache/appdata/
Share: /mnt/cache/domains/
Share: /mnt/cache/system/

Capture1.PNG

Share this post


Link to post
1 hour ago, jcato said:

Hello, I've been running this since it came out, but the last couple of days, I'm getting an email about the cron job. Not sure why. Any idea?

 

Subject: cron for user root /usr/local/sbin/mover force 2>/dev/null

 

No Size Argument Supplied
Share: /mnt/cache/Backups/
Doing Find for Age Only
Age 5
Share: /mnt/cache/Downloads/
Doing Find for Age Only
Age 5
Share: /mnt/cache/TV/
Doing Find for Age Only
Age 5
Share: /mnt/cache/appdata/
Share: /mnt/cache/domains/
Share: /mnt/cache/system/

 

I think it's because I left some "echo" statements in the age_mover file.

If I send you a new file can you replace it and test with results?

Share this post


Link to post

jcato

Please try this file.

/usr/local/emhttp/plugins/ca.mover.tuning/age_mover

 

permissions (755)

-rwxr-xr-x 1 root root 5302 Apr  7 09:58 age_mover*

 

If this works I'll release an update.

age_mover

Share this post


Link to post
On 4/7/2020 at 7:09 AM, hugenbdd said:

jcato

Please try this file.

/usr/local/emhttp/plugins/ca.mover.tuning/age_mover

 

permissions (755)

-rwxr-xr-x 1 root root 5302 Apr  7 09:58 age_mover*

 

If this works I'll release an update.

age_mover 5.18 kB · 2 downloads

I wonder if that is why mover is not running for me after surpassing my 65% Threshold.  I'm now at 71% usage and the mover hasn't executed.  The syslog returns;

 

Apr 8 03:40:01 ****** root: Arguments not passed in please check your settings.
Apr 8 03:40:01 ****** root: Share: /mnt/cache/appdata/
Apr 8 03:40:01 ****** root: Share: /mnt/cache/domains/
Apr 8 03:40:01 ****** root: Share: /mnt/cache/downloads/
Apr 8 03:40:01 ****** root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 81: [: : integer expression expected
Apr 8 03:40:01 ****** root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 95: [: : integer expression expected
Apr 8 03:40:01 ****** root: Arguments not passed in please check your settings.
Apr 8 03:40:01 ****** root: Share: /mnt/cache/isos/
Apr 8 03:40:01 ****** root: Share: /mnt/cache/system/
Apr 8 03:40:02 ****** root: mover: finished
 

Share this post


Link to post
44 minutes ago, chizll said:

I wonder if that is why mover is not running for me after surpassing my 65% Threshold.  I'm now at 71% usage and the mover hasn't executed.  The syslog returns;

 

Apr 8 03:40:01 ****** root: Arguments not passed in please check your settings.
Apr 8 03:40:01 ****** root: Share: /mnt/cache/appdata/
Apr 8 03:40:01 ****** root: Share: /mnt/cache/domains/
Apr 8 03:40:01 ****** root: Share: /mnt/cache/downloads/
Apr 8 03:40:01 ****** root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 81: [: : integer expression expected
Apr 8 03:40:01 ****** root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 95: [: : integer expression expected
Apr 8 03:40:01 ****** root: Arguments not passed in please check your settings.
Apr 8 03:40:01 ****** root: Share: /mnt/cache/isos/
Apr 8 03:40:01 ****** root: Share: /mnt/cache/system/
Apr 8 03:40:02 ****** root: mover: finished
 

posted an update that

1.) Removes the "echo" statements from the age_mover file.  This should prevent the email to the admin.

2.) Checked the variable being passed in from the command line.  (this should fix your issue chizll).

 

Please let me know if you still run into issues.

Share this post


Link to post
27 minutes ago, hugenbdd said:

posted an update that

1.) Removes the "echo" statements from the age_mover file.  This should prevent the email to the admin.

2.) Checked the variable being passed in from the command line.  (this should fix your issue chizll).

 

Please let me know if you still run into issues.

Thank you!

Share this post


Link to post
On 4/7/2020 at 10:09 AM, hugenbdd said:

jcato

Please try this file.

/usr/local/emhttp/plugins/ca.mover.tuning/age_mover

 

permissions (755)

-rwxr-xr-x 1 root root 5302 Apr  7 09:58 age_mover*

 

If this works I'll release an update.

Sorry for delay, but I just did this. I'm not super linux-fluent, but I think I did it right. I updated cron job to run now. I think it did, as my used space on the cache went down. I didn't get an email.

 

I also see there is an update available. I assume this includes the changes in the this file and I can update like normal?

 

Thanks

 

Share this post


Link to post
57 minutes ago, jcato said:

Sorry for delay, but I just did this. I'm not super linux-fluent, but I think I did it right. I updated cron job to run now. I think it did, as my used space on the cache went down. I didn't get an email.

 

I also see there is an update available. I assume this includes the changes in the this file and I can update like normal?

 

Thanks

 

It does.  I would recommend updating.

Share this post


Link to post
Posted (edited)

Can someone explain how the age feature works in combination with the threshold %. 

 

- Will it move, say, over 30 day old files BEFORE my threshold % is met?

- Will it not move 10 day old files, even if my threshold % IS met?

 

I haven't tried it yet, but maybe it needs an "and/or" option for each condition?

Edited by samtrois

Share this post


Link to post
1 hour ago, samtrois said:

Can someone explain how the age feature works in combination with the threshold %. 

 

- Will it move, say, over 30 day old files BEFORE my threshold % is met?

- Will it not move 10 day old files, even if my threshold % IS met?

 

I haven't tried it yet, but maybe it needs an "and/or" option for each condition?

 

Threshold is the first thing it checks.  It won't run mover at all if the threshold is not met.  i.e. your cache is at 50% but you have it set to 60% in the config.

 

It will not move 30 day old files before.. as it hasn't been "run". (see above)

 

If your threshold is met, and you have "30" for the number of days old.  Then any file older than 30 days old will be moved off.  Anything younger will be left on the cache. 

 

This does mean that you can have some really old files on the cache if you are building up the files to meet your threshold.  I would suggest setting the threshold to a very low number if you want to move files off without building them up on the cache drive.

 

Share this post


Link to post

Thanks, good to know. 

I dont mind having old files on cache, I almost exclusively use this for movies and TV, so I kindof want to keep "new" stuff on cache for quicker access.

I have ~500gb cache, mover checks hourly, and threshold currently set at 70%. So I might try setting the days to 20.


To confirm, mover will check every hour if my cache is at 70%, when it is, it will only move things older than 20days. 

 

Lastly, is there an override or something, say I download a bunch in a short period of time nothing will get moved because its so new right.??

 

Share this post


Link to post
9 minutes ago, samtrois said:

Thanks, good to know. 

I dont mind having old files on cache, I almost exclusively use this for movies and TV, so I kindof want to keep "new" stuff on cache for quicker access.

I have ~500gb cache, mover checks hourly, and threshold currently set at 70%. So I might try setting the days to 20.


To confirm, mover will check every hour if my cache is at 70%, when it is, it will only move things older than 20days. 

 

Lastly, is there an override or something, say I download a bunch in a short period of time nothing will get moved because its so new right.??

 

That's why I modified this plugin.  So that my recent shows would stay on cache.  :)

I haven't been able to test, (as I want to keep my files on the cache drive) but it might be possible to over-ride by hitting the "move now" button on the main tab.  Alternatively, you could just turn off the "age" option..

 

This tends to work better with a much larger cache drive.  I would suggest at least 1TB or larger (I have 2TB) and am able to keep about 40 days worth of files if I'm not "back filling" things.

 

Share this post


Link to post

Yeah sometimes my cache fills up if I 'backfill' or have a few large (incomplete) torrents, like games or something. Or radarr/sonarr decide to not delete the moved file for some reason (I know, another issue I should fix)

 

Regular day to day my 500gb is fine, I don't mind if stuff gets moved to the array, I just try and keep the disks spun down when possible. I think a 1-2tb is a bit overkill for most of my use. Id rather spend that money on another large drive for the array, and bump one of my older smaller drives to an unassigned device for my incomplete downloads

Share this post


Link to post

Is it possible to direct mover to transfer sparse files without expanding them to full size onto the destination disk?

 

If it's something mover just doesn't support yet, maybe an alternative could be a rule to ignore files if their sparseness value is (for e.g.) less than 0.1 or a rule that effectively avoids a 10GB file becoming a 200GB file.

 

It looks like Find has a %S directive for showing a file's sparseness:

https://www.gnu.org/software/findutils/manual/html_mono/find.html#Size-Directives

 

I've been looking into doing this myself on my own system as a sanity check before mover starts, as I've managed to lock up my system at least twice due to trying to backup VMs to the protected array, then finding the same mover task still running 24hrs later, filling the array with 0s 😅

 

Share this post


Link to post

I think this can be done.  However, as I mentioned before I will need to rewrite some of the code to handle more options.  Right now, I have basically hardcoded the "find" command for each of the options and also a "combined" find for age/size.  I need to change it so that it creates the find based on all the options. (likely with a case statement)...

 

In my brief research for sparse...  I found this site.. Which combines the output with a gawk statement.  

https://www.thegeekdiary.com/how-to-find-all-the-sparse-file-in-linux/

 

Any chance you have an example of just a find statement not combined with gawk?

Share this post


Link to post

I'm still looking to further refine it, but the following will return the filepath of any files on /mnt/cache/isos over 1M that have a sparseness value over 0.1 (or ratio of 10 to 1):

find "/mnt/cache/isos" -depth -size +1M -printf '%S:%p\0' |   awk -v RS='\0' -F : '$1 > 0.1 {sub(/^[^:]*:/, ""); print}'

Adjusted from the example here: https://unix.stackexchange.com/a/86446

 

This takes into account floating point values like 3.8147e-06 returned by %S on very tiny files.

 

If the pipe to awk is an issue, there might be ways around it, which I'm about to look into.

Share this post


Link to post
On 3/14/2020 at 2:17 PM, hugenbdd said:

That is not working for me..

Your access is 7 days before your creation/modified entries....

Finally figured this one out.

Because I modified the file after reading it. So that makes sense.

 

The problem is because it has the name "creation/modified" it sounds counter intuitive.

 

 

Share this post


Link to post

Hi Guys,

 

I know this is a little long but please hear me out and let me know if you thing this is something we should be chasing as a community stuck at home. :) 

Pretty Pretty Please with a cherry on top can we have an option that allows for exclude files and or folders from the mover via perhaps name xyz or within a regex range (self configured regex string) this would allow a long standing issue to be at least half addressed.

 

Issue: some config folders allow for faster loading or web pages and also when running dockers usually a seperate data share is created but you don't want all the data in that share in the cache as it will fill the cache too quickly.

 

Hmm a better example is Nextcloud in docker and having a separate userdata folder as storing everything in cache is not really an awesome idea. thumbnails for pictures are stored in this userdata folder along with the images themselves which causes quite a bit of delay in nextcloud reading from spinning disk.

 

The same can be said for plex or emby in some cases where folder art, meta data and images are stored with the media, by allowing the option to exclude files or folders via regex string or name you can set your entire share directory to Yes for caching and when the mover runs it will move all except the imprtant files off the cache to the array.

 

Hell even as far as particular game install files if you know what your doing you can effectively speed up your networked game directories of massive size or per game.

 

This gives the ability to improve docker efficacy astronomically while not being as risky as caching in ram.

 

Ideally it would be great if unraid had the option to select folders or files to be cached and give the same options given to the shares however this is something they where aware of a in early 2018 i think and never came to life so having this plugin skip allocated files and folders is the next best thing.

 

Is this something we can look at. 

 

I'm happy to give it a crack developing such a thing on the back of this plugin to help out. 

 

cheers guys :) :) :) 

Share this post


Link to post
33 minutes ago, IronBeardKnight said:

Hi Guys,

 

I know this is a little long but please hear me out and let me know if you thing this is something we should be chasing as a community stuck at home. :) 

Pretty Pretty Please with a cherry on top can we have an option that allows for exclude files and or folders from the mover via perhaps name xyz or within a regex range (self configured regex string) this would allow a long standing issue to be at least half addressed.

 

Issue: some config folders allow for faster loading or web pages and also when running dockers usually a seperate data share is created but you don't want all the data in that share in the cache as it will fill the cache too quickly.

 

Hmm a better example is Nextcloud in docker and having a separate userdata folder as storing everything in cache is not really an awesome idea. thumbnails for pictures are stored in this userdata folder along with the images themselves which causes quite a bit of delay in nextcloud reading from spinning disk.

 

The same can be said for plex or emby in some cases where folder art, meta data and images are stored with the media, by allowing the option to exclude files or folders via regex string or name you can set your entire share directory to Yes for caching and when the mover runs it will move all except the imprtant files off the cache to the array.

 

Hell even as far as particular game install files if you know what your doing you can effectively speed up your networked game directories of massive size or per game.

 

This gives the ability to improve docker efficacy astronomically while not being as risky as caching in ram.

 

Ideally it would be great if unraid had the option to select folders or files to be cached and give the same options given to the shares however this is something they where aware of a in early 2018 i think and never came to life so having this plugin skip allocated files and folders is the next best thing.

 

Is this something we can look at. 

 

I'm happy to give it a crack developing such a thing on the back of this plugin to help out. 

 

cheers guys :) :) :) 

I've thought about this also.

 

Found this site on how to use a regex inside of a find command.  However, it uses the "-type f" option.  And I would rather stay away from that or else all folders will stay on your cache drive (most likely empty).

https://www.cyberciti.biz/faq/find-command-exclude-ignore-files/

 

Could you provide a find command that "excludes" the files/folders you don't want to move?  It will make it easier for me to assess how to put it into the code.

 

I'm assuming this would also need to be able to be combined with the size/age options also?

 

And, fyi... I still need to recode how I create the request in the script to make multiple options easier to handle.

 

Share this post


Link to post
12 hours ago, hugenbdd said:

I've thought about this also.

 

Found this site on how to use a regex inside of a find command.  However, it uses the "-type f" option.  And I would rather stay away from that or else all folders will stay on your cache drive (most likely empty).

https://www.cyberciti.biz/faq/find-command-exclude-ignore-files/

 

Could you provide a find command that "excludes" the files/folders you don't want to move?  It will make it easier for me to assess how to put it into the code.

 

I'm assuming this would also need to be able to be combined with the size/age options also?

 

And, fyi... I still need to recode how I create the request in the script to make multiple options easier to handle.

 

Nice! ill have a gander around internet and try some testing now to see if I can get something together.

 

Share this post


Link to post
Posted (edited)

I might have to take a quick shortcut here and give props to bergware/dynamix for his folder caching plugin.

 

https://github.com/bergware/dynamix/tree/master/source/cache-dirs

 

This plugin has done most of the heavy lifting for us in a way. Its not exactly what we need as unfortunately ram is more expansive than SSD or NVME.

 

However to list the check boxes this git provides for a start (with minor tweaking to meet our needs). :)

 

 

GUI & PLAN

This picture below shows the kind of option menu that would be perfect although ideally we would like it to go deeper into the folder structure if needed for the user  as this one only collects the first level of folder names.

image.thumb.png.a780e5de04d8e0f4108cc930a5417a1b.png 

Once a single or multiple folder locations are selected

image.png.c163b864c344cf5d2542c1e4b7ed0f0b.png

we then pipe the directory values into a recursive scan to list/pass every file and / or folder location string there after to the mover script to then miss as needed.

Using perhaps these commands its possible to get a bunch of information that we may need to a list, array or variable string i'm really not sure how or where the actual mover script is, I don't know the best method of transferring the location strings to the mover script itself.

 

commands:

 

https://www.cyberciti.biz/faq/linux-show-directory-structure-command-line/


#This gives us share folders that are setup in unraid to be cached. -d (directories only) -L (Tree Level deep from stated location)
tree -d -L 1 /mnt/cache 

 

 

https://www.explainshell.com/explain?cmd=find+.+-type+f+-name+"*confli*"+-exec+rm+-i+-f+{}+\;

 

#This line will search and return the entire provided location (string or variable)

find /mnt/cache -print

 

#This line will search and return the entire provided location (string or variable) for any file or folder that has the exact name of NextCloud
find /mnt/cache -name "NextCloud" -print
 

#This line will search and return the entire provided location (string or variable) all .png files 
find /mnt/cache -name "*.png" -print

 

This would be an very effective way to select folders to exclude from move and thus stay cached providing you can go deeper into the folder structure than what is coded in the folder caching plugin code from what I can see it looks to be pretty simple to change but then having the GUI adjust size for the structure as you navigate i'm a little unsure as web dev is not my apple pie just yet :)

 

 

 OPTIONAL Future Improvement

 

Perhaps the below could be optionally configured per folder but if left blank then everything in that folder would be classed as excluded from move.

image.png.83cd45b50234e760fedfcd17c09a063d.png 

Just a thought. and would require further thought or dynamicly generated User defined options to be generated per your selected folder.  MUCH Thought out.

 

 

CURRENT OPTION LOGIC

In regards to how your existing options will be affected or affect this I believe this may be easier than first thought by myself.

We add Boolean menu option like your others  under the folder selection list with the title "Include other Mover conditions on these Folders/files" .

 

its just a matter of having an if statement in the mover script check the passed in location strings and if a match skip that directory and child items.

 

I'm not sure if this is what you where actually looking for @hugenbdd but it took me a little while to put this together so I hope its useful.

 

Happy to try and help any way I can.

 

Edited by IronBeardKnight

Share this post


Link to post

I have spent some time today working on some test code to replace what is in the "age_mover" (Custom mover bash script based off of unRAID's default).

 

It now consists of several inputs 

Postion - Input

1 - start/stop/kill (from base script)

2  - Age in days

3 - Size in M

4 - Sparsenes value (1-9) (A . will be placed in the find before this value)

5 - Exclude filelist

 

Example of my test script in action to create the "find" Commnand

 ./movertest start 30 45 2 '/tmp/text.txt'

 

Age supplied
Size supplied
Sparness supplied
Skipfilelist supplied
SKIP FILE: /tmp/text.txt
Find string: find /mnt/cache/TV -depth -mtime +30 -size +45M
Find string: find /mnt/cache/TV -depth -mtime +30 -size +45M -printf '%S:%p\0' | awk -v RS='\0' -F : '$1 > 0.2 {sub(/^[^:]*:/, ""); print}'
Adding Skip File List
Skip File List Path: /tmp/text.txt
aftr string: find /mnt/cache/TV -depth -mtime +30 -size +45M -printf '%S:%p\0' | awk -v RS='\0' -F : '$1 > 0.2 {sub(/^[^:]*:/, ""); print}' | grep -vFf '/tmp/text.txt'
 

 

It would be nice if you guys could test this with  your needs and see if the find command is correct.  (Script attached)

 

The "exclude file" should just be a list of files with their full path... in /mnt/cache of course.

Also, change line 13 to your cache path.

SHAREPATH="/mnt/cache/TV"

 

 

 

 

IronBeardKnight

Once I have the functionality of an "ignore filelist" working, I think the next step would be to work on a GUI to create that file list. (Probably several versions away.)

 

movertest

 

 

Share this post


Link to post
On 4/21/2020 at 3:43 AM, hugenbdd said:

I have spent some time today working on some test code to replace what is in the "age_mover" (Custom mover bash script based off of unRAID's default).

 

It now consists of several inputs 

Postion - Input

1 - start/stop/kill (from base script)

2  - Age in days

3 - Size in M

4 - Sparsenes value (1-9) (A . will be placed in the find before this value)

5 - Exclude filelist

 

Example of my test script in action to create the "find" Commnand

 ./movertest start 30 45 2 '/tmp/text.txt'

 

Age supplied
Size supplied
Sparness supplied
Skipfilelist supplied
SKIP FILE: /tmp/text.txt
Find string: find /mnt/cache/TV -depth -mtime +30 -size +45M
Find string: find /mnt/cache/TV -depth -mtime +30 -size +45M -printf '%S:%p\0' | awk -v RS='\0' -F : '$1 > 0.2 {sub(/^[^:]*:/, ""); print}'
Adding Skip File List
Skip File List Path: /tmp/text.txt
aftr string: find /mnt/cache/TV -depth -mtime +30 -size +45M -printf '%S:%p\0' | awk -v RS='\0' -F : '$1 > 0.2 {sub(/^[^:]*:/, ""); print}' | grep -vFf '/tmp/text.txt'
 

 

It would be nice if you guys could test this with  your needs and see if the find command is correct.  (Script attached)

 

The "exclude file" should just be a list of files with their full path... in /mnt/cache of course.

Also, change line 13 to your cache path.

SHAREPATH="/mnt/cache/TV"

 

 

 

 

IronBeardKnight

Once I have the functionality of an "ignore filelist" working, I think the next step would be to work on a GUI to create that file list. (Probably several versions away.)

 

movertest 1.71 kB · 0 downloads

 

 

Apologies my work has been very busy as of late. 

 

Agreed! functionality first always.

 

I'm happy to give this some testing over the week after I finish work each day however the real  progress will be made on the weekend most likely.

 

Ill get back at you with my results, HOWEVER if anyone else in the forum is interested by all means jump in as well and giver the script a test as well. 

 

@hugenbdd Ill get back at you asap. :) 
 

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.