[Plugin] Mover Tuning


Recommended Posts

On 4/21/2020 at 3:43 AM, hugenbdd said:

I have spent some time today working on some test code to replace what is in the "age_mover" (Custom mover bash script based off of unRAID's default).

 

Thank you for this.  I've got several old systems for homelab use now that I'll be able to test this on later today.

Link to comment

Hi @hugenbdd  

 

Please find attached your script containing my notes and recommendations for things and why I recommend them.

 

I hope that was ok. I am not sure as some people get iffy editing their files so I only added some notes if you would like to make some minor tweaks and test with a new script. 

 

I have explained as best I can as to why  you may have been getting inconsistent results using the find command in the past but let me know if you need more.

 

@LeoFender I am not sure if you have had a chance to test as well yet but please feel free to also review my information and proposedmovertest changes.

 

 

Oh P.S I'm not the best at spelling and I have lazy grammar so please excuse. :)

Edited by IronBeardKnight
Link to comment
1 hour ago, IronBeardKnight said:

Hi @hugenbdd  

 

Please find attached your script containing my notes and recommendations for things and why I recommend them.

 

I hope that was ok. I am not sure as some people get iffy editing their files so I only added some notes if you would like to make some minor tweaks and test with a new script. 

 

I have explained as best I can as to why  you may have been getting inconsistent results using the find command in the past but let me know if you need more.

 

@LeoFender I am not sure if you have had a chance to test as well yet but please feel free to also review my information and proposed movertest 5.98 kB · 1 download  changes.

 

 

Oh P.S I'm not the best at spelling and I have lazy grammar so please excuse. :)

Thanks for taking a look at it.

 

Few explanations on your comments.

 

Sharepath is just there to mimic the variable used in the real mover script.

 

Input variables need to have some value assigned.  I use 0 as a "non value", and then later "skip" adding that section to the find command if it's set to zero.

 

Size is "legacy" for me, and takes the size based on the schedule screen.

 

Sparseness - I need someone to test this as I'm not sure what legitimate output is.  But I think I got the script to output the find command like the example I gave.

 

I didn't notice the file extension request before.  I'll work on adding that.

 

I exclude based on a pipe to grep and a file list.....

 

mtime will remain in days.  It's why I originally modified the script.  I use it to let my cache get close to 80% full and slowly move aging files off.  i.e. Think new shows for the past 45 days.

 

 

Link to comment

ok cool that is what i was trying to convey with the mtime parameters, maybe i did not correctly convoy correctly.

 

From my testing for you i have deduced:

mtime:

 

mtime 0   [Younger than 24hrs and Older]

mtime +0   [Older than 24hrs]

mtime +1    [Older than 48hrs]

mtime +2   [Older than 72hrs]

 

if you "skip" from the statement mtime completely based on == 0 or == +0 then this only allowing for 48hrs or older in your script. I'm only going off what i have seen so far.

 

 

Looking at the schedular options for mover I was just wondering if the options actually reflected the time frames true and / or may cause some confusion.

 

I'm happy to test sparseness for sure but I am having a hard time understanding what it actually is to test it.

:)

 

 

 

Link to comment
On 4/26/2020 at 12:32 AM, IronBeardKnight said:

ok cool that is what i was trying to convey with the mtime parameters, maybe i did not correctly convoy correctly.

 

From my testing for you i have deduced:

mtime:

 

mtime 0   [Younger than 24hrs and Older]

mtime +0   [Older than 24hrs]

mtime +1    [Older than 48hrs]

mtime +2   [Older than 72hrs]

 

if you "skip" from the statement mtime completely based on == 0 or == +0 then this only allowing for 48hrs or older in your script. I'm only going off what i have seen so far.

 

 

Looking at the schedular options for mover I was just wondering if the options actually reflected the time frames true and / or may cause some confusion.

 

I'm happy to test sparseness for sure but I am having a hard time understanding what it actually is to test it.

:)

 

 

 

Well, the mtime is interesting..  Found this post.  I'll try and make some chages/test so that older than 1 day works right.

https://unix.stackexchange.com/questions/92346/why-does-find-mtime-1-only-return-files-older-than-2-days

 

Link to comment

Can the setting 'Only move at this threshold of used cache space' be used in conjunction with the scheduler?  Such that the mover is scheduled to only run once a day, but it will only run if the used space is above the threshold?  Or does this setting always invoke the mover the moment the threshold is reached?

Edited by IamSpartacus
Link to comment
29 minutes ago, IamSpartacus said:

Can the setting 'Only move at this threshold of used cache space' be used in conjunction with the scheduler?  Such that the mover is scheduled to only run once a day, but it will only run if the used space is above the threshold?  Or does this setting always invoke the mover the moment the threshold is reached?

It only runs when scheduled.  Then checks the % of cache used to determine if it should continue to move files.

 

Example: Schedule set for 3AM with 65% cache used setting.

 

At 3AM the mover script runs.

Check the space used on the cache.

If space used greater than or equal to 65% it continue to invoke the mover script.

If space used less than 65% it will exit the script with an entry in the log file with something like "mover not running as it has not reached the used threshold".

 

Link to comment
2 minutes ago, hugenbdd said:

It only runs when scheduled.  Then checks the % of cache used to determine if it should continue to move files.

 

Example: Schedule set for 3AM with 65% cache used setting.

 

At 3AM the mover script runs.

Check the space used on the cache.

If space used greater than or equal to 65% it continue to invoke the mover script.

If space used less than 65% it will exit the script with an entry in the log file with something like "mover not running as it has not reached the used threshold".

 

Perfect!  That's exactly what I need.  Thank you!

Link to comment
13 hours ago, hugenbdd said:

Well, the mtime is interesting..  Found this post.  I'll try and make some chages/test so that older than 1 day works right.

https://unix.stackexchange.com/questions/92346/why-does-find-mtime-1-only-return-files-older-than-2-days

 

hahahah I tried to explain it much simpler so others reading this could understand as well, that is exactly the same thing was talking about.

 

I did not know about this thread though. :)

 

I'm still working out some other kinks from my side with some other features I'm working on and will try to test the Sparsness if i understand it correctly when I get some more time. 

  • Like 1
Link to comment
2 hours ago, Sensei73 said:

Nice work! I'm interested in the functionality of including/excluding directory as I moved all my files to /mnt/user/media to be able to use hardlink move with Sonarr and Radarr. If you want me to test 👋

Almost done.  Doing some basic testing on my box, but life seems to get in the way.

 

I let you exclude based on either a comma separated list (something like .mkv,.mp4)  or through a file that contains a list of files. (full cache path). Both options can be chosen at the same time, as I just "pipe" them after the find command.  And remember that this only moves files off /mnt/cache/<sharename> to the array.

 

csv list: grep -v -e '.mkv

Exclude filelist: grep -vFf 'tmptest.txt'

 

 

https://imgur.com/a/C2aMxw1

 

 

 

 

Link to comment
3 minutes ago, Kaiser said:

Is somewhere a guide how to install this? It is really cool. But since Im on day 1 with unraid, Im kinda lost. 

 

Install Community Applications plugin (pinned near the top of this same subforum), then use it to install any other plugins or dockers.

 

 

Link to comment

New version just released.  I would like some to test and let me know if you run into issues.

 

I rewrote how the new "age_mover" creates it's string to call Unraids mover binary.  This will allow me to add extra request easier in the future as I am now just creating a string and then executing it.

 

Added
1.) Sparseness (.1 as an example)
2.) Ignore based on list inside of a file (provide a path) (ignore filelist)
3.) Ignore based on file types (comma separated list of file extension.  .mkv,.mp4,.txt )

Link to comment

Is it possible to have mover trigger when cache free space is below a certain level? I have it set to weekly but sometimes my cache gets filled up faster than that so rather than get a free space alert and have to do something I’d like it to just invoke mover

 

 

Sent from my iPhone using Tapatalk

Link to comment
1 hour ago, nickp85 said:

Is it possible to have mover trigger when cache free space is below a certain level? I have it set to weekly but sometimes my cache gets filled up faster than that so rather than get a free space alert and have to do something I’d like it to just invoke mover

 

 

Sent from my iPhone using Tapatalk

Easiest way to come close to what you want is to change mover to "hourly" and then set the "threshold of used cache" to your desired % of when to actually move files.  Mover will not move anything if your cache is under that percent.  It just exits with a log entry.

 

https://imgur.com/gallery/ODc9sBV

 

  • Like 1
Link to comment

Is it normal that the button on the main page for mover sits at “mover is running” but the Unraid log shows that the threshold wasn’t met so not moving. Seems to think the process is still running but it isn’t.

 

Update: actually it looks like manually invoking mover ignores the configured rules. Is that expected?

 

 

Sent from my iPhone using Tapatalk

Link to comment
28 minutes ago, nickp85 said:

Is it normal that the button on the main page for mover sits at “mover is running” but the Unraid log shows that the threshold wasn’t met so not moving. Seems to think the process is still running but it isn’t.

 

Update: actually it looks like manually invoking mover ignores the configured rules. Is that expected?

 

 

Sent from my iPhone using Tapatalk

 

I believe that is expected.  I did not change the code around that part of the plug-in, as the button press probably still directs to the original mover bash script.

 

  • Like 1
Link to comment
16 hours ago, hugenbdd said:

 

I believe that is expected.  I did not change the code around that part of the plug-in, as the button press probably still directs to the original mover bash script.

 

weird! I am currently testing with this plugin with a folder on ignore list, and invoking the script with the mover button works as expected!

 

Your script is working perfectly (as today). I can now have hardlink copy with sonarr and mover working perfectly! Thanks again.

 

PS: If I may, a suggestion, It could be useful to have a debug option, to see what's happening in a log file.

Edited by Sensei73
Link to comment

I think there's a bug where the script doesn't quote share directories so e.g. a share with the name "Time Machine" will cause this error:

May 11 05:42:07 NAS root: find: '/mnt/cache/Time': No such file or directory
May 11 05:42:07 NAS root: find: 'Machine/': No such file or directory

which I hadn't seen previously (though it's possible I missed it.)

 

It's probably not best practice to include spaces in share names and I've solved it by renaming but thought I'd mention it.

 

Great plugin by the way, thanks.

Link to comment
1 hour ago, CS01-HS said:

I think there's a bug where the script doesn't quote share directories so e.g. a share with the name "Time Machine" will cause this error:


May 11 05:42:07 NAS root: find: '/mnt/cache/Time': No such file or directory
May 11 05:42:07 NAS root: find: 'Machine/': No such file or directory

which I hadn't seen previously (though it's possible I missed it.)

 

It's probably not best practice to include spaces in share names and I've solved it by renaming but thought I'd mention it.

 

Great plugin by the way, thanks.

 

You should not have to rename it.

 

The strings in the file should be being passed through this plugin with any paths being encased in double quote.

 

@hugenbdd I think I also mentioned this to you in my notes from first testings with you as well, All file Paths need to be encased in "" otherwise the script language thinks its a new parameter or command.

 

Also form one terrible speller too probably someone how made a mistake, you have a spelling error :) 

I'm setting up for some testing tonight.

Spelling mistake.PNG

Link to comment
2 hours ago, CS01-HS said:

I think there's a bug where the script doesn't quote share directories so e.g. a share with the name "Time Machine" will cause this error:


May 11 05:42:07 NAS root: find: '/mnt/cache/Time': No such file or directory
May 11 05:42:07 NAS root: find: 'Machine/': No such file or directory

which I hadn't seen previously (though it's possible I missed it.)

 

It's probably not best practice to include spaces in share names and I've solved it by renaming but thought I'd mention it.

 

Great plugin by the way, thanks.

Oh wow, big miss on my part.  I don't have any shares with spaces in them..  Will work on a fix shortly.

Link to comment
53 minutes ago, IronBeardKnight said:

 

You should not have to rename it.

 

The strings in the file should be being passed through this plugin with any paths being encased in double quote.

 

@hugenbdd I think I also mentioned this to you in my notes from first testings with you as well, All file Paths need to be encased in "" otherwise the script language thinks its a new parameter or command.

 

Also form one terrible speller too probably someone how made a mistake, you have a spelling error :) 

I'm setting up for some testing tonight.

Spelling mistake.PNG

Missed the space issue in the notes.. Sorry.

Will fix my fat finger mistake shortly...

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.