[Plugin] Mover Tuning

[wgstarks]

4 hours ago, wgstarks said:

         Does the traditional mover schedule effect when the actions set in the tuning settings are performed? Seem to remember that it does. I set this to hourly and mover ran on the hour. Still testing to be sure the filters are being respected though. I've enabled logging and will include them in my next post if I'm still having issues.

Changing the mover schedule didn't work. Now mover is moving all files every hour (schedule set to hourly) ignoring the pool percentage setting which is 60% and the force move all files which is 30 3 * * *.

 

mover_log.txt

Edited October 17, 2021 by wgstarks
typo

[hugenbdd]

4 hours ago, wgstarks said:

I may have found the issue-

 

 

Does the traditional mover schedule effect when the actions set in the tuning settings are performed? Seem to remember that it does. I set this to hourly and mover ran on the hour. Still testing to be sure the filters are being respected though. I've enabled logging and will include them in my next post if I'm still having issues.

Yes

There are two - Move All files.

One that runs based on the cron entry in the configs that calls the original mover (from the original plug-in way back...).  And another that is based off percentage used.  Two different sections of code....

[wgstarks]

4 minutes ago, hugenbdd said:

Yes

There are two - Move All files.

One that runs based on the cron entry in the configs that calls the original mover (from the original plug-in way back...).  And another that is based off percentage used.  Two different sections of code....

So "Mover Schedule" and "Force move of all files on a schedule" both perform the same function (mover start)?

[hugenbdd]

43 minutes ago, wgstarks said:

Changing the mover schedule didn't work. Now mover is moving all files every hour (schedule set to hourly) ignoring the pool percentage setting which is 60% and the force move all files which is 30 3 * * *.

 

mover_log.txt 19.77 kB · 3 downloads

I think you want to remove the "Force move of all files on a schedule:"  (Set to no)

 

And keep the settings for "Move All from Cache-Yes shares when disk is above a certain percentage:"  Yes, your %.

 

This checks to see if the cache is above your set threshold.  If it is, it will move everything.  Based on the scheduler.

 

Looking back at your original question.. The plug-in just check's based on the schedule not based on the % used.  Best way to follow this would be to..

1.) Mover Schedul: Hourly

2.) Set your percentage to move files. (Only move at this threshold of used cache space:  20%)

3.) Set % to move all file if greater than your percentage.

 

You will still have an "hour" delay between checking.  But usually a large enough hard drive and combination of low enough threshold make this workable.

[hugenbdd]

1 hour ago, gandalf15 said:

seems like the plugin is broken. Mover worked without it, but it caused 100% CPU usage and IO Wait to level that plex didnt work anymore. So I installed the plugin - which made my cache drive to fill up to 100% and all docker stopped.

I had to remove it again. Someone else have this problem?

Plug-in still works.  however...

It's easy to set enough stringent configs that the cache does fill up.  I agree, cache - prefer is the most common issue.

 

Best way to check to see why a cache is filling up outside of cache-prefer is to run in test mode and review the logs.  

[wgstarks]

3 minutes ago, hugenbdd said:

I think you want to remove the "Force move of all files on a schedule:"  (Set to no)

 

And keep the settings for "Move All from Cache-Yes shares when disk is above a certain percentage:"  Yes, your %.

 

This checks to see if the cache is above your set threshold.  If it is, it will move everything.  Based on the scheduler.

 

Looking back at your original question.. The plug-in just check's based on the schedule not based on the % used.  Best way to follow this would be to..

1.) Mover Schedul: Hourly

2.) Set your percentage to move files. (Only move at this threshold of used cache space:  20%)

3.) Set % to move all file if greater than your percentage.

 

You will still have an "hour" delay between checking.  But usually a large enough hard drive and combination of low enough threshold make this workable.

I think I've got it now. I was looking for a way to also use this plugin to move all files (ignoring the other filters) daily. Thought the force move of all files would do that for me but I see now that it won't work. I'll go back to running a manual mover via user scripts on a daily schedule.

[gandalf15]

6 hours ago, hugenbdd said:

Plug-in still works.  however...

It's easy to set enough stringent configs that the cache does fill up.  I agree, cache - prefer is the most common issue.

 

Best way to check to see why a cache is filling up outside of cache-prefer is to run in test mode and review the logs.  

My cache is 2TB ( 2 1TB raid0 SSDs). I changed one share - which is the download share - to "Use cache pool: Yes":

after that it filled up to the top.

The mover is set to hourly:

 

Those settings filled it up to 20 kbytes left.

I stopped using the cache except for the mount drive and the app data in the past - since the mover made plex unusable.

I am thankful for help - it looked like the plugin broke the mover, since when I pressed "Move" it did nothing. I should have been clearer - it was not like I was going to say that your work doesn't work, I thought more a update might have caused some issues or something.

Thank you very much for your help

[wgstarks]

Still not working right. For some reason files are all being moved every hour. Disk usage percentage setting is being ignored.

 

Current settings-

 

 

 

mover_log.txt

[hugenbdd]

I will be out for a few days and won't be able to look at the issues until Wed. evening at the earliest.

 

The best way to get the settings you want is to set the plug-in to test mode and enable logging.  Then review the syslog for the cache entries.

[i1mran92]

On 9/28/2021 at 11:39 PM, polishprocessors said:

Actually I had renamed my Media folder to media but it shows up in /boot/config/shares as Media.cfg

Think it’s safe to rename that to media.cfg ?

 

hi there. sorry to intrude. i think i'm having the same problem. i previously rename my share from Data to data, and now mover tuning detects Data. normal mover works without issue.

 

were you able to rename the file?

if so, would you mind guide me on how to do it. pretty noob when using the command line.

thank you in advance.

[hugenbdd]

2 hours ago, i1mran92 said:

 

hi there. sorry to intrude. i think i'm having the same problem. i previously rename my share from Data to data, and now mover tuning detects Data. normal mover works without issue.

 

were you able to rename the file?

if so, would you mind guide me on how to do it. pretty noob when using the command line.

thank you in advance.

Yes I believe it's safe to rename the file with the correct case.

You should make a backup of the file and store it somewhere else just in case though.

 

FYI - This directory will list the files/shares.

cd /boot/config/shares

 

[i1mran92]

On 10/27/2021 at 3:17 AM, hugenbdd said:

Yes I believe it's safe to rename the file with the correct case.

You should make a backup of the file and store it somewhere else just in case though.

 

FYI - This directory will list the files/shares.

cd /boot/config/shares

 

thank you very sir. i was able to make the changes. it detects the correct folder now.

 

also, if you dont mind, do you know how i can use the user script so that i can close qbitorrent before the mover start and open qbitorrent after the mover ends. i tried it, and it doesnt seem to work. any advice would be awesome.

 

i added this into the mover tuning before and after mover start script

/boot/config/plugins/user.scripts/scripts/stopping_qbittorrent

/boot/config/plugins/user.scripts/scripts/starting_qbitorrent

 

some screenshot of the settings.

 

[gandalf15]

I still have the problem actually. I enabled logging and saw that it tries to move "Media" - but my share is named "media". I have no Idea why it thinks its with capital. Everywhere i used "media" and not "Media". All works except the mover?

Is it save to just rename "media" share to "Media" without breaking everything?

[hugenbdd]

23 minutes ago, gandalf15 said:

I still have the problem actually. I enabled logging and saw that it tries to move "Media" - but my share is named "media". I have no Idea why it thinks its with capital. Everywhere i used "media" and not "Media". All works except the mover?

Is it save to just rename "media" share to "Media" without breaking everything?

look to see what the config file is set to

 

ls -ltr /boot/config/shares

 

If it mis-matches (based on the case of the letter) then that is the problem.  Else, maybe post your debug log files?

[hugenbdd]

On 10/28/2021 at 1:59 AM, i1mran92 said:

thank you very sir. i was able to make the changes. it detects the correct folder now.

 

also, if you dont mind, do you know how i can use the user script so that i can close qbitorrent before the mover start and open qbitorrent after the mover ends. i tried it, and it doesnt seem to work. any advice would be awesome.

 

i added this into the mover tuning before and after mover start script

/boot/config/plugins/user.scripts/scripts/stopping_qbittorrent

/boot/config/plugins/user.scripts/scripts/starting_qbitorrent

 

some screenshot of the settings.

 

Sorry, can't help much here.  Maybe try a simple script that echo's out to the sys log and see what happens.  That section just calls the path that is put into it...

[gandalf15]

15 minutes ago, hugenbdd said:

look to see what the config file is set to

 

ls -ltr /boot/config/shares

 

If it mis-matches (based on the case of the letter) then that is the problem.  Else, maybe post your debug log files?

There are all my mounts  - and one named "media.cfg". So it seems right there. I dont know though why your mover plugin thinks its "Media"?

[captaincu]

I've noticed that setting mover tuning to move based on age at greater than 1 days old only moves once the file hits 48 hours from MTIME. That makes sense to me, but going with this logic I'd expect setting it to 0 days would move files only when they hit the 24 hour mark using MTIME. Unfortunately, in practice it moves all files even if it's less than 24 hours. Is there a way for me to move files that have been on the cache for at least 24 hours?
 

Move files off cache based on age? Yes

Move files that are greater than this many days old: 0

 

[captaincu]

On 10/31/2021 at 11:23 AM, captaincu said:

I've noticed that setting mover tuning to move based on age at greater than 1 days old only moves once the file hits 48 hours from MTIME. That makes sense to me, but going with this logic I'd expect setting it to 0 days would move files only when they hit the 24 hour mark using MTIME. Unfortunately, in practice it moves all files even if it's less than 24 hours. Is there a way for me to move files that have been on the cache for at least 24 hours?
 

Move files off cache based on age? Yes

Move files that are greater than this many days old: 0

 

I did my own digging and found that the -mtime +1 command returns files modified >= 2 days ago. Thus, to get files >= 1 day old, -mtime +0 should be used.

Source

 

Currently, the mover test logs indicates that -mtime isn't used at all when "Move files that are greater than this many days old" is set to 0. This results in everything being moved. The next increment of 1 runs -mtime +1, which selects files modified >= 2 days ago.

 

I think this setting should be modified such that setting it to 0 omits -mtime and setting it to 1 starts at -mtime +0. The next integers should increase -mtime by 1.

[hugenbdd]

5 minutes ago, captaincu said:

I did my own digging and found that the -mtime +1 command returns files modified >= 2 days ago. Thus, to get files >= 1 day old, -mtime +0 should be used.

Source

 

Currently, the mover test logs indicates that -mtime isn't used at all when "Move files that are greater than this many days old" is set to 0. This results in everything being moved. The next increment of 1 runs -mtime +1, which selects files modified >= 2 days ago.

 

I think this setting should be modified such that setting it to 0 omits -mtime and setting it to 1 starts at -mtime +0. The next integers should increase -mtime by 1.

Sorry for the late reply.

That makes sense.  I'll do some testing for the next release. 

[snipes1259]

Hello,

 

Running into an issue with the mover tuner at the moment. The intent is to have the mover run every hour, and only move files that are >7 days old, however if the cache is >75% full, run the standard mover/move everything on a Cache:YES share to the array.

 

I have my settings set to what I think are correct, i.e. only move files that are greater than 7 days and 'move all from cache:yes shares' to yes and 75%. 

 

When testing the settings with the Move Now button follows plug-in filters and Test Mode set to yes, I can see mover picking up on files that are less than 7 days old in the log. However, once I set test mode to off and manually invoke the Mover, it runs without actually moving any files and doesn't detect any files that need to be moved. 

 

Any idea what's gong on?

 

 

[hugenbdd]

Can you post the syslog entries with logging enabled for when it is not working?

[snipes1259]

1 hour ago, hugenbdd said:

Can you post the syslog entries with logging enabled for when it is not working?

See attached

mover_log.txt

[hugenbdd]

6 minutes ago, snipes1259 said:

See attached

mover_log.txt 6.6 kB · 2 downloads

Nov 4 21:31:33 WheelzServe root: mvlogger: /mnt/cache PCT USED: 86
Nov 4 21:31:33 WheelzServe root: mvlogger: Pool /mnt/cache is above 'move all' percentage, moving all cache-yes shares to array
Nov 4 21:31:33 WheelzServe root: mvlogger: Not in Test mode
Nov 4 21:31:33 WheelzServe root: find: unknown predicate `-depth+= | /usr/local/sbin/move -d 1'
Nov 4 21:31:33 WheelzServe root: mvlogger: Complete Mover Command: find /mnt/cache/Media -depth

 

Will look into this tomorrow morning. 

Could be an issue with the code, or could be an issue with the share config files.

[hugenbdd]

11 hours ago, snipes1259 said:

See attached

mover_log.txt 6.6 kB · 3 downloads

Sent you a message with updated file to test with.  Will cut a release if it works for you. (After looking into the find date issue above.)

[hugenbdd]

New update fixing the two issues above.  I hope removing the 0 from the days old drop down does not cause issues.   However, setting it to 1 will move with a "find -mtime 0" now.

 

11-07-2021

- Updated "move all" code to fix wrong variable and script error.
- Removed 0 from days old drop down
- Updated age_mover to day - 1 to be able to move files 24 hours old instead of 48. (See post from captaincu on 11/3/2021)