[Plugin] Mover Tuning


Recommended Posts

16 hours ago, aqua said:

I am on 6.9.2 and recently noticed that my cache drive was filling up and that mover wasn't doing anything. Mover tuning had pretty much everything set to "No" and the "Only move at this threshold of used cache space" set to 0%, but nothing would move. Didn't get anything to change until I set this to an actual percentage (I chose 30%) then ran mover. Is this a bug? I have been using for years without changing settings and not having errors.

0% is working for me.

 

Can you do the follow.

1.) Verify you have the latest version installed. (2021.04.20)

2.) Settings: Mover Logging: Enabled

3.) Screen shot of your settings page.

4.) Post the section from your log file that looks similar to this after it runs.

Apr 27 08:35:48 Tower root: mvlogger: Age supplied
Apr 27 08:35:48 Tower root: mvlogger: Size supplied
Apr 27 08:35:48 Tower root: mvlogger: Sparness supplied
Apr 27 08:35:48 Tower root: mvlogger: No Skipfiles Argument Supplied
Apr 27 08:35:48 Tower root: mvlogger: No Skipfiles Argument Supplied
Apr 27 08:35:48 Tower root: mvlogger: No Before Script Argument Supplied
Apr 27 08:35:48 Tower root: mvlogger: No After Script Argument Supplied
Apr 27 08:35:48 Tower root: mvlogger: CTIME Argument: no
Apr 27 08:35:48 Tower root: mvlogger: *********************************MOVER START*******************************
Apr 27 08:35:48 Tower root: mover: started
Apr 27 08:35:48 Tower root: mvlogger: No Script to Run.
Apr 27 08:35:48 Tower root: mvlogger: Share Name Only: Backups
Apr 27 08:35:48 Tower root: mvlogger: Cache Pool Name: cache
Apr 27 08:35:48 Tower root: mvlogger: cache Threshold Pct:
Apr 27 08:35:48 Tower root: mvlogger: OVERALL Threshold: 0
Apr 27 08:35:48 Tower root: mvlogger: Share Path: /mnt/cache/Backups
Apr 27 08:35:48 Tower root: mvlogger: Pool Pct Used: 60 %
Apr 27 08:35:48 Tower root: mvlogger: DFTPCT LIMIT USED FOR SETTING: 0
Apr 27 08:35:48 Tower root: mvlogger: Threshold Used: 0
Apr 27 08:35:48 Tower root: mvlogger: Skipfiletypes string: find "/mnt/cache/Backups" -depth
Apr 27 08:35:48 Tower root: mvlogger: Complete Mover Command: find "/mnt/cache/Backups" -depth | /usr/local/sbin/move -d 1

Link to comment

Yes, was/am running 2021.04.20. I noticed yesterday that when I was checking settings, before changing to a percentage to make things run, I think I had tried setting by age and had a weird thing where if I had it on "No" then I could select the number of days but if I had it on "Yes" then that selection was greyed out. Anyways, it is working on 0% now when I tested it so I guess things are back to normal, though I don't know what the issue was.

 

So I am clear with the tuner, the mover settings schedule (set to daily) just invoke the mover and then the mover tuning settings get looked at to determine whether mover does anything, right? As I'm set now, I just want it to run ever night and move everything off the cache (except cache prefer) which is why I have things set to 0% and everything else set to no (except allowing mover to run during parity check/rebuild).

 

I think I had logging on for a bit when it wasn't working, but then disabled it later - would the logs for that still be available? I can try to dig them up if they would be useful - though I don't know that I've looked at logs before and am not immediately sure where to go to grab them.

Link to comment
15 minutes ago, aqua said:

Yes, was/am running 2021.04.20. I noticed yesterday that when I was checking settings, before changing to a percentage to make things run, I think I had tried setting by age and had a weird thing where if I had it on "No" then I could select the number of days but if I had it on "Yes" then that selection was greyed out. Anyways, it is working on 0% now when I tested it so I guess things are back to normal, though I don't know what the issue was.

 

So I am clear with the tuner, the mover settings schedule (set to daily) just invoke the mover and then the mover tuning settings get looked at to determine whether mover does anything, right? As I'm set now, I just want it to run ever night and move everything off the cache (except cache prefer) which is why I have things set to 0% and everything else set to no (except allowing mover to run during parity check/rebuild).

 

I think I had logging on for a bit when it wasn't working, but then disabled it later - would the logs for that still be available? I can try to dig them up if they would be useful - though I don't know that I've looked at logs before and am not immediately sure where to go to grab them.

 

So I am clear with the tuner, the mover settings schedule (set to daily) just invoke the mover and then the mover tuning settings get looked at to determine whether mover does anything, right?

Currently - This will likely change soon.  I plan to make the button have the option of using default unraid mover, or to follow the settings.  (Just working on initial code now based on Squid's recommendation).

 

Logging - Depending on the version, Logging might not have been "created".  Not sure it's worth going back and looking for it.  I have logging sorted out now so users can at least enable it and give me a good idea of what/where might be failing.

Link to comment
21 hours ago, flyize said:

Would it be possible to allow 'Move files that are greater than this many days old' to be 1-7, and then by 5s? I'd really like to be able to keep things on there for a week.

 

edit: Or maybe even make it an input box?

I can add some more single digits.  Working on some back-end changes.  Will be a week or two before I get the next release out.

  • Like 1
Link to comment

###2020.05.06

- Added 0 to options of days old. (Will move anything greater than 24 hours old)

- Added 6,7,8,9 to options of days old.

- Increased days old limit to 765

- Removed "Original Mover" button after paypal donation link.

- Added "Move Now button follows plug-in filters:" to make "Move Now" button run based off of plug-in settings (Yes), or run original mover (No).

- Moved some code around in mover.php so ionice would be initiated.

- Updated mover code for moving from array to cache (generally cache-prefer) to follow Squid's update on 4/11/2021. (Fix for shares with space in their name)

 

 

 

 

**********CAUTION****************

Move Now button now calls unraids original move now code!!!!!  It will move almost everything, to change set Yes to "

Move Now button follows plug-in filters:" (Last question in section)

 

Future Changes:

Add an option to invoke original mover (i.e. everything) if percent used reaches a user selectable percent. (Must be higher than the entry for "Only move at this threshold of used cache space:")

  • Like 1
Link to comment
6 hours ago, tjb_altf4 said:

Sorry if this has been covered, but does the plugin work with multiple pools? Any gotchas?

Thanks.

Yes works with multiple pools.  

Gotcha's?

1.) Try to avoid spaces in share or pool names.

2.) All fields from gui are applied to all pools.  i.e. it's all the same.  with one exception at the moment.

2a.) You can specify a different percentage per pool.   

Create an entry in ca.mover.tuning.cfg file. (example: /boot/config/plugins/ca.mover.tuning             cachetv="65")

- "cachetv" being the name of the pool.

- if using single digits leading zero required. (i.e. cachetv="01" for 1 percent)

3.) I plan to expand this type of entry for a few other fields in the future. 

4.) As with any share setting, be very careful when setting cache-prefer.  Several posts where users are filling up their cache drive when using that setting.

  • Like 1
Link to comment
2 minutes ago, hugenbdd said:

As with any share setting, be very careful when setting cache-prefer. 

Any way you could intercept that setting in your plugin and inform the user if the total of data in that share is larger than the designated pool size? I feel that would at least warn people that they may be in for a bad time if they proceed.

Link to comment
1 minute ago, jonathanm said:

Any way you could intercept that setting in your plugin and inform the user if the total of data in that share is larger than the designated pool size? I feel that would at least warn people that they may be in for a bad time if they proceed.

Partially

To do it right, it would take some effort.  I can't just check the share size, but have to check each drive for the share, then add up it's size on disks.  And after that, then I can only throw an error to syslog and then not "move" any data.

Link to comment
1 hour ago, hugenbdd said:

not "move" any data.

That's not ideal, as I can envision a situation where you want as much as possible of a set of files on the pool restricted by the free space setting.

 

I was hoping for a warning that alerts the user that they may not actually want that setting, but allow it if forced.

Link to comment

To use the mover script do I point to just the folder or to the folder/file name? Also, is a preexisting user script the correct format so I could just turn off the schedule for it and point mover tuner to that?  I'm utilizing the scripts to stop my torrent container before mover and restart afterwards so that seeding torrents are moved to the array. 

Edited by mmag05
Link to comment
4 hours ago, mmag05 said:

To use the mover script do I point to just the folder or to the folder/file name? Also, is a preexisting user script the correct format so I could just turn off the schedule for it and point mover tuner to that?  I'm utilizing the scripts to stop my torrent container before mover and restart afterwards so that seeding torrents are moved to the array. 

I don't understand your question.

 

Install the plug-in through CA community Apps, search for "Mover Tuner" and it should come up.

 

To run scripts before and after, fill in the full path to the file you want run and make sure it has execution permissions.

Link to comment

New Release

 

2020.05.13

  • Fixed issue during a mover stop command, where it wasn't exiting correctly.
  • Added Test mode.  - Will run the find command outputting all the files that would have been sent to mover to the syslog if logging enabled. (Lets you see if your filter choices are correct.)
  • Add "Move All" files in a share of cache-yes to the array if cache pool is above a certain % (Must be greater than "Only move at this threshold of used cache space:"
    • Plug-in code will still run through each share after this function.  However, it shouldn't move anything if a share path on cache is empty.
       

Some big changes.  I tested as best I could, please let me know if you run into issues. 

Link to comment

Now that test mode is nice!!! I've been bumping up my days just to see if it'll move something and then restoring it to the days I want them to remain after I watched it move files. 

 

Meaning in English, thanks for the add. I wish I had it earlier, but now we have it. ;)

Link to comment
1 hour ago, kizer said:

Now that test mode is nice!!! I've been bumping up my days just to see if it'll move something and then restoring it to the days I want them to remain after I watched it move files. 

 

Meaning in English, thanks for the add. I wish I had it earlier, but now we have it. ;)

Yeah, I was tired of commenting out the line in the age_mover file while testing.  Figured it would help people who have complicated filters. 

 

Side note, it will call the "before/after scripts".  The test mode just won't send files to mover, it just executes the "custom find command".

Link to comment
5 hours ago, Choohoo said:

Thank you for this! Is there a way to set it so that it auto moves from cache->array (ie. auto press move) if the cache disk utilization is above x percent?

Kind of.

You have to set the schedule to run every hour or daily.  This plug-in does not monitor the size of the cache except when it is run on a schedule. (So hourly, daily, weekly, etc)

 

Link to comment

Is there a setting that has the mover run when shutting down the array? because I was trying to turn off my server but the mover started when pressing stop and is moving everything now... is this a new setting or something? I don't recall this happening before.

 

syslog:

May 15 15:52:34 Unraid emhttpd: shcmd (7093746): rm -f /etc/avahi/services/smb.service
May 15 15:52:34 Unraid emhttpd: Stopping mover...
May 15 15:52:34 Unraid emhttpd: shcmd (7093748): /usr/local/sbin/mover stop
May 15 15:52:34 Unraid root: Starting Mover
May 15 15:52:34 Unraid root: Forcing turbo write on
May 15 15:52:34 Unraid kernel: mdcmd (107): set md_write_method 1
May 15 15:52:34 Unraid kernel: 
May 15 15:52:34 Unraid root: ionice -c 2 -n 0 nice -n 0 /usr/local/sbin/mover.old 
May 15 15:52:34 Unraid root: mover: started

 

Edited by radwimp
Link to comment
26 minutes ago, itimpi said:

There was a bug reported in the plugin that could cause this but I thought it was now fixed.   Are you sure you are on the latest version of the plugin?

Oh, my bad. I was on the version just previous. I'll update. This was a minor bug, or would my data be weird since the array was trying to turn off? Thanks.

Link to comment
23 hours ago, radwimp said:

Oh, my bad. I was on the version just previous. I'll update. This was a minor bug, or would my data be weird since the array was trying to turn off? Thanks.

Your data "should" be okay.  If the array shutdown during a move, it's possible the file that the mover binary was in the process of sending to the array might be corrupt, but I doubt that all the files in your cache are corrupt.

 

Latest release did fix this issue.

Link to comment
On 5/13/2021 at 1:55 PM, hugenbdd said:

Added Test mode.  - Will run the find command outputting all the files that would have been sent to mover to the syslog if logging enabled. (Lets you see if your filter choices are correct.)

 

This sounds great, but where does it show the output of the find command?

Link to comment
8 minutes ago, acosmichippo said:

 

This sounds great, but where does it show the output of the find command?

Added Test mode.  - Will run the find command outputting all the files that would have been sent to mover to the syslog if logging enabled. (Lets you see if your filter choices are correct.)

  • Like 2
Link to comment
19 minutes ago, acosmichippo said:

 

This sounds great, but where does it show the output of the find command?

upper right log icon/link 

or if in the console....

/var/log/syslog

 

But like Simon F mentioned, make sure logging is enabled and you have it in test mode. (One of the last options in schedule page for the plug-in.)

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.