[Plugin] Mover Tuning


464 posts in this topic Last Reply

Recommended Posts

Hi Squid, I just want to say a big THANK YOU for this plugin.  I've been using unraid for over a year and am sad I'm only just now learning about this.  It is one of the most useful plugins I've installed, and makes SSD caches so much more powerful!

Edited by acosmichippo
Link to post
  • Replies 463
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

CA Mover Tuning       A simple little plugin (mainly created for my own purposes) that will let you fine tune the operation of the mover script On scheduled runs of mover, o

I would suggest removing the plug in, then moving everything.   I'm working on getting a Dev box up and running so I can continue working on this for the new Beta.

New Release ###2021.04.15 - Changed text for field description of "Force move of files on a schedule:" to "Force move of all files on a schedule:" - No log entry for this, only an email

Posted Images

1 hour ago, acosmichippo said:

Hi Squid, I just want to say a big THANK YOU for this plugin.  I've been using unraid for over a year and am sad I'm only just now learning about this.  It is one of the most useful plugins I've installed, and makes SSD caches so much more powerful!

@hugenbdd took over this plugin a while ago.  The thanks on the current iterations go to him.

Link to post

Just upgraded to stable 6.9.1 and also upgraded mover tuning to 2021.3.17.  Mover didn't automatically move.  I have it set to move at 4:30 am in the morning.  the syslog shows nothing at that time.

 

Apr 7 03:39:05 Skynet emhttpd: read SMART /dev/sdd Apr 7 04:00:01 Skynet Plugin Auto Update: Checking for available plugin updates Apr 7 04:00:08 Skynet Plugin Auto Update: Checking for language updates Apr 7 04:00:08 Skynet Plugin Auto Update: Community Applications Plugin Auto Update finished Apr 7 04:52:37 Skynet emhttpd: read SMART /dev/sdc Apr 7 04:52:39 Skynet emhttpd: read SMART /dev/sdj Apr 7 04:52:55 Skynet emhttpd: read SMART /dev/sdi Apr 7 04:53:09 Skynet emhttpd: read SMART /dev/sde Apr 7 04:54:59 Skynet emhttpd: spinning down /dev/sdd Apr 7 05:09:20 Skynet emhttpd: spinning down /dev/sdi Apr 7 05:10:41 Skynet emhttpd: spinning down /dev/sde Apr 7 05:12:28 Skynet emhttpd: spinning down /dev/sdj

 

should I revert to an older version of mover?  or delete the plug in and reinstall?  

Link to post
9 minutes ago, cbr600ds2 said:

Just upgraded to stable 6.9.1 and also upgraded mover tuning to 2021.3.17.  Mover didn't automatically move.  I have it set to move at 4:30 am in the morning.  the syslog shows nothing at that time.

 

Apr 7 03:39:05 Skynet emhttpd: read SMART /dev/sdd Apr 7 04:00:01 Skynet Plugin Auto Update: Checking for available plugin updates Apr 7 04:00:08 Skynet Plugin Auto Update: Checking for language updates Apr 7 04:00:08 Skynet Plugin Auto Update: Community Applications Plugin Auto Update finished Apr 7 04:52:37 Skynet emhttpd: read SMART /dev/sdc Apr 7 04:52:39 Skynet emhttpd: read SMART /dev/sdj Apr 7 04:52:55 Skynet emhttpd: read SMART /dev/sdi Apr 7 04:53:09 Skynet emhttpd: read SMART /dev/sde Apr 7 04:54:59 Skynet emhttpd: spinning down /dev/sdd Apr 7 05:09:20 Skynet emhttpd: spinning down /dev/sdi Apr 7 05:10:41 Skynet emhttpd: spinning down /dev/sde Apr 7 05:12:28 Skynet emhttpd: spinning down /dev/sdj

 

should I revert to an older version of mover?  or delete the plug in and reinstall?  

Manual mover seems to not have worked as well

 

from  syslog - 

 

Apr  7 20:25:36 Skynet emhttpd: read SMART /dev/sdn
Apr  7 20:45:41 Skynet emhttpd: read SMART /dev/sdc
Apr  7 20:45:43 Skynet emhttpd: read SMART /dev/sdj
Apr  7 20:45:56 Skynet emhttpd: read SMART /dev/sde
Apr  7 20:45:59 Skynet emhttpd: read SMART /dev/sdi
Apr  7 20:57:16 Skynet emhttpd: shcmd (9078): /usr/local/sbin/update_cron
Apr  7 20:57:31 Skynet emhttpd: shcmd (9079): /usr/local/sbin/mover &> /dev/null &
Apr  7 21:02:22 Skynet emhttpd: spinning down /dev/sdi
Apr  7 21:03:28 Skynet emhttpd: spinning down /dev/sde

 

 

Link to post
3 minutes ago, cbr600ds2 said:

Manual mover seems to not have worked as well

 

from  syslog - 

 

Apr  7 20:25:36 Skynet emhttpd: read SMART /dev/sdn
Apr  7 20:45:41 Skynet emhttpd: read SMART /dev/sdc
Apr  7 20:45:43 Skynet emhttpd: read SMART /dev/sdj
Apr  7 20:45:56 Skynet emhttpd: read SMART /dev/sde
Apr  7 20:45:59 Skynet emhttpd: read SMART /dev/sdi
Apr  7 20:57:16 Skynet emhttpd: shcmd (9078): /usr/local/sbin/update_cron
Apr  7 20:57:31 Skynet emhttpd: shcmd (9079): /usr/local/sbin/mover &> /dev/null &
Apr  7 21:02:22 Skynet emhttpd: spinning down /dev/sdi
Apr  7 21:03:28 Skynet emhttpd: spinning down /dev/sde

 

 

Few things.

 

1.) Make sure you have some settings in the config.  i.e 10% cache used.

2.) Change the time slightly and hit apply.

3.) If changing your configs don't move anything, re-install, and then update the file found in my post from a week or so ago. (This will allow us to see logs better. and set some of the config settings. (link embedded below)

 

Or, try the "manual mover, caution moves everything button in the lower right of the config section, this calls the original mover code.

 

 

        

 

 

Link to post

HI 

Back to having major issues with plugin updated and now keeps screwing up system

I cannot get Mover to work from button i get 

Apr 8 13:46:40 Bearcave root: cat: /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg: No such file or directory
Apr 8 13:46:40 Bearcave root: cat: /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg: No such file or directory
Apr 8 13:46:40 Bearcave root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 181: [: 26: unary operator expected
Apr 8 13:46:40 Bearcave root: cat: /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg: No such file or directory
Apr 8 13:46:40 Bearcave root: cat: /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg: No such file or directory
Apr 8 13:46:40 Bearcave root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 181: [: 26: unary operator expected
Apr 8 13:46:40 Bearcave root: cat: /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg: No such file or directory
Apr 8 13:46:40 Bearcave root: cat: /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg: No such file or directory
Apr 8 13:46:40 Bearcave root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 181: [: 26: unary operator expected
Apr 8 13:46:40 Bearcave root: mover: finished
Apr 8 13:47:19 Bearcave ntpd[1601]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized

Link to post
40 minutes ago, OsoPolar said:

HI 

Back to having major issues with plugin updated and now keeps screwing up system

I cannot get Mover to work from button i get 

Apr 8 13:46:40 Bearcave root: cat: /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg: No such file or directory
Apr 8 13:46:40 Bearcave root: cat: /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg: No such file or directory
Apr 8 13:46:40 Bearcave root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 181: [: 26: unary operator expected
Apr 8 13:46:40 Bearcave root: cat: /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg: No such file or directory
Apr 8 13:46:40 Bearcave root: cat: /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg: No such file or directory
Apr 8 13:46:40 Bearcave root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 181: [: 26: unary operator expected
Apr 8 13:46:40 Bearcave root: cat: /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg: No such file or directory
Apr 8 13:46:40 Bearcave root: cat: /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg: No such file or directory
Apr 8 13:46:40 Bearcave root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 181: [: 26: unary operator expected
Apr 8 13:46:40 Bearcave root: mover: finished
Apr 8 13:47:19 Bearcave ntpd[1601]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized

Looks like maybe you haven't set any config items?  Set the days old to 1, save, and then try and see if it moves files that older than a day.  (Original Manual Mover is now in the bottom right.  i.e. it calls the standard unraid mover.)

Link to post
1 hour ago, hugenbdd said:

Looks like maybe you haven't set any config items?  Set the days old to 1, save, and then try and see if it moves files that older than a day.  (Original Manual Mover is now in the bottom right.  i.e. it calls the standard unraid mover.)

HI 

I changed from 0% to 5% and this kicked it moving but still not sure it is actually working properly. It keeps filling up and stopping please see my settings attached. It had been running 30 hours i had to reboot it because it was moving nothing but Move remained greyed out.

 

You can see i already have 1 day set!

 

Note even though changing the % or time get it moving. When it finishes if i go back and hit move again it fails! Whatever i change now it makes no difference will not manually move

 

New log

Dell - Screenshot 08-04-2021 - 03.png

Edited by OsoPolar
Link to post

I have a question - For "Move files off cache based on age?" setting, it seems like the plugin checks the "Last Modified" date which corresponds to when the file was first created, even if that's before you downloaded it.  So if you download something older than your "days old" setting, it gets moved to the array on the very next Mover run.

 

Would it be possible to reference either the "Date Changed" or "Date Accessed" dates instead?  Seems like either of those are more representative of when the files are actually acquired on the cache.  At least in my use case.

 

Thank you!

Edited by acosmichippo
Link to post
2 hours ago, acosmichippo said:

I have a question - For "Move files off cache based on age?" setting, it seems like the plugin checks the "Last Modified" date which corresponds to when the file was first created, even if that's before you downloaded it.  So if you download something older than your "days old" setting, it gets moved to the array on the very next Mover run.

 

Would it be possible to reference either the "Date Changed" or "Date Accessed" dates instead?  Seems like either of those are more representative of when the files are actually acquired on the cache.  At least in my use case.

 

Thank you!

If you can give me an example in a find command, I can incorporate it.

 

We went over the "access date" about a year ago.  (Way back in this thread).  I believe we can't use that as the share's don't use atime, I can't remember exactly, but it wasn't possible.

 

You may want to add a "touch" to the files you extract to set the current time on them.  This should keep the issue from happening for you.  Some applications allow custom script/commands once a file is extracted.

Link to post

I'm not sure if its just me or what. Running 6.9.1, but I also noticed this at 6.9 too. 

 

I set my Mover Schedule to hourly and hourly my drives spin up and nothing happens since my Settings for Mover Tuning are set to 10% at 20Days. Of course my files are not 20days old so they shouldn't be moved, but why spin up some drives?

 

I decided maybe its just a fluke. So I set my mover to move at 21:35. I waited and sure enough my drives spun up and still nothing moved because it shouldn't be moved since my files are not 20days old. 

 

This is what is in my Logs at that very moment:

Apr 8 21:35:08 Tower emhttpd: read SMART /dev/sdg
Apr 8 21:35:30 Tower emhttpd: read SMART /dev/sdf
Apr 8 21:35:31 Tower emhttpd: read SMART /dev/sdd
Apr 8 21:35:38 Tower emhttpd: read SMART /dev/sdc

 

Odd thing is its only spinning up (4) of the (6) Drives (1) being parity and the other (3) are data drives. The drives that are being spun up very well could be where the data was to be stored "if" they was moved.  

Link to post
On 4/7/2021 at 9:13 PM, hugenbdd said:

Few things.

 

1.) Make sure you have some settings in the config.  i.e 10% cache used.

2.) Change the time slightly and hit apply.

3.) If changing your configs don't move anything, re-install, and then update the file found in my post from a week or so ago. (This will allow us to see logs better. and set some of the config settings. (link embedded below)

 

Or, try the "manual mover, caution moves everything button in the lower right of the config section, this calls the original mover code.

 

 

        

 

 

that worked...well 1 & 2 worked.  Thanks!

Link to post

But since updating the plugin it seems to be completely broken :(

usually, mover moves my items if the cache hits 70% or more. sometimes i hit the manual mover button in the Main page.

But i noticed it didn't do anything when i hit the manual Move button.

Then my cache went over 70% and it did nothing, but the "Original Move Now - Caution moves everything" does work appears to be the only thing that does work for mover. i pressed it after it went to 90%.

 

My cache filled with files within a 12 hour period, but i have age set to no.

 

Capture.PNG

Edited by Maticks
Link to post
11 minutes ago, Maticks said:

But since updating the plugin it seems to be completely broken :(

usually, mover moves my items if the cache hits 70% or more. sometimes i hit the manual mover button in the Main page.

But i noticed it didn't do anything when i hit the manual Move button.

Then my cache went over 70% and it did nothing, but the "Original Move Now - Caution moves everything" does work appears to be the only thing that does work for mover. i pressed it after it went to 90%.

 

My cache filled with files within a 12 hour period, but i have age set to no.

 

Capture.PNG

Yup, that should be moving when it hits 70%.

You can either install the file I linked to a while back that put's all the "echo" statements back in the logs, so I can see what's happening, or wait a week.  I plan to update how log statements are made with the plug-in.

Link to post
1 minute ago, hugenbdd said:

Yup, that should be moving when it hits 70%.

You can either install the file I linked to a while back that put's all the "echo" statements back in the logs, so I can see what's happening, or wait a week.  I plan to update how log statements are made with the plug-in.

I can press the manual move button for a week that's fine :)

Link to post
9 minutes ago, OsoPolar said:

Did you get chance to read my reply

Hi

You can copy the file I posted a while back to enable the echo/log statements, or wait a week till I update to handle the logs better.

 

Not much I can troubleshoot at the moment without the log statements.  Once I have the log statements, we can test the "find" command that it creates from the console to see why it may not be working as expected.

Link to post
13 hours ago, hugenbdd said:

If you can give me an example in a find command, I can incorporate it.

 

Like this?  For these files, the "atime" and "ctime" are both less than 10 days, but the "mtime" is greater than 10 days.  If "atime" cannot be used, can "ctime"?

 

oot@unraid:/mnt/user/Media/TV Shows/Stanley Tucci - Searching for Italy/Season 01# find *.mkv -atime -10
Stanley Tucci - Searching for Italy - S01E01 - Naples and the Amalfi Coast - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E02 - Rome - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E03 - Bologna - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E04 - Milan - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E05 - Tuscany - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E06 - Sicily - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv

root@unraid:/mnt/user/Media/TV Shows/Stanley Tucci - Searching for Italy/Season 01# find *.mkv -ctime -10
Stanley Tucci - Searching for Italy - S01E01 - Naples and the Amalfi Coast - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E02 - Rome - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E03 - Bologna - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E04 - Milan - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E05 - Tuscany - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E06 - Sicily - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv

root@unraid:/mnt/user/Media/TV Shows/Stanley Tucci - Searching for Italy/Season 01# find *.mkv -mtime +10
Stanley Tucci - Searching for Italy - S01E01 - Naples and the Amalfi Coast - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E02 - Rome - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E03 - Bologna - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E04 - Milan - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E05 - Tuscany - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E06 - Sicily - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv

 

Link to post
On 4/6/2021 at 7:16 PM, bobokun said:

Would it be possible to choose which cache pool to look at for the



Only move at this threshold of used cache space

I currently have a cache pool called "cache" which I use to store my appdata and for my downloads I use a cache pool called "scratch" and want the mover to be invoked when the downloads cache gets close to 75% full.

Actually had instructions in a past release.... 

 

- age_mover bash script now check for global threshold percent or a manual entry in the ca.mover.tuning.cfg file. (example: /boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg   - Entry of   cachetv="65")

- "cachetv" being the name of the pool.

- if using single digits leading zero required. (i.e. cachetv="01" for 1 percent)

 

This will over-ride the "global" percentage selected in the GUI for the cache pool (or pools) entered in the config file.

Edited by hugenbdd
fixed path
Link to post
On 4/9/2021 at 12:45 AM, kizer said:

Of course my files are not 20days old so they shouldn't be moved, but why spin up some drives?

Unless you cache-dirs, how can it know how old your files are? And cache-dirs isn't guaranteed to always have that information available in RAM since RAM might be needed for something else.

Link to post
1 hour ago, trurl said:

Unless you cache-dirs, how can it know how old your files are? And cache-dirs isn't guaranteed to always have that information available in RAM since RAM might be needed for something else.

 

Because I thought its supposed to be looking at the age of the files just on my SSD. Maybe I've gone a little bonkers lol. Unless your telling me its evaluating the files on the share not the files on the SSD which would cause drive spin up. If that is the case then I should set this plugin to check when I know my drives are already spun up say during a Plex Maintenance check. Normally that check is Daily 6AM-8AM. 

 

If it is looking at files from the "Share" not the Cache/SSD that would make total since of what is going on. However its spinning up my Parity for no reason if it isn't moving, unless that is just a characteristic of its preparing to move, but it actually doesn't.  

 

Either way I'm not complaining. I'm just trying to understand why its doing something. If its doing exactly what its supposed to be and the way its supposed to be I'll adapt. ;)

 

I'll throw some more files on my machine and test the plugin again. If it spins up specific drives based on what I placed on my Cache then I'll know for sure that its checking shares not just the SSD and I'll adjust accordingly. 

 

Link to post
57 minutes ago, kizer said:

 

Because I thought its supposed to be looking at the age of the files just on my SSD. Maybe I've gone a little bonkers lol. Unless your telling me its evaluating the files on the share not the files on the SSD which would cause drive spin up. If that is the case then I should set this plugin to check when I know my drives are already spun up say during a Plex Maintenance check. Normally that check is Daily 6AM-8AM. 

 

If it is looking at files from the "Share" not the Cache/SSD that would make total since of what is going on. However its spinning up my Parity for no reason if it isn't moving, unless that is just a characteristic of its preparing to move, but it actually doesn't.  

 

Either way I'm not complaining. I'm just trying to understand why its doing something. If its doing exactly what its supposed to be and the way its supposed to be I'll adapt. ;)

 

I'll throw some more files on my machine and test the plugin again. If it spins up specific drives based on what I placed on my Cache then I'll know for sure that its checking shares not just the SSD and I'll adjust accordingly. 

 

Not sure what in the plug-in could be causing this. 

 

There is a "Find" command, but that is based on the cache path.

 

Also, it checks the percentage of disk used on the cache pool.

Link to post
6 minutes ago, hugenbdd said:

Not sure what in the plug-in could be causing this. 

 

There is a "Find" command, but that is based on the cache path.

 

Also, it checks the percentage of disk used on the cache pool.

 

Ok, I'm not nuts then. Lol

Like I said I'm not complaining I just wanted to bring it up unless I'm the only one seeing this. My setup isn't anything special and Im pointing everything at the Cache Pool.

I'm running two pools, but this was happening before I added the second SSD pool of 1 drive with 6.9.1 aka ssdBackup SSD. 

 

I don't recall this happening at all when I was running 6.8.3, but I noticed it when I installed it on 6.9.1 and I thought it was a bug, but nobody else mentioned it so I uninstalled it thinking it was possibly something new in the OS and just waited to see if anybody else pointed it out. 

Link to post

New release to enable logging based on the standard Mover Setting.

 

2021.04.14

Logs now based off of standard "Mover Settings" for "Mover Logging:" of enabled or disabled. (Very detailed logs)

Changed text for field description of "Force move of All files on a schedule:" to "Force move of files on a schedule:"

 

image.png.619ef685c3015deb67f669126289a3a0.png

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.