[Plugin] Mover Tuning


Recommended Posts

On 4/9/2021 at 10:26 AM, acosmichippo said:

 

Like this?  For these files, the "atime" and "ctime" are both less than 10 days, but the "mtime" is greater than 10 days.  If "atime" cannot be used, can "ctime"?

 


oot@unraid:/mnt/user/Media/TV Shows/Stanley Tucci - Searching for Italy/Season 01# find *.mkv -atime -10
Stanley Tucci - Searching for Italy - S01E01 - Naples and the Amalfi Coast - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E02 - Rome - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E03 - Bologna - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E04 - Milan - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E05 - Tuscany - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E06 - Sicily - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv

root@unraid:/mnt/user/Media/TV Shows/Stanley Tucci - Searching for Italy/Season 01# find *.mkv -ctime -10
Stanley Tucci - Searching for Italy - S01E01 - Naples and the Amalfi Coast - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E02 - Rome - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E03 - Bologna - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E04 - Milan - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E05 - Tuscany - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E06 - Sicily - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv

root@unraid:/mnt/user/Media/TV Shows/Stanley Tucci - Searching for Italy/Season 01# find *.mkv -mtime +10
Stanley Tucci - Searching for Italy - S01E01 - Naples and the Amalfi Coast - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E02 - Rome - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E03 - Bologna - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E04 - Milan - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E05 - Tuscany - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv
Stanley Tucci - Searching for Italy - S01E06 - Sicily - [WEBRip-1080p, , x264, 8-bit] [AAC, 2.0].mkv

 

I think mtime is still the "time" type we should be searching on.

 

From google:

Unix keeps 3 timestamps for each file: mtime, ctime, and atime. Most people seem to understand atime (access time), it is when the file was last read. There does seem to be some confusion between mtime and ctime though. ctime is the inode change time while mtime is the file modification time.

 

Maybe it could be an option in the settings....

Link to comment

Maybe I am not understanding the option right, but what is the added value of the "Force move of files on a schedule" if it does the exact same thing as the mover schedule under the normal mover settings.

They way I had this setup before was that the normal mover settings checked every hour if the cache was over a certain threshold and if it was it would move files off. That way the cache never became totally full and this was ideal to prevent issues with vm's, dockers or app data. But it would only do this if it was really needed.

And then in the night I had the force move all files running so it moved all files to the array when the server was not in use and was not bothering anyone.

They way the plugin is setup now there seems now way for me to keep this functionality?

Or am I missing something here or misunderstanding something? Is there a reason this is no longer possible? Would it not be possible to have the force move option invoke the old mover?

 

I am  asking to see how to best use this new plugin functionality as currently I might just not understand it.

Link to comment
56 minutes ago, TheIstar said:

Maybe I am not understanding the option right, but what is the added value of the "Force move of files on a schedule" if it does the exact same thing as the mover schedule under the normal mover settings.

They way I had this setup before was that the normal mover settings checked every hour if the cache was over a certain threshold and if it was it would move files off. That way the cache never became totally full and this was ideal to prevent issues with vm's, dockers or app data. But it would only do this if it was really needed.

And then in the night I had the force move all files running so it moved all files to the array when the server was not in use and was not bothering anyone.

They way the plugin is setup now there seems now way for me to keep this functionality?

Or am I missing something here or misunderstanding something? Is there a reason this is no longer possible? Would it not be possible to have the force move option invoke the old mover?

 

I am  asking to see how to best use this new plugin functionality as currently I might just not understand it.

 

Lets say you want all your files to move when your drive is 50%. So the Plugin will move your files when it is 50%

 

Lets say you want all of your files to move when your drive is 50%, but at the end of the month before Parity is ran you want all your files in the array regardless of how full your SSD is. You have that option by using the Move on a Schedule within the Plugin, but you can't move on a schedule with the original unraid mover because the Plugin supersedes it. 

 

in all sense the old Mover is not used anymore unless you click the Move button within the plugin settings in the bottom right corner which will trigger the default unraid mover. 

Link to comment
3 hours ago, hugenbdd said:

I think mtime is still the "time" type we should be searching on.

 

From google:

Unix keeps 3 timestamps for each file: mtime, ctime, and atime. Most people seem to understand atime (access time), it is when the file was last read. There does seem to be some confusion between mtime and ctime though. ctime is the inode change time while mtime is the file modification time.

 

Maybe it could be an option in the settings....

 

 

An option would be great if you can get around to it!  I neglected to mention all those files were "new" to me within the last 10 days, but the mtime was ~60 days ago.  So from my point of view it was just not accurate and mover moved them the very first opportunity despite my "age" setting at 10 days.

Edited by acosmichippo
Link to comment
9 hours ago, kizer said:

 

Lets say you want all your files to move when your drive is 50%. So the Plugin will move your files when it is 50%

 

Lets say you want all of your files to move when your drive is 50%, but at the end of the month before Parity is ran you want all your files in the array regardless of how full your SSD is. You have that option by using the Move on a Schedule within the Plugin, but you can't move on a schedule with the original unraid mover because the Plugin supersedes it. 

 

in all sense the old Mover is not used anymore unless you click the Move button within the plugin settings in the bottom right corner which will trigger the default unraid mover. 

 

Yes, it would be great if it worked like that, but as other people have also already pointed out, the move on a schedule does not seem to move all files anymore unless the cache is filled to the percentage you set. That is why I asked what the use of it is now.

Link to comment

New Release

###2021.04.15

- Changed text for field description of "Force move of files on a schedule:" to "Force move of all files on a schedule:"

- No log entry for this, only an email with the output will be sent, like most cron jobs.

 

 

Basically I put it back to how it used to work.    The cron entry in the configs now calls unraids original mover file.  None of the plug-in settings will be read (other than the time/s specified in the cron text field).

  • Like 1
  • Thanks 1
Link to comment
9 hours ago, TheIstar said:

 

Yes, it would be great if it worked like that, but as other people have also already pointed out, the move on a schedule does not seem to move all files anymore unless the cache is filled to the percentage you set. That is why I asked what the use of it is now.

 

Ah I get what your saying.

Looks like there was an update to address this right below your reply. 

  • Thanks 1
Link to comment

Saw a few posts on Reddit with cache dealing with locked files, or downloaders getting slowed down during move.

 

Would it be helpful if I added two new sections to the tuner?

 

- Script to run before mover starts 

 

- Script to run when mover ends

 

I'm thinking this would allow people to shut down dockers or pause downloaders.

This will probably run before checking any of the "bounds".  i.e. if a pool cache % used is hit or not.

 

Thoughts?

Link to comment

Honestly I see no downside to offering a field to add a call out to a script before starting or ending. I think it would be rather useful honestly. 

 

Obviously as long as when it does its check which some of us have to set to Hourly, Daily, Weekly or whatever it doesn't trigger the scripts unless it really does the move. lol

Link to comment
4 hours ago, hugenbdd said:

New Release

###2021.04.15

- Changed text for field description of "Force move of files on a schedule:" to "Force move of all files on a schedule:"

- No log entry for this, only an email with the output will be sent, like most cron jobs.

 

 

Basically I put it back to how it used to work.    The cron entry in the configs now calls unraids original mover file.  None of the plug-in settings will be read (other than the time/s specified in the cron text field).

 

This is awesome! Thanks for that. :)

Link to comment
1 hour ago, kizer said:

Honestly I see no downside to offering a field to add a call out to a script before starting or ending. I think it would be rather useful honestly. 

 

Obviously as long as when it does its check which some of us have to set to Hourly, Daily, Weekly or whatever it doesn't trigger the scripts unless it really does the move. lol

At the moment, it would run before doing any checks at all.

 

In the future, if mover changes like I think it will, then I can incorporate a running of the scripts if or not depending on if something moves.

Link to comment
21 hours ago, hugenbdd said:

Saw a few posts on Reddit with cache dealing with locked files, or downloaders getting slowed down during move.

 

Would it be helpful if I added two new sections to the tuner?

 

- Script to run before mover starts 

 

- Script to run when mover ends

 

I'm thinking this would allow people to shut down dockers or pause downloaders.

This will probably run before checking any of the "bounds".  i.e. if a pool cache % used is hit or not.

 

Thoughts?

New Update

2021.04.16

Added "Script to run before mover" text field. Enter the path to a custom script to run before mover starts.

Added "Script to run after mover" text field. Enter the path to a custom script to run after mover finishes.

 

 

These will ALWAYS run, even if the filters remove all possible files.  i.e. "Mover not needed".  I plan to change this in the future to only run if something is found.  But that is months away at the earliest.

 

Should probably stay away from spaces in the path/filename.  I only tested with a simple "Hello World" script.

 

next up, adding Ctime/Mtime option.

  • Thanks 2
Link to comment

I've found an issue with this plugin, at least on my setup, and i'm not sure where this would be configured.

 

All my shares are in lowercase, no mixed cases, but the mover is failing to actually move anything with this plug-in enabled because it's seemingly doing some sort of .title() function and making the first letter uppercase.

 

Apr 19 04:20:20 server root: mvlogger: Share Name Only: Personal
Apr 19 04:20:20 server root: mvlogger: Cache Pool Name:
Apr 19 04:20:20 server root: mvlogger: No shareCachePool entry found in config file, defaulting to cache
Apr 19 04:20:20 server root: mvlogger: cache Threshold Pct:
Apr 19 04:20:20 server root: mvlogger: OVERALL Threshold: 0
Apr 19 04:20:20 server root: mvlogger: Share Path: /mnt/cache/Personal
Apr 19 04:20:20 server root: mvlogger: Pool Pct Used:  92 %
Apr 19 04:20:20 server root: mvlogger: DFTPCT LIMIT USED FOR SETTING: 0
Apr 19 04:20:20 server root: mvlogger: Threshold Used: 0
Apr 19 04:20:20 server root: mvlogger: Skipfiletypes string: find "/mnt/cache/Personal" -depth
Apr 19 04:20:20 server root: mvlogger: Complete Mover Command: find "/mnt/cache/Personal" -depth | /usr/local/sbin/move -d 1
Apr 19 04:20:20 server root: find: '/mnt/cache/Personal': No such file or directory

 

However, the mounted share is called personal.

root@server:/mnt/cache# ls -l /mnt/cache | grep -i person
drwxrwxrwx 1 nobody users   30 Feb 28 11:17 personal/

 

If I remove the plug-in, mover works fine, but with it installed, the mover fails to find and move any files.

 

EDIT:

I think I found where it's getting confused.

The share name is personal, it's configured as such in the UI

Screenshot here

 

However on /boot/config/shares it's stored as Personal.cfg.

It's one of my older shares, so perhaps the convention was different back then. I think perhaps it should just read what's in /mnt/cache and not from /boot/config/shares, if that's indeed what is happening. 

Edited by Klainn
Link to comment
5 hours ago, Klainn said:

I've found an issue with this plugin, at least on my setup, and i'm not sure where this would be configured.

 

All my shares are in lowercase, no mixed cases, but the mover is failing to actually move anything with this plug-in enabled because it's seemingly doing some sort of .title() function and making the first letter uppercase.

 


Apr 19 04:20:20 server root: mvlogger: Share Name Only: Personal
Apr 19 04:20:20 server root: mvlogger: Cache Pool Name:
Apr 19 04:20:20 server root: mvlogger: No shareCachePool entry found in config file, defaulting to cache
Apr 19 04:20:20 server root: mvlogger: cache Threshold Pct:
Apr 19 04:20:20 server root: mvlogger: OVERALL Threshold: 0
Apr 19 04:20:20 server root: mvlogger: Share Path: /mnt/cache/Personal
Apr 19 04:20:20 server root: mvlogger: Pool Pct Used:  92 %
Apr 19 04:20:20 server root: mvlogger: DFTPCT LIMIT USED FOR SETTING: 0
Apr 19 04:20:20 server root: mvlogger: Threshold Used: 0
Apr 19 04:20:20 server root: mvlogger: Skipfiletypes string: find "/mnt/cache/Personal" -depth
Apr 19 04:20:20 server root: mvlogger: Complete Mover Command: find "/mnt/cache/Personal" -depth | /usr/local/sbin/move -d 1
Apr 19 04:20:20 server root: find: '/mnt/cache/Personal': No such file or directory

 

However, the mounted share is called personal.


root@server:/mnt/cache# ls -l /mnt/cache | grep -i person
drwxrwxrwx 1 nobody users   30 Feb 28 11:17 personal/

 

If I remove the plug-in, mover works fine, but with it installed, the mover fails to find and move any files.

 

EDIT:

I think I found where it's getting confused.

The share name is personal, it's configured as such in the UI

Screenshot here

 

However on /boot/config/shares it's stored as Personal.cfg.

It's one of my older shares, so perhaps the convention was different back then. I think perhaps it should just read what's in /mnt/cache and not from /boot/config/shares, if that's indeed what is happening. 

Can you tell me what you have in this directory?

 /boot/config/pools/

 

Original mover loops through config files like this

  # Check for objects to move from pools to array
  for POOL in /boot/config/pools/*.cfg ; do
    for SHAREPATH in /mnt/$(basename "$POOL" .cfg)/*/ ; do
 

 

 

Where my mover loops through shares. (like this)

    for SHARECFG in /boot/config/shares/* ; do


        if grep -qs 'shareUseCache="yes"' "$SHARECFG" ; then
 

 

My gut reaction is that the file Personal.cfg should be renamed to personal.cfg.  however, I'm not sure what impact that will have on your shares.  Everything should be case sensitive.

Link to comment
root@server:~# ls -l /boot/config/pools/
total 32
-rw------- 1 root root 510 Apr 10 20:55 cache.cfg

root@server:~# cat /boot/config/pools/cache.cfg
diskFsType="btrfs"
diskComment=""
diskWarning=""
diskCritical=""
diskUUID="413d0ae9-06d1-4a97-80cd-34f008d70811"
diskShareEnabled="yes"
diskShareFloor="0"
diskExport="e"
diskFruit="no"
diskSecurity="public"
diskReadList=""
diskWriteList=""
diskVolsizelimit=""
diskCaseSensitive="auto"
diskExportNFS="-"
diskExportNFSFsid="10"
diskSecurityNFS="public"
diskHostListNFS=""
diskId="SanDisk_SDSSDH3_512G_202059801545"
diskIdSlot="-"
diskType="Cache"
diskSpindownDelay="-1"
diskSpinupGroup=""

 

That's the contents of the directory and file.

 

I figured what you recommended would be the fix, but like you, I'm not sure how that might impact things.

I've been using this plugin for many moons and it's only after a move on 3/11 or 3/21 that it stopped cleaning out the cache. I don't recall it ever being an issue before then.

 

Link to comment
3 hours ago, Klainn said:

root@server:~# ls -l /boot/config/pools/
total 32
-rw------- 1 root root 510 Apr 10 20:55 cache.cfg

root@server:~# cat /boot/config/pools/cache.cfg
diskFsType="btrfs"
diskComment=""
diskWarning=""
diskCritical=""
diskUUID="413d0ae9-06d1-4a97-80cd-34f008d70811"
diskShareEnabled="yes"
diskShareFloor="0"
diskExport="e"
diskFruit="no"
diskSecurity="public"
diskReadList=""
diskWriteList=""
diskVolsizelimit=""
diskCaseSensitive="auto"
diskExportNFS="-"
diskExportNFSFsid="10"
diskSecurityNFS="public"
diskHostListNFS=""
diskId="SanDisk_SDSSDH3_512G_202059801545"
diskIdSlot="-"
diskType="Cache"
diskSpindownDelay="-1"
diskSpinupGroup=""

 

That's the contents of the directory and file.

 

I figured what you recommended would be the fix, but like you, I'm not sure how that might impact things.

I've been using this plugin for many moons and it's only after a move on 3/11 or 3/21 that it stopped cleaning out the cache. I don't recall it ever being an issue before then.

 

Can you do an ls -ltr of this directory?   /boot/config/shares

 

I think we should reach out to unraid and see what they might suggest.  If shares should "EXACTLY" match their display names.  i.e. caps or not.  My guess is that it should match exactly.

Link to comment
14 minutes ago, hugenbdd said:

/boot/config/shares

Judging from diagnostics I have seen, this folder is often a mess. Lots of .cfg files that no longer correspond to any shares, lots of shares with no .cfg file (those have default setting of cache-no), some .cfg files that are duplicates except for upper/lower case

Link to comment
36 minutes ago, trurl said:

Judging from diagnostics I have seen, this folder is often a mess. Lots of .cfg files that no longer correspond to any shares, lots of shares with no .cfg file (those have default setting of cache-no), some .cfg files that are duplicates except for upper/lower case

Can Klainn rename his to lower case to match the display name?  Will there be any issues from doing that?

Link to comment
15 minutes ago, hugenbdd said:

Can Klainn rename his to lower case to match the display name?  Will there be any issues from doing that?

Typically what I have seen in diagnostics is the .cfg file that doesn't match case will not be used and that share will have default settings with no .cfg file. Anonymized diagnostics makes it difficult to sort that out though since the share names only show first and last characters with other replaced with -

 

If changes are made to those files outside the webUI then the array would need to be restarted to get them read in again.

 

Might be instructive to get a listing (as you asked for) of that folder and also a screenshot of the Shares page for comparison.

Link to comment
On 4/16/2021 at 3:13 PM, acosmichippo said:

 

sweet!  thank you so much.

New Release

 

2021.04.20

Added "CTIME" option under Age. This allows the "find" command to use ctime instead of mtime.

 

But, thinking I might need to tweak it again to remove the +.... (Please confirm if you use this option) 

Example find command:  find "/mnt/cache/Backup" -depth -ctime +30

  • Thanks 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.