[Plugin] Mover Tuning

[naturalcarr]

14 minutes ago, trurl said:

By "anymore", do you mean it did work but now it doesn't?

 

The Unraid OS is in RAM and a fresh copy is loaded from flash at each boot. So any change you might have made in /usr doesn't survive reboot.

 

Not quite, I meant nothing I was adding to mover file in /usr/local/sbin was being executed, but I now see that there's another mover file in usr/local/emhttp/plugins/ca.mover.tuning and my scripts ran when I added them there, I was a bit of an idiot and didn't look around enough. I didn't know about /usr being volatile, thank you for telling me, I'll add a line to my "go" file to over-write the mover on boot.

 

EDIT:

So I've been a bigger idiot than I originally anticipated, I've been adding things to /sbin this whole time, I accidentally copied the whole directory to root at some point today and have been in there this whole time.

Edited December 28, 2019 by naturalcarr

[jebusfreek666]

On 12/19/2018 at 1:09 PM, Squid said:

Once again, its exactly what it does.

 

You set mover to run every hour.

 

In the tuner settings, you tell it to only move at 90%.  And you also tell it to do a force move (via a cron schedule) once per day.

I am trying to set this up just like this. Check hourly, if above 90% move. I also want it to move daily at 4 am. I see the part about setting it up to force move "via a cron schedule" but have no idea what this is or how to do it. Can someone please help?

 

Edit:

After googling a bit, I think i would just input "0 4 * * *", does that look right?

Edited February 1, 2020 by jebusfreek666

[Squid]

Yup

[hugenbdd]

First, I would like to thank your for a great script!

 

Secondly, any thoughts about adding a section to move files based on how old they are? (I didn't notice it being brought up in the previous 5 pages of this thread)

 

I love to code and "play" with things, so hopefully I'm not stepping on your feet here with the code below.  If I am, let me know and I'll delete the post.

 

Modify the Schedules page to add two entries. (Age yes/no, and days old 0-200)

Move files off cache based on age?
: <select name="age" size='1' class='narrow'>
  <?=mk_option($cfg['age'],'yes','Yes')?>
        <?=mk_option($cfg['age'],'no','No')?>
</select>

> Select if you want to move files off of the cache based on their age or days old.

Move files that are greather than this many days old:
: <select name="daysold" size="1" class='narrow'>
<?for ($dt=0;$do<200;$do+=5):?>
<?=mk_option($cfg['daysold'], $do, "$do")?>
<?endfor;?>
</select>

> Select the number of days old a file has to be in order to move.

Update the mover.php to call a "new" mover script modified for the mtime command.

	if ( $cfg['enableTurbo'] == "yes" ) {
		logger("Forcing turbo write on");
		exec("/usr/local/sbin/mdcmd set md_write_method 1");
	}


	if ($cfg['age'] == "yes" ) {

		$niceLevel = $cfg['moverNice'] ?: "0";
		$ioLevel = $cfg['moverIO'] ?: "-c 2 -n 0";
		logger("ionice $ioLevel nice -n $niceLevel /usr/local/sbin/age_mover start");
		passthru("ionice $ioLevel nice -n $niceLevel /usr/local/sbin/age_mover start {$cfg['daysold']}");

		if ( $cfg['enableTurbo'] == "yes" ) {
			logger("Restoring original turbo write mode");
			exec("/usr/local/sbin/mdcmd set md_write_method {$vars['md_write_method']}");
		}
	}
	else {

	$niceLevel = $cfg['moverNice'] ?: "0";
	$ioLevel = $cfg['moverIO'] ?: "-c 2 -n 0";
	logger("ionice $ioLevel nice -n $niceLevel /usr/local/sbin/mover.old $options");
	passthru("ionice $ioLevel nice -n $niceLevel /usr/local/sbin/mover.old $options");

	if ( $cfg['enableTurbo'] == "yes" ) {
		logger("Restoring original turbo write mode");
		exec("/usr/local/sbin/mdcmd set md_write_method {$vars['md_write_method']}");
	}	
}

 

 

New "mover" script would have to be created, or at least modified the "mover.old" file to look for a second argument of "age".

Add "AGE" variable near the top of the script.

AGE=$2

 

Change the "Find" command, near line 70 to include "files only", and the mtime.

  # Check for objects to move from cache to array
  for SHAREPATH in /mnt/cache/*/ ; do
    SHARE=$(basename "$SHAREPATH")
    if grep -qs 'shareUseCache="yes"' "/boot/config/shares/${SHARE}.cfg" ; then
      find "${SHAREPATH%/}" -depth -type f -mtime +${AGE} | /usr/local/sbin/move -d $LOGLEVEL
    fi
  done

 

 

Thoughts?

Any issues I'm not seeing?

 

 

[Squid]

43 minutes ago, hugenbdd said:

Secondly, any thoughts about adding a section to move files based on how old they are? (I didn't notice it being brought up in the previous 5 pages of this thread)

I'm very hesitant to actually start modifying / replacing system files to accomplish that.  Feel free to modify and/or fork this yourself though.

[hugenbdd]

20 hours ago, Squid said:

I'm very hesitant to actually start modifying / replacing system files to accomplish that.  Feel free to modify and/or fork this yourself though.

Okay.

I forked it, and was able to get the plugin installed.

Can I PM you with the fork to review?  Would love your feedback! 

 

This is my first attempt at making a plug-in, and I only made minor edits to your files in my fork.  I also, didn't modify the system files.  I made a new "mover" file with my edits in the plugin directory.  So if Limetech modifies the file I would have to update my version.

[Squid]

Sure

[hugenbdd]

Thanks for posting the update and review Squid!

 

Updated from Community Applications, and it should look similar to this

 

I added two new fields to Squid's section of Mover Tuning.

1. Yes/No - move files off cache based on age?
2. 5-355 - Move files that are equal or greater than the number of days old.

 

I have set my mover schedule to daily for 3AM and I also turned on my logs to review this new code.

 

 

This modification adds a new mover script modified from Unraid's original mover.  The basic modification is to add the mtime flag to the find command that is piped into the mover binary.

Original find code inside the mover script

     find "${SHAREPATH%/}" -depth | /usr/local/sbin/move -d $LOGLEVEL

Modified find code

     find "${SHAREPATH%/}" -depth -mtime +${AGE} | /usr/local/sbin/move -d $LOGLEVEL

 

 

Mover Tuning should work the same as it ever has if you leave the "Move files off cache based on age?" to "No"

 

Some may ask why I have done this mod, it's so I can keep my most recent files on the cache where they will age and move off as they get older.  This allows me to keep my drives spun down most of the time.

 

Let me know if you run into any issues.

 

 

 

 

 

[Squid]

Just now, hugenbdd said:

Thanks for posting the update and review Squid!

 

Forgot to post about this.  @hugenbdd has taken over maintenance of this plugin, with some nice new features.  However, in order to switch to this new version, you do have to uninstall this app, and then reinstall from CA.  (Because of the differing installation URLs, an update will not appear while still on the original)

[danktankk]

much appreciated.  installed without any issues.

[SpeedyVV]

@hugenbdd, great running start! Installed.

 

I would like to ask for an additional feature/field:

    - Move files off cache based on age of the last time the file was accessed.

 

My use case is based on the fact that I have a lot of old files that are commonly used.

 

Thanks for your consideration.

[hugenbdd]

2 hours ago, SpeedyVV said:

@hugenbdd, great running start! Installed.

 

I would like to ask for an additional feature/field:

    - Move files off cache based on age of the last time the file was accessed.

 

My use case is based on the fact that I have a lot of old files that are commonly used.

 

Thanks for your consideration.

I might be able to do this based on the "atime" of a file.  The only concern I have is that there may be other processes that interact with the file resetting it.

 

I did a small test with a file in the /tmp directory and ls -lu will show the last read time.  However, it appears certain action won't change the atime.  Such as moving the file to a different directory.

 

I will take a look at adding it as an option.

 

[Squid]

2 minutes ago, hugenbdd said:

I might be able to do this based on the "atime" of a file.  The only concern I have is that there may be other processes that interact with the file resetting it.

 

I did a small test with a file in the /tmp directory and ls -lu will show the last read time.  However, it appears certain action won't change the atime.  Such as moving the file to a different directory.

 

I will take a look at adding it as an option.

 

The shares are (and the disks) are not mounted to support atime

Feb 13 15:58:26 unraidA emhttpd: shcmd (139): /usr/local/sbin/shfs /mnt/user -disks 4194303 1024 -o noatime,allow_other -o remember=0  |& logger

 

[hugenbdd]

9 minutes ago, Squid said:

The shares are (and the disks) are not mounted to support atime

Feb 13 15:58:26 unraidA emhttpd: shcmd (139): /usr/local/sbin/shfs /mnt/user -disks 4194303 1024 -o noatime,allow_other -o remember=0  |& logger

 

Thanks for the quick reply!

[SpeedyVV]

24 minutes ago, Squid said:

The shares are (and the disks) are not mounted to support atime

Feb 13 15:58:26 unraidA emhttpd: shcmd (139): /usr/local/sbin/shfs /mnt/user -disks 4194303 1024 -o noatime,allow_other -o remember=0  |& logger

 

I tried "stat" and it works. Could that be used?

 

Quote

root@Rui-unRAID:/mnt/user/appdata/dupeGuru/log/nginx# stat -c %x access.log
2020-03-06 17:05:41.272210428 +0000

Quote

root@Rui-unRAID:/mnt/user/appdata/dupeGuru/log/nginx# stat access.log
  File: access.log
  Size: 12225           Blocks: 24         IO Block: 4096   regular file
Device: 29h/41d Inode: 10977524091716103  Links: 1
Access: (0644/-rw-r--r--)  Uid: (   99/  nobody)   Gid: (  100/   users)
Access: 2020-03-06 17:05:41.272210428 +0000
Modify: 2020-03-13 20:04:21.812377107 +0000
Change: 2020-03-13 20:04:21.812377107 +0000
 Birth: -

 

Edited March 14, 2020 by SpeedyVV

[hugenbdd]

17 minutes ago, SpeedyVV said:

I tried "stat" and it works. Could that be used?

 

 

That is not working for me..

Your access is 7 days before your creation/modified entries....

[SpeedyVV]

I have my UnRaid on reverse time 😋... actually was wondering about that myself.

Strange, just launched the console from the Unraid menu, typed "stat" and it worked.

[wgstarks]

looks like a minor bug-

I can specify an age for files to be moved based on age even when this feature is disabled.

 

 

The same is not true for forcing a move on a schedule. I have to enable the feature before I can specify a schedule.

 

 

[loftyDan]

I love your new version, specifically the ability to let files rest on the cache for a while.  Thank you for your work.  Is there a way to use ages that aren't in the dropdown menu?  Right now, I can only choose none, 5, 10 or more days for example.  My cache space is a little bit tight, and I'd love to use 3 days for example.

[hugenbdd]

3 hours ago, wgstarks said:

looks like a minor bug-

I can specify an age for files to be moved based on age even when this feature is disabled.

 

 

The same is not true for forcing a move on a schedule. I have to enable the feature before I can specify a schedule.

 

 

Fixed in update. (3.22.2020)

[hugenbdd]

1 hour ago, loftyDan said:

I love your new version, specifically the ability to let files rest on the cache for a while.  Thank you for your work.  Is there a way to use ages that aren't in the dropdown menu?  Right now, I can only choose none, 5, 10 or more days for example.  My cache space is a little bit tight, and I'd love to use 3 days for example.

Added 1-4 as a selection in latest update. (today)

3.22.2020

[loftyDan]

55 minutes ago, hugenbdd said:

Added 1-4 as a selection in latest update. (today)

3.22.2020

Awesome. My cache drive and I thank you. 

[guythnick]

Huge fan of your update to add the X days old setting to Mover.  I have been trying to code up a very similar implementation of this myself.  This is much better than using a user script to do the job.

 

One thing that I think would be very beneficial (at least for me), would be to add another option to NOT move files under X MB.  I use this for .jpeg, .tbn, .nfo, .srt, etc. so that media server software will grab all of this from cache rather than spinning up the array.  I currently have a script that runs after mover that will copy back the files under a certain threshold. Similar to below:

 

rsync -avrt --info=progress2 --append-verify --max-size=3m "/mnt/disk2/media/Films/" "/mnt/cache/media/Films/"

Would you be able to add in a filter to not move files under a specified size?  It would eliminate my need for the after script if so, and I think it beneficial for many users that mainly use the array as a media server.

Edited March 23, 2020 by guythnick

[hugenbdd]

2 hours ago, guythnick said:

Huge fan of your update to add the X days old setting to Mover.  I have been trying to code up a very similar implementation of this myself.  This is much better than using a user script to do the job.

 

One thing that I think would be very beneficial (at least for me), would be to add another option to NOT move files under X MB.  I use this for .jpeg, .tbn, .nfo, .srt, etc. so that media server software will grab all of this from cache rather than spinning up the array.  I currently have a script that runs after mover that will copy back the files under a certain threshold. Similar to below:

 

rsync -avrt --info=progress2 --append-verify --max-size=3m "/mnt/disk2/media/Films/" "/mnt/cache/media/Films/"

Would you be able to add in a filter to not move files under a specified size?  It would eliminate my need for the after script if so, and I think it beneficial for many users that mainly use the array as a media server.

This should be doable.  However, I need to think a bit on how to present the option.  

 

Should this be run independent as well as with the age options?  So someone may not care about the age option, but does care about the size option?  And someone else may want to run the age option along with the size option..   The later being easier right now.

 

The reason this should be do-able is that the files sent over to the mover, are just piped in with a find command.  So any options that find has, I should be able to add.

 

Just wondering, what size files are you not moving?  under 1 MB?

[guythnick]

Quote

Should this be run independent as well as with the age options?

Yes, I think should be independent of the age option, but both can be used.

Quote

Just wondering, what size files are you not moving?

Under 3MB seems to capture all the images and subtitle files that I want to keep on the cache.

 

If it is piping into the find command, I believe you really only need to add one switch to the command when the option is selected:

 

-size +xM 

where x would be the integer in megabytes.