naturalcarr 3 Posted December 28, 2019 Share Posted December 28, 2019 (edited) 14 minutes ago, trurl said: By "anymore", do you mean it did work but now it doesn't? The Unraid OS is in RAM and a fresh copy is loaded from flash at each boot. So any change you might have made in /usr doesn't survive reboot. Not quite, I meant nothing I was adding to mover file in /usr/local/sbin was being executed, but I now see that there's another mover file in usr/local/emhttp/plugins/ca.mover.tuning and my scripts ran when I added them there, I was a bit of an idiot and didn't look around enough. I didn't know about /usr being volatile, thank you for telling me, I'll add a line to my "go" file to over-write the mover on boot. EDIT: So I've been a bigger idiot than I originally anticipated, I've been adding things to /sbin this whole time, I accidentally copied the whole directory to root at some point today and have been in there this whole time. Edited December 28, 2019 by naturalcarr Quote Link to post
jebusfreek666 7 Posted February 1, 2020 Share Posted February 1, 2020 (edited) On 12/19/2018 at 1:09 PM, Squid said: Once again, its exactly what it does. You set mover to run every hour. In the tuner settings, you tell it to only move at 90%. And you also tell it to do a force move (via a cron schedule) once per day. I am trying to set this up just like this. Check hourly, if above 90% move. I also want it to move daily at 4 am. I see the part about setting it up to force move "via a cron schedule" but have no idea what this is or how to do it. Can someone please help? Edit: After googling a bit, I think i would just input "0 4 * * *", does that look right? Edited February 1, 2020 by jebusfreek666 Quote Link to post
hugenbdd 11 Posted February 27, 2020 Share Posted February 27, 2020 First, I would like to thank your for a great script! Secondly, any thoughts about adding a section to move files based on how old they are? (I didn't notice it being brought up in the previous 5 pages of this thread) I love to code and "play" with things, so hopefully I'm not stepping on your feet here with the code below. If I am, let me know and I'll delete the post. Modify the Schedules page to add two entries. (Age yes/no, and days old 0-200) Move files off cache based on age? : <select name="age" size='1' class='narrow'> <?=mk_option($cfg['age'],'yes','Yes')?> <?=mk_option($cfg['age'],'no','No')?> </select> > Select if you want to move files off of the cache based on their age or days old. Move files that are greather than this many days old: : <select name="daysold" size="1" class='narrow'> <?for ($dt=0;$do<200;$do+=5):?> <?=mk_option($cfg['daysold'], $do, "$do")?> <?endfor;?> </select> > Select the number of days old a file has to be in order to move. Update the mover.php to call a "new" mover script modified for the mtime command. if ( $cfg['enableTurbo'] == "yes" ) { logger("Forcing turbo write on"); exec("/usr/local/sbin/mdcmd set md_write_method 1"); } if ($cfg['age'] == "yes" ) { $niceLevel = $cfg['moverNice'] ?: "0"; $ioLevel = $cfg['moverIO'] ?: "-c 2 -n 0"; logger("ionice $ioLevel nice -n $niceLevel /usr/local/sbin/age_mover start"); passthru("ionice $ioLevel nice -n $niceLevel /usr/local/sbin/age_mover start {$cfg['daysold']}"); if ( $cfg['enableTurbo'] == "yes" ) { logger("Restoring original turbo write mode"); exec("/usr/local/sbin/mdcmd set md_write_method {$vars['md_write_method']}"); } } else { $niceLevel = $cfg['moverNice'] ?: "0"; $ioLevel = $cfg['moverIO'] ?: "-c 2 -n 0"; logger("ionice $ioLevel nice -n $niceLevel /usr/local/sbin/mover.old $options"); passthru("ionice $ioLevel nice -n $niceLevel /usr/local/sbin/mover.old $options"); if ( $cfg['enableTurbo'] == "yes" ) { logger("Restoring original turbo write mode"); exec("/usr/local/sbin/mdcmd set md_write_method {$vars['md_write_method']}"); } } New "mover" script would have to be created, or at least modified the "mover.old" file to look for a second argument of "age". Add "AGE" variable near the top of the script. AGE=$2 Change the "Find" command, near line 70 to include "files only", and the mtime. # Check for objects to move from cache to array for SHAREPATH in /mnt/cache/*/ ; do SHARE=$(basename "$SHAREPATH") if grep -qs 'shareUseCache="yes"' "/boot/config/shares/${SHARE}.cfg" ; then find "${SHAREPATH%/}" -depth -type f -mtime +${AGE} | /usr/local/sbin/move -d $LOGLEVEL fi done Thoughts? Any issues I'm not seeing? Quote Link to post
Squid 2894 Posted February 27, 2020 Author Share Posted February 27, 2020 43 minutes ago, hugenbdd said: Secondly, any thoughts about adding a section to move files based on how old they are? (I didn't notice it being brought up in the previous 5 pages of this thread) I'm very hesitant to actually start modifying / replacing system files to accomplish that. Feel free to modify and/or fork this yourself though. Quote Link to post
hugenbdd 11 Posted February 28, 2020 Share Posted February 28, 2020 20 hours ago, Squid said: I'm very hesitant to actually start modifying / replacing system files to accomplish that. Feel free to modify and/or fork this yourself though. Okay. I forked it, and was able to get the plugin installed. Can I PM you with the fork to review? Would love your feedback! This is my first attempt at making a plug-in, and I only made minor edits to your files in my fork. I also, didn't modify the system files. I made a new "mover" file with my edits in the plugin directory. So if Limetech modifies the file I would have to update my version. Quote Link to post
hugenbdd 11 Posted March 12, 2020 Share Posted March 12, 2020 Thanks for posting the update and review Squid! Updated from Community Applications, and it should look similar to this I added two new fields to Squid's section of Mover Tuning. Yes/No - move files off cache based on age? 5-355 - Move files that are equal or greater than the number of days old. I have set my mover schedule to daily for 3AM and I also turned on my logs to review this new code. This modification adds a new mover script modified from Unraid's original mover. The basic modification is to add the mtime flag to the find command that is piped into the mover binary. Original find code inside the mover script find "${SHAREPATH%/}" -depth | /usr/local/sbin/move -d $LOGLEVEL Modified find code find "${SHAREPATH%/}" -depth -mtime +${AGE} | /usr/local/sbin/move -d $LOGLEVEL Mover Tuning should work the same as it ever has if you leave the "Move files off cache based on age?" to "No" Some may ask why I have done this mod, it's so I can keep my most recent files on the cache where they will age and move off as they get older. This allows me to keep my drives spun down most of the time. Let me know if you run into any issues. Quote Link to post
Squid 2894 Posted March 12, 2020 Author Share Posted March 12, 2020 Just now, hugenbdd said: Thanks for posting the update and review Squid! Forgot to post about this. @hugenbdd has taken over maintenance of this plugin, with some nice new features. However, in order to switch to this new version, you do have to uninstall this app, and then reinstall from CA. (Because of the differing installation URLs, an update will not appear while still on the original) Quote Link to post
danktankk 5 Posted March 13, 2020 Share Posted March 13, 2020 much appreciated. installed without any issues. Quote Link to post
SpeedyVV 0 Posted March 14, 2020 Share Posted March 14, 2020 @hugenbdd, great running start! Installed. I would like to ask for an additional feature/field: - Move files off cache based on age of the last time the file was accessed. My use case is based on the fact that I have a lot of old files that are commonly used. Thanks for your consideration. Quote Link to post
hugenbdd 11 Posted March 14, 2020 Share Posted March 14, 2020 2 hours ago, SpeedyVV said: @hugenbdd, great running start! Installed. I would like to ask for an additional feature/field: - Move files off cache based on age of the last time the file was accessed. My use case is based on the fact that I have a lot of old files that are commonly used. Thanks for your consideration. I might be able to do this based on the "atime" of a file. The only concern I have is that there may be other processes that interact with the file resetting it. I did a small test with a file in the /tmp directory and ls -lu will show the last read time. However, it appears certain action won't change the atime. Such as moving the file to a different directory. I will take a look at adding it as an option. Quote Link to post
Squid 2894 Posted March 14, 2020 Author Share Posted March 14, 2020 2 minutes ago, hugenbdd said: I might be able to do this based on the "atime" of a file. The only concern I have is that there may be other processes that interact with the file resetting it. I did a small test with a file in the /tmp directory and ls -lu will show the last read time. However, it appears certain action won't change the atime. Such as moving the file to a different directory. I will take a look at adding it as an option. The shares are (and the disks) are not mounted to support atime Feb 13 15:58:26 unraidA emhttpd: shcmd (139): /usr/local/sbin/shfs /mnt/user -disks 4194303 1024 -o noatime,allow_other -o remember=0 |& logger Quote Link to post
hugenbdd 11 Posted March 14, 2020 Share Posted March 14, 2020 9 minutes ago, Squid said: The shares are (and the disks) are not mounted to support atime Feb 13 15:58:26 unraidA emhttpd: shcmd (139): /usr/local/sbin/shfs /mnt/user -disks 4194303 1024 -o noatime,allow_other -o remember=0 |& logger Thanks for the quick reply! Quote Link to post
SpeedyVV 0 Posted March 14, 2020 Share Posted March 14, 2020 (edited) 24 minutes ago, Squid said: The shares are (and the disks) are not mounted to support atime Feb 13 15:58:26 unraidA emhttpd: shcmd (139): /usr/local/sbin/shfs /mnt/user -disks 4194303 1024 -o noatime,allow_other -o remember=0 |& logger I tried "stat" and it works. Could that be used? Quote root@Rui-unRAID:/mnt/user/appdata/dupeGuru/log/nginx# stat -c %x access.log 2020-03-06 17:05:41.272210428 +0000 Quote root@Rui-unRAID:/mnt/user/appdata/dupeGuru/log/nginx# stat access.log File: access.log Size: 12225 Blocks: 24 IO Block: 4096 regular file Device: 29h/41d Inode: 10977524091716103 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 99/ nobody) Gid: ( 100/ users) Access: 2020-03-06 17:05:41.272210428 +0000 Modify: 2020-03-13 20:04:21.812377107 +0000 Change: 2020-03-13 20:04:21.812377107 +0000 Birth: - Edited March 14, 2020 by SpeedyVV Quote Link to post
hugenbdd 11 Posted March 14, 2020 Share Posted March 14, 2020 17 minutes ago, SpeedyVV said: I tried "stat" and it works. Could that be used? That is not working for me.. Your access is 7 days before your creation/modified entries.... Quote Link to post
SpeedyVV 0 Posted March 14, 2020 Share Posted March 14, 2020 I have my UnRaid on reverse time 😋... actually was wondering about that myself. Strange, just launched the console from the Unraid menu, typed "stat" and it worked. Quote Link to post
wgstarks 213 Posted March 22, 2020 Share Posted March 22, 2020 looks like a minor bug- I can specify an age for files to be moved based on age even when this feature is disabled. The same is not true for forcing a move on a schedule. I have to enable the feature before I can specify a schedule. Quote Link to post
loftyDan 0 Posted March 22, 2020 Share Posted March 22, 2020 I love your new version, specifically the ability to let files rest on the cache for a while. Thank you for your work. Is there a way to use ages that aren't in the dropdown menu? Right now, I can only choose none, 5, 10 or more days for example. My cache space is a little bit tight, and I'd love to use 3 days for example. Quote Link to post
hugenbdd 11 Posted March 22, 2020 Share Posted March 22, 2020 3 hours ago, wgstarks said: looks like a minor bug- I can specify an age for files to be moved based on age even when this feature is disabled. The same is not true for forcing a move on a schedule. I have to enable the feature before I can specify a schedule. Fixed in update. (3.22.2020) Quote Link to post
hugenbdd 11 Posted March 22, 2020 Share Posted March 22, 2020 1 hour ago, loftyDan said: I love your new version, specifically the ability to let files rest on the cache for a while. Thank you for your work. Is there a way to use ages that aren't in the dropdown menu? Right now, I can only choose none, 5, 10 or more days for example. My cache space is a little bit tight, and I'd love to use 3 days for example. Added 1-4 as a selection in latest update. (today) 3.22.2020 Quote Link to post
loftyDan 0 Posted March 22, 2020 Share Posted March 22, 2020 55 minutes ago, hugenbdd said: Added 1-4 as a selection in latest update. (today) 3.22.2020 Awesome. My cache drive and I thank you. Quote Link to post
guythnick 0 Posted March 23, 2020 Share Posted March 23, 2020 (edited) Huge fan of your update to add the X days old setting to Mover. I have been trying to code up a very similar implementation of this myself. This is much better than using a user script to do the job. One thing that I think would be very beneficial (at least for me), would be to add another option to NOT move files under X MB. I use this for .jpeg, .tbn, .nfo, .srt, etc. so that media server software will grab all of this from cache rather than spinning up the array. I currently have a script that runs after mover that will copy back the files under a certain threshold. Similar to below: rsync -avrt --info=progress2 --append-verify --max-size=3m "/mnt/disk2/media/Films/" "/mnt/cache/media/Films/" Would you be able to add in a filter to not move files under a specified size? It would eliminate my need for the after script if so, and I think it beneficial for many users that mainly use the array as a media server. Edited March 23, 2020 by guythnick Quote Link to post
hugenbdd 11 Posted March 24, 2020 Share Posted March 24, 2020 2 hours ago, guythnick said: Huge fan of your update to add the X days old setting to Mover. I have been trying to code up a very similar implementation of this myself. This is much better than using a user script to do the job. One thing that I think would be very beneficial (at least for me), would be to add another option to NOT move files under X MB. I use this for .jpeg, .tbn, .nfo, .srt, etc. so that media server software will grab all of this from cache rather than spinning up the array. I currently have a script that runs after mover that will copy back the files under a certain threshold. Similar to below: rsync -avrt --info=progress2 --append-verify --max-size=3m "/mnt/disk2/media/Films/" "/mnt/cache/media/Films/" Would you be able to add in a filter to not move files under a specified size? It would eliminate my need for the after script if so, and I think it beneficial for many users that mainly use the array as a media server. This should be doable. However, I need to think a bit on how to present the option. Should this be run independent as well as with the age options? So someone may not care about the age option, but does care about the size option? And someone else may want to run the age option along with the size option.. The later being easier right now. The reason this should be do-able is that the files sent over to the mover, are just piped in with a find command. So any options that find has, I should be able to add. Just wondering, what size files are you not moving? under 1 MB? Quote Link to post
guythnick 0 Posted March 24, 2020 Share Posted March 24, 2020 Quote Should this be run independent as well as with the age options? Yes, I think should be independent of the age option, but both can be used. Quote Just wondering, what size files are you not moving? Under 3MB seems to capture all the images and subtitle files that I want to keep on the cache. If it is piping into the find command, I believe you really only need to add one switch to the command when the option is selected: -size +xM where x would be the integer in megabytes. Quote Link to post
317 posts in this topic Last Reply
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.