hugenbdd

Threshold is the first thing it checks. It won't run mover at all if the threshold is not met. i.e. your cache is at 50% but you have it set to 60% in the config. It will not move 30 day old files before.. as it hasn't been "run". (see above) If your threshold is met, and you have "30" for the number of days old. Then any file older than 30 days old will be moved off. Anything younger will be left on the cache. This does mean that you can have some really old files on the cache if you are building up the files to meet your threshold. I would suggest setting the threshold to a very low number if you want to move files off without building them up on the cache drive.

It does. I would recommend updating.

posted an update that 1.) Removes the "echo" statements from the age_mover file. This should prevent the email to the admin. 2.) Checked the variable being passed in from the command line. (this should fix your issue chizll). Please let me know if you still run into issues.

jcato Please try this file. /usr/local/emhttp/plugins/ca.mover.tuning/age_mover permissions (755) -rwxr-xr-x 1 root root 5302 Apr 7 09:58 age_mover* If this works I'll release an update. age_mover

I think it's because I left some "echo" statements in the age_mover file. If I send you a new file can you replace it and test with results?

Updated (2020.03.28) release. Size can be run independently of the of the age setting. I was not able to test the condition on it's own due to the need to keep my current files in cache, but I did test age + size, and age. + I left some extra logging in the code for now. Which shouldn't hurt anything at this point. Let me know if you run into any issues. If I add any more commands to the "find", I think I will need a different route to program the selections. The only other type of modification I can think of right now, is to ignore certain file types. Such as .jpg or .gif files... But those should be ignored based on size.....

This should be doable. However, I need to think a bit on how to present the option. Should this be run independent as well as with the age options? So someone may not care about the age option, but does care about the size option? And someone else may want to run the age option along with the size option.. The later being easier right now. The reason this should be do-able is that the files sent over to the mover, are just piped in with a find command. So any options that find has, I should be able to add. Just wondering, what size files are you not moving? under 1 MB?

Added 1-4 as a selection in latest update. (today) 3.22.2020

Fixed in update. (3.22.2020)

That is not working for me.. Your access is 7 days before your creation/modified entries....

Thanks for the quick reply!

I might be able to do this based on the "atime" of a file. The only concern I have is that there may be other processes that interact with the file resetting it. I did a small test with a file in the /tmp directory and ls -lu will show the last read time. However, it appears certain action won't change the atime. Such as moving the file to a different directory. I will take a look at adding it as an option.

Thanks for posting the update and review Squid! Updated from Community Applications, and it should look similar to this I added two new fields to Squid's section of Mover Tuning. Yes/No - move files off cache based on age? 5-355 - Move files that are equal or greater than the number of days old. I have set my mover schedule to daily for 3AM and I also turned on my logs to review this new code. This modification adds a new mover script modified from Unraid's original mover. The basic modification is to add the mtime flag to the find command that is piped into the mover binary. Original find code inside the mover script find "${SHAREPATH%/}" -depth | /usr/local/sbin/move -d $LOGLEVEL Modified find code find "${SHAREPATH%/}" -depth -mtime +${AGE} | /usr/local/sbin/move -d $LOGLEVEL Mover Tuning should work the same as it ever has if you leave the "Move files off cache based on age?" to "No" Some may ask why I have done this mod, it's so I can keep my most recent files on the cache where they will age and move off as they get older. This allows me to keep my drives spun down most of the time. Let me know if you run into any issues.

Okay. I forked it, and was able to get the plugin installed. Can I PM you with the fork to review? Would love your feedback! This is my first attempt at making a plug-in, and I only made minor edits to your files in my fork. I also, didn't modify the system files. I made a new "mover" file with my edits in the plugin directory. So if Limetech modifies the file I would have to update my version.

First, I would like to thank your for a great script! Secondly, any thoughts about adding a section to move files based on how old they are? (I didn't notice it being brought up in the previous 5 pages of this thread) I love to code and "play" with things, so hopefully I'm not stepping on your feet here with the code below. If I am, let me know and I'll delete the post. Modify the Schedules page to add two entries. (Age yes/no, and days old 0-200) Move files off cache based on age? : <select name="age" size='1' class='narrow'> <?=mk_option($cfg['age'],'yes','Yes')?> <?=mk_option($cfg['age'],'no','No')?> </select> > Select if you want to move files off of the cache based on their age or days old. Move files that are greather than this many days old: : <select name="daysold" size="1" class='narrow'> <?for ($dt=0;$do<200;$do+=5):?> <?=mk_option($cfg['daysold'], $do, "$do")?> <?endfor;?> </select> > Select the number of days old a file has to be in order to move. Update the mover.php to call a "new" mover script modified for the mtime command. if ( $cfg['enableTurbo'] == "yes" ) { logger("Forcing turbo write on"); exec("/usr/local/sbin/mdcmd set md_write_method 1"); } if ($cfg['age'] == "yes" ) { $niceLevel = $cfg['moverNice'] ?: "0"; $ioLevel = $cfg['moverIO'] ?: "-c 2 -n 0"; logger("ionice $ioLevel nice -n $niceLevel /usr/local/sbin/age_mover start"); passthru("ionice $ioLevel nice -n $niceLevel /usr/local/sbin/age_mover start {$cfg['daysold']}"); if ( $cfg['enableTurbo'] == "yes" ) { logger("Restoring original turbo write mode"); exec("/usr/local/sbin/mdcmd set md_write_method {$vars['md_write_method']}"); } } else { $niceLevel = $cfg['moverNice'] ?: "0"; $ioLevel = $cfg['moverIO'] ?: "-c 2 -n 0"; logger("ionice $ioLevel nice -n $niceLevel /usr/local/sbin/mover.old $options"); passthru("ionice $ioLevel nice -n $niceLevel /usr/local/sbin/mover.old $options"); if ( $cfg['enableTurbo'] == "yes" ) { logger("Restoring original turbo write mode"); exec("/usr/local/sbin/mdcmd set md_write_method {$vars['md_write_method']}"); } } New "mover" script would have to be created, or at least modified the "mover.old" file to look for a second argument of "age". Add "AGE" variable near the top of the script. AGE=$2 Change the "Find" command, near line 70 to include "files only", and the mtime. # Check for objects to move from cache to array for SHAREPATH in /mnt/cache/*/ ; do SHARE=$(basename "$SHAREPATH") if grep -qs 'shareUseCache="yes"' "/boot/config/shares/${SHARE}.cfg" ; then find "${SHAREPATH%/}" -depth -type f -mtime +${AGE} | /usr/local/sbin/move -d $LOGLEVEL fi done Thoughts? Any issues I'm not seeing?

I don't think that is going to be implemented. Cache drive kind of acts as an L2ARC for writing. Nothing for read. Closest you get there is if a file is still on the cache drive. I believe any unused memory is also used for writing to the array. It has been requested that mover add a "move older than xxx date". This would allow us to keep the cache drive somewhat full with the most recent files, kind of acting as an L2ARC for reading.

Same Issue here. Following. However, did this and it seems to have fixed it. Will keep an eye on the log and see if the errors come back on. /etc/rc.d/rc.nginx restart

I'm not sure how ransomeware works. I would assume if it changes the file name, then yes, you are good. No, it should not upload the same files in a directory that have already successfully been uploaded. However, if they change, then yes, it's uploaded and overwritten.

If you use rclone sync it will overwrite what's on your cloud storage provider. But there is an easy way to get around this. Just use rclone copy that way any "file" you delete is not snyced up to the cloud. Copy would overwrite a file if it was changed though. So no protection from ransom ware. There is a backup command that is being discussed. https://forum.rclone.org/t/whats-the-latest-on-the-backup-command-development/632/5

I don't think you can do it from the GUI. I have mine in crontab. If your box is up all the time, you can make a temporary entry by doing a crontab -e crontab -e Add the following, assuming you have the same script name/location and time that I have. #Rclone Start, Nightly at midnight + 45 mintues 45 0 * * * /boot/rclone/scripts/rc_start.sh > /dev/null 2>&1 save with "ESC+: wq" (Or the same as "vi" editor)

Just a word of warning, I have only tested with 1 file in transfer. 1 file at a time seems to work better for me, that way, when rclone is killed, I only stop the transfer of 1 file instead of the default 4. This may not work so well with those that have very fast connections, or that transfer a lot of small files.

Thought this maybe helpful in here. I have my crontab entries setup to run the script at night and stop around 8am. However, I didn't like killing the script when it was almost done with an upload. i.e. Not Efficient. So I modified the kill script to see if there was under an hour left before the "last" file was to estimated to be complete. If so, it waits and checks every minute to see if its done. If it's done, or 70 minutes has passed, it then goes ahead and kills the rclone process's. I have also modified my rclone to send files up one at a time. i.e. 1 thread, and enabled bandwidth restrictions. Please review and let me know your thoughts, or if you find any errors. Crontab Entries #Rclone Start, Nightly at midnight + 45 mintues 45 0 * * * /boot/rclone/scripts/rc_start.sh > /dev/null 2>&1 #Rclone Enhanced Stop 8am 0 8 * * * /boot/rclone/scripts/rc_enhanced_stop.sh > /dev/null 2>&1 #Rclone Stop,9:15am 15 9 * * * /boot/rclone/scripts/rc_stop.sh > /dev/null 2>&1 Start scripts taken from the other rclone thread. (stignz https://lime-technology.com/forum/index.php?topic=46663.msg445897#msg445897) Start #!/bin/bash #description=RClone start. Usually start this at night when no one is using bandwidth. LOGFILE=/mnt/user/Public/rclone_logs/rclone-$(date "+%Y%m%d").log echo "-------------------------------------------------------------------------------------------------" $'\r'>> $LOGFILE 2>&1 echo rclone log $(date) $'\r'$'\r' >> $LOGFILE 2>&1 echo "Starting rclone copy" $'\r'>> $LOGFILE 2>&1 rclone sync --transfers=1 --bwlimit 1.6M '/mnt/user/Movies/Processed Movies' secret:'Data/Movies/Processed Movies/' >> $LOGFILE 2>&1 echo Competed rclone copy" $'\r'>> $LOGFILE 2>&1 echo "-------------------------------------------------------------------------------------------------" $'\r'>> $LOGFILE 2>&1 Enhanced Stop #!/bin/bash #description=Stop any active RClone process to save bandwidth when needed. Usually during the day. #This enhanced version will wait about 70 minutes for the last file to change. Once it has changed, it will kill rclone. function killrclonepids { ###echo "Killing pids here" PROCESSID=$(pgrep -d "," rclone) ###echo "ProcessID: $PROCESSID" if [ -n $PROCESSID ]; then echo "Attempting to terminate rclone" $'\r'>> $LOGFILE 2>&1 echo "processid of rclone: $PROCESSID" $'\r' >> $LOGFILE 2>&1 killall -v rclone rcloneorig echo "rclone job termination requested by cron at "$(date "+%T") $'\r'>> $LOGFILE 2>&1 echo "-------------------------------------------------------------------------------------------------" $'\r'>> $LOGFILE 2>&1 exit else echo "No PIDS found" exit fi } LOGFILE=/mnt/user/Public/rclone_logs/rclone-$(date "+%Y%m%d").log ###echo "LOGFILE: $LOGFILE" #Get last filename lastFile="$(cat $LOGFILE | grep 'done. avg:' | tail -n1 | awk -F '[:]' '{print $1}')" #Get time left timeLeft="$(cat $LOGFILE | grep 'done. avg:' | tail -n1 | awk -F '[/:]' '{print $7}' | sed -e 's/^[ \t]*//' | tr '[hm]' ':' | tr --delete s)" ###echo "LastFile: $lastFile" ###echo "timeLeft: $timeLeft" # Determine if there is more than 1 hour to go numberOfOccur=$(grep -o ":" <<<"$timeLeft" | wc -l) ###echo "numberOfOccur: $numberOfOccur" if [ $numberOfOccur -gt 1 ]; then echo "Kill all the pids, more than an hour to wait: $timeLeft" >> $LOGFILE 2>&1 killrclonepids else echo "Check every minute to see if the file is done yet!" >> $LOGFILE 2>&1 COUNTER=1 until [ $COUNTER -gt 70 ]; do sleep 1m (( COUNTER++ )) currentFile="$(cat $LOGFILE | grep 'done. avg:' | tail -n1 | awk -F '[:]' '{print $1}')" ###echo "currentFile: $currentFile" if [ "$lastFile" != "$currentFile" ]; then echo "The file has changed so kill rclone" >> $LOGFILE 2>&1 #sleep for 1 minute to make sure the file is saved sleep 1m killrclonepids fi done #we waited 70 minuts for it to finish. Kill it anyways. echo "Waited 70 minutes kill all pids" >> $LOGFILE 2>&1 killrclonepids fi #clean up any running or missed ways that rclone could still be running killrclonepids >> $LOGFILE 2>&1 Original Stop script #!/bin/bash #description=Stop any active RClone process to save bandwidth when needed. Usually during the day. LOGFILE=/mnt/user/Public/rclone_logs/rclone-$(date "+%Y%m%d").log PROCESSID=$(pgrep -d "," rclone) if [ -n $PROCESSID ]; then echo "Attempting to terminate rclone" $'\r'>> $LOGFILE 2>&1 echo "processid of rclone: $PROCESSID" $'\r' >> $LOGFILE 2>&1 killall -v rclone rcloneorig echo "rclone job termination requested by cron at "$(date "+%T") $'\r'>> $LOGFILE 2>&1 echo "-------------------------------------------------------------------------------------------------" $'\r'>> $LOGFILE 2>&1 fi

Bobbitb, for your request, couldn't you pull the amount transferred out of the log file? Maybe not ideal, but something to work in the meantime? I guess the downside is you would have to know what log file it was being written to and only have 1 instance running at a time? cat rclone_output.log | grep Transferred: | tail -2 | head -1 Transferred: 329.954 GBytes (2.197 MBytes/s)

I'm in the middle of it also. almost done with the second cycle. Mine quickly passes step 1 also. I think it's normal as its not a very large amount of data in step 1. ================================================================== 1.15 = unRAID server Pre-Clear disk /dev/sdc = cycle 2 of 3, partition start on sector 1 = = Step 1 of 10 - Copying zeros to first 2048k bytes DONE = Step 2 of 10 - Copying zeros to remainder of disk to clear it DONE = Step 3 of 10 - Disk is now cleared from MBR onward. DONE = Step 4 of 10 - Clearing MBR bytes for partition 2,3 & 4 DONE = Step 5 of 10 - Clearing MBR code area DONE = Step 6 of 10 - Setting MBR signature bytes DONE = Step 7 of 10 - Setting partition 1 to precleared state DONE = Step 8 of 10 - Notifying kernel we changed the partitioning DONE = Step 9 of 10 - Creating the /dev/disk/by* entries DONE = Step 10 of 10 - Verifying if the MBR is cleared. DONE = Post-Read in progress: 48% complete. ( 3,847,322,828,800 of 8,001,563,222,016 bytes read ) 200.1 MB/sec Disk Temperature: 28C, Elapsed Time: 88:15:36