hugenbdd

I would suggest opening a new thread under general support and post your log files. There are several other threads dealing with similar issues with mover and it looks like it could be a number of possible things.

Start with the stock mover and see if you can move 20-40GB of data off the cache first without it locking up your system. Make sure logging is enabled. I do not have a separate logging file. After you have confirmed it doesn't lock your system up with the stock mover, then install the plugin and just start off with one or two settings.

You may want to enable logging. Then check /var/log/syslog for an entry similar to this. May 27 01:05:01 Tower root: Starting Mover May 27 01:05:01 Tower root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start 30 0 0 '' '' May 27 01:05:01 Tower root: mover: started (Note: I do not have the priorities set to low, just "normal" so the values for ionice and nice will be different than yours.)

Agree with Alturismo Disable Turbo. It also looks like you are not using the latest updated plug in. If you can't "update", you may have to uninstall and then look for the new one with my username after Squid's. The author should be "Andrew Zawadzki Modified by hugenbdd"

I didn't modify any of the previous part of the script around setting the performance... Can you post a screen shot of your settings and maybe the log entry for mover and I can review my code.

Nice script! Love the Plex section of it. Question, why are you using rsync instead of the mover provided binary? The binary just needs a file list. To move from cache to disk the file list needs contain the /mnt/cache/<SHARENAME>... and for array to cache /mnt/disk<##>/<SHARENAME> Example call from the mover bash script. # Check for objects to move from array to cache for SHAREPATH in $(ls -dv /mnt/disk[0-9]*/*/) ; do SHARE=$(basename "$SHAREPATH") if grep -qs 'shareUseCache="prefer"' "/boot/config/shares/${SHARE}.cfg" ; then find "${SHAREPATH%/}" -depth | /usr/local/sbin/move -d $LOGLEVEL fi done

Yup, spelling mistake still there. I'll hold off on a release just for it though. Next release won't be until I can test spareness and add the "smaller/large" request around size.

Thanks for the sparse file link. I will be able to test/recreate now. I created the find script based on a post a page or so back, but wasn't really sure how to test it. Option 1: This should be possible and a smaller change overall. Option 2: Sounds like you want options applied per share. I'm not ready to change the code so it supports different settings for different shares. There is a LOT of work there...

At this point I think I would rather write code that does the "or"... i.e. move to free space and move aging files off without regard to any other settings. Will be easier to code at this time. But,,, I would like some time to think about this and how I would go about doing it. One thing to keep in mind... Currently mover run's based on the scheduler options. So "hourly". You could create your own cron job though to make it more often.

I have thought about this.... Would all options be used? If so, what if the options used, don't allow us to reach free space desired? If not, what options should be used if any? Do we add 1 file per share to a "total files to move" file list until we get to the desired free space? (I haven't testing sending in a mixed file list to the binary mover, but I think it will work with no problems.) Or.. Is this just - move old files over until % free? so if you choose this option, all others are ignored or grayed out in the scheduler screen? What would the inputs be for % free? 90-10%

Let us know if you find a way for last accessed. Shares are mounted with atime excluded... Which means it's not kept on the file. (look a few pages back for discussion.) As I would love to modify it for that ability, so that cache can act as a "read" cache also. Closer to tiered storage.

Bug fix release just now. (5/11/2020) Added ability for spaces in share names (quotes around share path) and fixed spelling mistake.

Missed the space issue in the notes.. Sorry. Will fix my fat finger mistake shortly...

Oh wow, big miss on my part. I don't have any shares with spaces in them.. Will work on a fix shortly.

I believe that is expected. I did not change the code around that part of the plug-in, as the button press probably still directs to the original mover bash script.

Easiest way to come close to what you want is to change mover to "hourly" and then set the "threshold of used cache" to your desired % of when to actually move files. Mover will not move anything if your cache is under that percent. It just exits with a log entry. https://imgur.com/gallery/ODc9sBV

New version just released. I would like some to test and let me know if you run into issues. I rewrote how the new "age_mover" creates it's string to call Unraids mover binary. This will allow me to add extra request easier in the future as I am now just creating a string and then executing it. Added 1.) Sparseness (.1 as an example) 2.) Ignore based on list inside of a file (provide a path) (ignore filelist) 3.) Ignore based on file types (comma separated list of file extension. .mkv,.mp4,.txt )

Almost done. Doing some basic testing on my box, but life seems to get in the way. I let you exclude based on either a comma separated list (something like .mkv,.mp4) or through a file that contains a list of files. (full cache path). Both options can be chosen at the same time, as I just "pipe" them after the find command. And remember that this only moves files off /mnt/cache/<sharename> to the array. csv list: grep -v -e '.mkv Exclude filelist: grep -vFf 'tmptest.txt' https://imgur.com/a/C2aMxw1

It only runs when scheduled. Then checks the % of cache used to determine if it should continue to move files. Example: Schedule set for 3AM with 65% cache used setting. At 3AM the mover script runs. Check the space used on the cache. If space used greater than or equal to 65% it continue to invoke the mover script. If space used less than 65% it will exit the script with an entry in the log file with something like "mover not running as it has not reached the used threshold".

Well, the mtime is interesting.. Found this post. I'll try and make some chages/test so that older than 1 day works right. https://unix.stackexchange.com/questions/92346/why-does-find-mtime-1-only-return-files-older-than-2-days

Thanks for taking a look at it. Few explanations on your comments. Sharepath is just there to mimic the variable used in the real mover script. Input variables need to have some value assigned. I use 0 as a "non value", and then later "skip" adding that section to the find command if it's set to zero. Size is "legacy" for me, and takes the size based on the schedule screen. Sparseness - I need someone to test this as I'm not sure what legitimate output is. But I think I got the script to output the find command like the example I gave. I didn't notice the file extension request before. I'll work on adding that. I exclude based on a pipe to grep and a file list..... mtime will remain in days. It's why I originally modified the script. I use it to let my cache get close to 80% full and slowly move aging files off. i.e. Think new shows for the past 45 days.

I have spent some time today working on some test code to replace what is in the "age_mover" (Custom mover bash script based off of unRAID's default). It now consists of several inputs Postion - Input 1 - start/stop/kill (from base script) 2 - Age in days 3 - Size in M 4 - Sparsenes value (1-9) (A . will be placed in the find before this value) 5 - Exclude filelist Example of my test script in action to create the "find" Commnand ./movertest start 30 45 2 '/tmp/text.txt' Age supplied Size supplied Sparness supplied Skipfilelist supplied SKIP FILE: /tmp/text.txt Find string: find /mnt/cache/TV -depth -mtime +30 -size +45M Find string: find /mnt/cache/TV -depth -mtime +30 -size +45M -printf '%S:%p\0' | awk -v RS='\0' -F : '$1 > 0.2 {sub(/^[^:]*:/, ""); print}' Adding Skip File List Skip File List Path: /tmp/text.txt aftr string: find /mnt/cache/TV -depth -mtime +30 -size +45M -printf '%S:%p\0' | awk -v RS='\0' -F : '$1 > 0.2 {sub(/^[^:]*:/, ""); print}' | grep -vFf '/tmp/text.txt' It would be nice if you guys could test this with your needs and see if the find command is correct. (Script attached) The "exclude file" should just be a list of files with their full path... in /mnt/cache of course. Also, change line 13 to your cache path. SHAREPATH="/mnt/cache/TV" IronBeardKnight Once I have the functionality of an "ignore filelist" working, I think the next step would be to work on a GUI to create that file list. (Probably several versions away.) movertest

I've thought about this also. Found this site on how to use a regex inside of a find command. However, it uses the "-type f" option. And I would rather stay away from that or else all folders will stay on your cache drive (most likely empty). https://www.cyberciti.biz/faq/find-command-exclude-ignore-files/ Could you provide a find command that "excludes" the files/folders you don't want to move? It will make it easier for me to assess how to put it into the code. I'm assuming this would also need to be able to be combined with the size/age options also? And, fyi... I still need to recode how I create the request in the script to make multiple options easier to handle.

I think this can be done. However, as I mentioned before I will need to rewrite some of the code to handle more options. Right now, I have basically hardcoded the "find" command for each of the options and also a "combined" find for age/size. I need to change it so that it creates the find based on all the options. (likely with a case statement)... In my brief research for sparse... I found this site.. Which combines the output with a gawk statement. https://www.thegeekdiary.com/how-to-find-all-the-sparse-file-in-linux/ Any chance you have an example of just a find statement not combined with gawk?

That's why I modified this plugin. So that my recent shows would stay on cache. I haven't been able to test, (as I want to keep my files on the cache drive) but it might be possible to over-ride by hitting the "move now" button on the main tab. Alternatively, you could just turn off the "age" option.. This tends to work better with a much larger cache drive. I would suggest at least 1TB or larger (I have 2TB) and am able to keep about 40 days worth of files if I'm not "back filling" things.