April 26, 201115 yr You can use rsync now with the "-c command" option: -c command = use command instead of "find" ("command" should be quoted if it has embedded spaces) Ah. I also noticed the -a flag, which allows me to exclude well enough: /boot/cache_dirs -i Media -d 10 -w -s -a "-name Backups -prune -o -name Other -prune -o -name Books -prune -o -name Audiobooks -prune -o -name Pictures -prune -o -name 'Guitar Tab' -prune -o -name Software -prune -o"
May 3, 201115 yr Hi there... Love the script and this is exactly what I'm looking for... but I'm not too sure if it's working as I expected. I've added the arg "-d 5" to scan 5 levels deep, but I can never get past the 1st level before the disks spin up. My shell skills are a little week (haven't written an sh since I was an intern), but from what I can tell, directories are cached from "build_dir_list()" function. This function has a predefined "-maxdepth 1" and does not use "$maxdepth". I've changed the 1 to 5 and I swear that fixes my problem. Any have similar experiences? Am I crazy? Suggestions? Thanks! -----20 min later update----- Ok, I see. Line 472 is where the caching starts and uses the arg -d. My question still stands... not sure why I get spin up after 2nd drill in.
May 3, 201115 yr Author Hi there... Love the script and this is exactly what I'm looking for... but I'm not too sure if it's working as I expected. I've added the arg "-d 5" to scan 5 levels deep, but I can never get past the 1st level before the disks spin up. My shell skills are a little week (haven't written an sh since I was an intern), but from what I can tell, directories are cached from "build_dir_list()" function. This function has a predefined "-maxdepth 1" and does not use "$maxdepth". I've changed the 1 to 5 and I swear that fixes my problem. Any have similar experiences? Am I crazy? Suggestions? Thanks! -----20 min later update----- Ok, I see. Line 472 is where the caching starts and uses the arg -d. My question still stands... not sure why I get spin up after 2nd drill in. Your "drill" in might be reading other files besides the directory entries. (scanning contents of files, displaying thumbnail images, etc) Those other files are not cached. The initial list of directories is just the top level directories. Use the "-F" option to put the cache_dirs in the foreground to watch as it does its scanning. It will help to understand what it is doing. Add the "-v" option to see even more detail when running in the foreground. If you find your media player is reading the .jpg or .xml files, you can add a line something like this in the loop just above line 472. find /mnt/user/Movies -type f \( -name *.jpg -o -name *.xml \) -exec cat "{}" >/dev/null \; It will read the contents of all the .jpg and .xml files and send the output to /dev/null. the side effect is that their contents is now in the disk buffer cache. Modify as needed with the name of your own "Movies" share. Joe L.
May 4, 201115 yr got cachedirs restarting on me every few hours and have no clue why it doesn't really bother me but there has to be a reason why it does this ? It begun after last reboot when i changed a 750g HDD for a 2tb HDD attached a zipped syslog this is in my go script /boot/cache_dirs -d 5 -m 3 -M 5 -p 200 -e ".custom" -w but also tried with -d 4 to see if it would make a difference syslog.zip
May 5, 201115 yr Your "drill" in might be reading other files besides the directory entries. (scanning contents of files, displaying thumbnail images, etc) Those other files are not cached. The initial list of directories is just the top level directories. Use the "-F" option to put the cache_dirs in the foreground to watch as it does its scanning. It will help to understand what it is doing. Add the "-v" option to see even more detail when running in the foreground. If you find your media player is reading the .jpg or .xml files, you can add a line something like this in the loop just above line 472. find /mnt/user/Movies -type f \( -name *.jpg -o -name *.xml \) -exec cat "{}" >/dev/null \; It will read the contents of all the .jpg and .xml files and send the output to /dev/null. the side effect is that their contents is now in the disk buffer cache. Modify as needed with the name of your own "Movies" share. Joe L. Thanks Joe! Your explanation makes sense.
May 11, 201115 yr Was this problem ever resolved? I'm getting the same errors in syslog, made the cache_pressure change as suggested, but don't have any telnet sessions open. Running cache_dirs with just the -w switch. Thanks! update it is still doing it ... mover starts running and a few minutes later i get these in my terminal Try `grep --help' for more information. head: cannot open `/ver/run/mover.pid' for reading: No such file or directory Usage: grep [OPTION]... PATTERN [FILE]... Try `grep --help' for more information. ERROR: List of process IDs must follow -p. ********* simple selection ********* ********* selection by list ********* -A all processes -C by command name -N negate selection -G by real group ID (supports names) -a all w/ tty except session leaders -U by real user ID (supports names) -d all except session leaders -g by session OR by effective group name -e all processes -p by process ID T all processes on this terminal -s processes in the sessions given a all w/ tty, including other users -t by tty g OBSOLETE -- DO NOT USE -u by effective user ID (supports names) r only running processes U processes for specified users x processes w/o controlling ttys t by tty *********** output format ********** *********** long options *********** -o,o user-defined -f full --Group --User --pid --cols --ppid -j,j job control s signal --group --user --sid --rows --info -O,O preloaded -o v virtual memory --cumulative --format --deselect -l,l long u user-oriented --sort --tty --forest --version -F extra full X registers --heading --no-heading --context ********* misc options ********* -V,V show version L list format codes f ASCII art forest -m,m,-L,-T,H threads S children in sum -y change -l format -M,Z security data c true command name -c scheduling class -w,w wide output n numeric WCHAN,UID -H process hierarchy nothing in syslog though... so you really need to have a telnet session open to have to receive these
May 12, 201115 yr I just poked at the "ERROR: List of process IDs must follow -p." issue, and found the problem. On line 460 of the cache_dirs script, there's a typo. Replace "/ver/run/mover.pid" with "/var/run/mover.pid". That should clear it up.
May 12, 201115 yr I just poked at the "ERROR: List of process IDs must follow -p." issue, and found the problem. On line 460 of the cache_dirs script, there's a typo. Replace "/ver/run/mover.pid" with "/var/run/mover.pid". That should clear it up. Will give that a shot... thanks tbone!
May 16, 201115 yr How do You restart cache_dirs from unmenu I created a file "93-unmenu_user_script_start_cache_dirs" in the unmenu dir on the flash with the below content (in linux-style carridgereturn) and get a button in the user scripts #define USER_SCRIPT_LABEL Start cache_dirs #define USER_SCRIPT_DESCR Start cache_dirs script /boot/custom/cache_dirs/cache_dirs -w -d 5 But when I hit that user script button, unmenu hangs on me untill I invoke cache_dirs -q in a terminal session and also manually kill one last remauning instance of cache_dirs that shows up in a "ps -e | grep dir" listing. Why does unmenu hang like that? I've tried nohup'ing the command but no luck for me there - please help!
May 16, 201115 yr hi this is mine userscript and it works Start #710-unmenu_user_script_Start_CacheDirs #define USER_SCRIPT_LABEL Start CacheDirs #define USER_SCRIPT_DESCR start Cachedirs echo "/boot/cache_dirs -d 5 -m 3 -M 5 -p 200 -e ".custom" -e "mysql" -e "Thumbnails" -w" | at now Stop #711-unmenu_user_script_Stop_CacheDirs #define USER_SCRIPT_LABEL Stop CacheDirs #define USER_SCRIPT_DESCR stop Cachedirs /boot/cache_dirs -q
May 16, 201115 yr Why does unmenu hang like that? I've tried nohup'ing the command but no luck for me there - please help! It hangs because unmenu is waiting for cache_dirs to return some output... which it will not because of the way it runs in the background. The post below your, or above mine as the case were, will work to start cache_dirs.
May 17, 201115 yr Thanks guys! While I'm at it, is there a way to use cache_dirs to cache folder.jpg's to prevent my mediaportal htpc spinning up drives when browsing folders ? Or am I looking at adding a "cp folder.jpg null" or something to that effect somewhere in the script ?
May 17, 201115 yr Author Thanks guys! While I'm at it, is there a way to use cache_dirs to cache folder.jpg's to prevent my mediaportal htpc spinning up drives when browsing folders ? Or am I looking at adding a "cp folder.jpg null" or something to that effect somewhere in the script ? Around line 476, add a line like the one in blue below (note, change the name of the user-share to match your share name. Mine is "Movies") Since my media player stores lots of information in .xml files, they too are cached by copying them to /dev/null): # always cache root dirs on each of the disks for i in /mnt/disk* /mnt/cache $user_share_dir do find $i -maxdepth 1 -noleaf >/dev/null 2>/dev/null done find /mnt/user/Movies -type f \( -name *.jpg -o -name *.xml \) -exec cat "{}" >/dev/null \; echo "$dir_list" | while read share_dir do for i in /mnt/disk* /mnt/cache $user_share_dir do # If the directory does not exist on this disk, don't do recursive "directory scan" [ ! -d "$i"/"$share_dir" ] && continue # Perform a recursive "find" on /mnt/disk??/share $command "$i"/"$share_dir" $args $depth >/dev/null 2>&1 [ $background = "no" -a $verbose = "yes" ] && echo "Executing $command $i/$share_dir $args $depth" done done
May 17, 201115 yr You make it look so easy, Joe So, do You have a specially customized Joe-version of the script for Your own use ? I have had cache-dirs on my to-do list for so long and only got around to installing it this week. Should have done it earlier, it's brilliant! Love the stop array detection, You are truly an artist at this. About the stop-detection, I have been thinking about adding some lines to kill other apps.. Not sure I want to keep waiting for 5.0 final.
May 17, 201115 yr You make it look so easy, Joe So, do You have a specially customized Joe-version of the script for Your own use ? I have had cache-dirs on my to-do list for so long and only got around to installing it this week. Should have done it earlier, it's brilliant! Love the stop array detection, You are truly an artist at this. About the stop-detection, I have been thinking about adding some lines to kill other apps.. Not sure I want to keep waiting for 5.0 final. do a search for unRAID_addon_control.sh There is some setup and reading that you will have to do but it does work.
May 17, 201115 yr Joe, I just downloaded version 1.6.5. A documentation comment and request. Reading the code, there is mention of the 'p' option on line 41 (added in version 1.6) but there is no mention of the 'p' switch/option in the script help (function usage) lines 74-98. This means that if I do a: cache_dirs -h I will not see any reference to the 'p' switch. Considering your comments on lines 129-136, I think it is important that people know about changing cache_pressure and the 'p' switch should be documented. Please consider updating the usage function to include 'p' switch. Thanks, Rick
May 19, 201115 yr Why does unmenu hang like that? I've tried nohup'ing the command but no luck for me there - please help! It hangs because unmenu is waiting for cache_dirs to return some output... which it will not because of the way it runs in the background. The post below your, or above mine as the case were, will work to start cache_dirs. I don't think that's accurate. When cache_dirs starts, it does output a string to the console. Looks like this on mine: cache_dirs process ID 5317 started, To terminate it, type: cache_dirs -q Further, I've now tried to invoke a script of mine that starts cache_dirs with my favorite arguments (that way it will always start with same arguments goth when called from go script and from unmenu). I've added an echo command - script looks like this: cache_dirs_args='-w -s -d 5 -e "Backup" -e "Games" -e "MP3BACKUP"' /boot/custom/cache_dirs/cache_dirs $cache_dirs_args echo "cache_dirs started in background with argsuments" $cache_dirs_args Still the same problem - unmenu hangs when I invoke my script (that I'm positively sure does output text). I can use the AT workarround (Thnx sacretagent), just curious as to why this happens with the way I had done it. Trying to learn here
May 19, 201115 yr Author Why does unmenu hang like that? I've tried nohup'ing the command but no luck for me there - please help! It hangs because unmenu is waiting for cache_dirs to return some output... which it will not because of the way it runs in the background. The post below your, or above mine as the case were, will work to start cache_dirs. I don't think that's accurate. When cache_dirs starts, it does output a string to the console. Looks like this on mine: cache_dirs process ID 5317 started, To terminate it, type: cache_dirs -q Further, I've now tried to invoke a script of mine that starts cache_dirs with my favorite arguments (that way it will always start with same arguments goth when called from go script and from unmenu). I've added an echo command - script looks like this: cache_dirs_args='-w -s -d 5 -e "Backup" -e "Games" -e "MP3BACKUP"' /boot/custom/cache_dirs/cache_dirs $cache_dirs_args echo "cache_dirs started in background with argsuments" $cache_dirs_args Still the same problem - unmenu hangs when I invoke my script (that I'm positively sure does output text). I can use the AT workarround (Thnx sacretagent), just curious as to why this happens with the way I had done it. Trying to learn here unMENU waits for all output. If you want it to return you must re-direct both standard output and error output from file descriptor 2. Try something like might work (I've not tried it): cache_dirs -w 2>&1 >/dev/null
May 23, 201115 yr find /mnt/user/Movies -type f \( -name *.jpg -o -name *.xml \) -exec cat "{}" >/dev/null \; Hi Joe. Adding this line as suggested seems to constantly keep my disks spinning as it searches for the files. As far as I can tell, almost all disks (since the movies are spread across almost all disks) have not been spun down for three days since the reboot after changes have been made. Is this unusual or should it take a while? Also, my movies are note under one user share but split into several alphabetical bins (e.g. E-J, etc.). Is this the correct syntax to have to search all the Movie shares - find /mnt/user/Movies*? Thanks Joe!
May 23, 201115 yr Author find /mnt/user/Movies -type f \( -name *.jpg -o -name *.xml \) -exec cat "{}" >/dev/null \; Hi Joe. Adding this line as suggested seems to constantly keep my disks spinning as it searches for the files. As far as I can tell, almost all disks (since the movies are spread across almost all disks) have not been spun down for three days since the reboot after changes have been made. Is this unusual or should it take a while? Also, my movies are note under one user share but split into several alphabetical bins (e.g. E-J, etc.). Is this the correct syntax to have to search all the Movie shares - find /mnt/user/Movies*? Thanks Joe! If your movies are not in a user-share named "Movies" then the sample command is the wrong syntax AND you will need to modify it. Easiest is to probably say something like: find /mnt/user/ -type f \( -name *.jpg -o -name *.xml \) -exec cat "{}" >/dev/null \; If your share names have spaces, or square braces, as you indicated, the user-share names might need to be quoted to eliminate the shell from expanding the names. Instead of adding the above line, type it on the command line, and see what it does. Instead of "-exec cat "{} >/dev/null \;" you can put find /mnt/user/ -type f \( -name *.jpg -o -name *.xml \) -print while you experiment with how to format the find commands.
May 23, 201115 yr If your movies are not in a user-share named "Movies" then the sample command is the wrong syntax AND you will need to modify it. Easiest is to probably say something like: find /mnt/user/ -type f \( -name *.jpg -o -name *.xml \) -exec cat "{}" >/dev/null \; If your share names have spaces, or square braces, as you indicated, the user-share names might need to be quoted to eliminate the shell from expanding the names. Instead of adding the above line, type it on the command line, and see what it does. Instead of "-exec cat "{} >/dev/null \;" you can put find /mnt/user/ -type f \( -name *.jpg -o -name *.xml \) -print while you experiment with how to format the find commands. Thanks Joe for the tips. I modified the shares to contain no brackets, but they do contain spaces. My current syntax, tested at the command line seemed to locate the files correctly: find "/mnt/user/Movies G-J" -type f \( -name *.jpg -o -name *.xml \) -exec cat "{}" >/dev/null \; However, after running the script, it seems to be exhibiting the same behavior of spinning up the disks as it traverses the shares searching for the files; eventually, almost all disks are spinning. I have attached an excerpt fro the syslog. Thanks! cache_excerpt.txt
May 23, 201115 yr Author If your movies are not in a user-share named "Movies" then the sample command is the wrong syntax AND you will need to modify it. Easiest is to probably say something like: find /mnt/user/ -type f \( -name *.jpg -o -name *.xml \) -exec cat "{}" >/dev/null \; If your share names have spaces, or square braces, as you indicated, the user-share names might need to be quoted to eliminate the shell from expanding the names. Instead of adding the above line, type it on the command line, and see what it does. Instead of "-exec cat "{} >/dev/null \;" you can put find /mnt/user/ -type f \( -name *.jpg -o -name *.xml \) -print while you experiment with how to format the find commands. It probably indicates you have more total space in all the combined .jpg files on your server than can fit in the available buffer cache.
May 23, 201115 yr It probably indicates you have more total space in all the combined .jpg files on your server than can fit in the available buffer cache. As I recall, your collection is vastly larger than mine (with corresponding .jpg and .xml). How does your server manage it? Do you run across similar entries in your syslog?
May 23, 201115 yr Author It probably indicates you have more total space in all the combined .jpg files on your server than can fit in the available buffer cache. As I recall, your collection is vastly larger than mine (with corresponding .jpg and .xml). How does your server manage it? Do you run across similar entries in your syslog? I don't attempt to get them to not spin up.
July 13, 201114 yr I can't believe this hasn't been caught by anyone else, but there is an error on line 460 in the script mover_pid=`head -n 1 /ver/run/mover.pid` it should be mover_pid=`head -n 1 /var/run/mover.pid` As far as I know, no implementation of unraid has a /ver directory.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.