cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up

[coppit]

You can use rsync now with the "-c command" option:

-c command   = use command instead of "find"

   ("command" should be quoted if it has embedded spaces)

 

Ah. I also noticed the -a flag, which allows me to exclude well enough:

 

/boot/cache_dirs -i Media -d 10 -w -s -a "-name Backups -prune -o -name Other -prune -o -name Books -prune -o -name Audiobooks -prune -o -name Pictures -prune -o -name 'Guitar Tab' -prune -o -name Software -prune -o"

[petsheep]

Hi there...

 

Love the script and this is exactly what I'm looking for... but I'm not too sure if it's working as I expected. I've added the arg "-d 5" to scan 5 levels deep, but I can never get past the 1st level before the disks spin up. My shell skills are a little week (haven't written an sh since I was an intern), but from what I can tell, directories are cached from "build_dir_list()" function. This function has a predefined "-maxdepth 1" and does not use "$maxdepth". I've changed the 1 to 5 and I swear that fixes my problem. Any have similar experiences? Am I crazy? Suggestions?

 

Thanks!

 

-----20 min later update-----

Ok, I see. Line 472 is where the caching starts and uses the arg -d. My question still stands... not sure why I get spin up after 2nd drill in.

[Joe L.]

Hi there...

 

Love the script and this is exactly what I'm looking for... but I'm not too sure if it's working as I expected. I've added the arg "-d 5" to scan 5 levels deep, but I can never get past the 1st level before the disks spin up. My shell skills are a little week (haven't written an sh since I was an intern), but from what I can tell, directories are cached from "build_dir_list()" function. This function has a predefined "-maxdepth 1" and does not use "$maxdepth". I've changed the 1 to 5 and I swear that fixes my problem. Any have similar experiences? Am I crazy? Suggestions?

 

Thanks!

 

-----20 min later update-----

Ok, I see. Line 472 is where the caching starts and uses the arg -d. My question still stands... not sure why I get spin up after 2nd drill in.

Your "drill" in might be reading other files besides the directory entries.  (scanning contents of files, displaying thumbnail images, etc) Those other files are not cached.

 

The initial list of directories is just the top level directories.  Use the "-F" option to put the cache_dirs in the foreground to watch as it does its scanning.  It will help to understand what it is doing.  Add the "-v" option to see even more detail when running  in the foreground.

 

If you find your media player is reading the .jpg or .xml files, you can add a line something like this in the loop just above line 472.

find /mnt/user/Movies -type f \( -name *.jpg -o -name *.xml \) -exec cat "{}" >/dev/null \;

 

It will read the contents of all the .jpg and .xml files and send the output to /dev/null.  the side effect is that their contents is now in the disk buffer cache.  Modify as needed with the name of your own "Movies" share.

 

 

Joe L.

[sacretagent]

got cachedirs restarting on me every few hours and have no clue why

it doesn't really bother me but there has to be a reason why it does this ?

It begun after last reboot when i changed a 750g HDD for a 2tb HDD

attached a zipped syslog

 

this is in my go script

/boot/cache_dirs -d 5 -m 3 -M 5 -p 200 -e ".custom" -w

 

but also tried with -d 4 to see if it would make a difference

syslog.zip

[petsheep]

Your "drill" in might be reading other files besides the directory entries.  (scanning contents of files, displaying thumbnail images, etc) Those other files are not cached.

 

The initial list of directories is just the top level directories.  Use the "-F" option to put the cache_dirs in the foreground to watch as it does its scanning.  It will help to understand what it is doing.  Add the "-v" option to see even more detail when running  in the foreground.

 

If you find your media player is reading the .jpg or .xml files, you can add a line something like this in the loop just above line 472.

find /mnt/user/Movies -type f \( -name *.jpg -o -name *.xml \) -exec cat "{}" >/dev/null \;

 

It will read the contents of all the .jpg and .xml files and send the output to /dev/null.  the side effect is that their contents is now in the disk buffer cache.  Modify as needed with the name of your own "Movies" share.

 

Joe L.

 

Thanks Joe! Your explanation makes sense.

[arcane]

Was this problem ever resolved?  I'm getting the same errors in syslog, made the cache_pressure change as suggested, but don't have any telnet sessions open.  Running cache_dirs with just the -w switch.  Thanks!

 

update

 

it is still doing it ... mover starts running and a few minutes later i get these in my terminal

 

Try `grep --help' for more information.
head: cannot open `/ver/run/mover.pid' for reading: No such file or directory
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
ERROR: List of process IDs must follow -p.
********* simple selection *********  ********* selection by list *********
-A all processes                      -C by command name
-N negate selection                   -G by real group ID (supports names)
-a all w/ tty except session leaders  -U by real user ID (supports names)
-d all except session leaders         -g by session OR by effective group name
-e all processes                      -p by process ID
T  all processes on this terminal     -s processes in the sessions given
a  all w/ tty, including other users  -t by tty
g  OBSOLETE -- DO NOT USE             -u by effective user ID (supports names)
r  only running processes             U  processes for specified users
x  processes w/o controlling ttys     t  by tty
*********** output format **********  *********** long options ***********
-o,o user-defined  -f full            --Group --User --pid --cols --ppid
-j,j job control   s  signal          --group --user --sid --rows --info
-O,O preloaded -o  v  virtual memory  --cumulative --format --deselect
-l,l long          u  user-oriented   --sort --tty --forest --version
-F   extra full    X  registers       --heading --no-heading --context
                    ********* misc options *********
-V,V  show version      L  list format codes  f  ASCII art forest
-m,m,-L,-T,H  threads   S  children in sum    -y change -l format
-M,Z  security data     c  true command name  -c scheduling class
-w,w  wide output       n  numeric WCHAN,UID  -H process hierarchy

 

nothing in syslog though... so you really need to have a telnet session open to have to receive these

 

[tbone]

I just poked at the "ERROR: List of process IDs must follow -p." issue, and found the problem.  On line 460 of the cache_dirs script, there's a typo.  Replace "/ver/run/mover.pid" with "/var/run/mover.pid".  That should clear it up.

[arcane]

I just poked at the "ERROR: List of process IDs must follow -p." issue, and found the problem.  On line 460 of the cache_dirs script, there's a typo.  Replace "/ver/run/mover.pid" with "/var/run/mover.pid".  That should clear it up.

 

Will give that a shot... thanks tbone!

[MortenSchmidt]

How do You restart cache_dirs from unmenu

 

I created a file "93-unmenu_user_script_start_cache_dirs" in the unmenu dir on the flash with the below content (in linux-style carridgereturn) and get a button in the user scripts

 

#define USER_SCRIPT_LABEL Start cache_dirs
#define USER_SCRIPT_DESCR Start cache_dirs script 
/boot/custom/cache_dirs/cache_dirs -w -d 5

 

But when I hit that user script button, unmenu hangs on me untill I invoke cache_dirs -q in a terminal session and also manually kill one last remauning instance of cache_dirs that shows up in a "ps -e | grep dir" listing. Why does unmenu hang like that? I've tried nohup'ing the command but no luck for me there - please help!

[sacretagent]

hi this is mine userscript and it works

 

Start

 

#710-unmenu_user_script_Start_CacheDirs
#define USER_SCRIPT_LABEL Start CacheDirs
#define USER_SCRIPT_DESCR start Cachedirs
echo "/boot/cache_dirs -d 5 -m 3 -M 5 -p 200 -e ".custom" -e "mysql" -e "Thumbnails" -w" | at now

 

Stop

#711-unmenu_user_script_Stop_CacheDirs
#define USER_SCRIPT_LABEL Stop CacheDirs
#define USER_SCRIPT_DESCR stop Cachedirs
/boot/cache_dirs -q

[prostuff1]

Why does unmenu hang like that? I've tried nohup'ing the command but no luck for me there - please help!

It hangs because unmenu is waiting for cache_dirs to return some output... which it will not because of the way it runs in the background.

 

The post below your, or above mine as the case were, will work to start cache_dirs.

[MortenSchmidt]

Thanks guys!

 

While I'm at it, is there a way to use cache_dirs to cache folder.jpg's to prevent my mediaportal htpc spinning up drives when browsing folders ?

 

Or am I looking at adding a "cp folder.jpg null" or something to that effect somewhere in the script ?

[Joe L.]

Thanks guys!

 

While I'm at it, is there a way to use cache_dirs to cache folder.jpg's to prevent my mediaportal htpc spinning up drives when browsing folders ?

 

Or am I looking at adding a "cp folder.jpg null" or something to that effect somewhere in the script ?

Around line 476, add a line like the one in blue below  (note, change the name of the user-share to match your share name.  Mine is "Movies")

Since my media player stores lots of information in .xml files, they too are cached by copying them to /dev/null):

 

# always cache root dirs on each of the disks

  for i in /mnt/disk* /mnt/cache $user_share_dir

  do

    find $i -maxdepth 1 -noleaf >/dev/null 2>/dev/null

  done

find /mnt/user/Movies -type f \( -name *.jpg -o -name *.xml \) -exec cat "{}" >/dev/null \;

 

  echo "$dir_list" | while read share_dir

  do

    for i in /mnt/disk* /mnt/cache $user_share_dir

    do

      # If the directory does not exist on this disk, don't do recursive "directory scan"

      [ ! -d "$i"/"$share_dir" ] && continue

 

      # Perform a recursive "find" on /mnt/disk??/share

      $command "$i"/"$share_dir" $args $depth >/dev/null 2>&1

      [ $background = "no" -a $verbose = "yes" ] && echo "Executing $command $i/$share_dir $args $depth"

    done

  done

[MortenSchmidt]

You make it look so easy, Joe 

 

So, do You have a specially customized Joe-version of the script for Your own use ?

 

I have had cache-dirs on my to-do list for so long and only got around to installing it this week. Should have done it earlier, it's brilliant! Love the stop array detection, You are truly an artist at this.

 

About the stop-detection, I have been thinking about adding some lines to kill other apps.. Not sure I want to keep waiting for 5.0 final.

 

[prostuff1]

You make it look so easy, Joe 

 

So, do You have a specially customized Joe-version of the script for Your own use ?

 

I have had cache-dirs on my to-do list for so long and only got around to installing it this week. Should have done it earlier, it's brilliant! Love the stop array detection, You are truly an artist at this.

 

About the stop-detection, I have been thinking about adding some lines to kill other apps.. Not sure I want to keep waiting for 5.0 final.

 

 

do a search for unRAID_addon_control.sh

 

There is some setup and reading that you will have to do but it does work.

[rsmereka]

Joe,

 

I just downloaded version 1.6.5.

 

A documentation comment and request. Reading the code, there is mention of the 'p' option on line 41 (added in version 1.6) but there is no mention of the 'p' switch/option in the script help (function usage) lines 74-98. This means that if I do a:

cache_dirs -h

I will not see any reference to the 'p' switch.

 

Considering your comments on lines 129-136, I think it is important that people know about changing cache_pressure and the 'p' switch should be documented.

 

Please consider updating the usage function to include 'p' switch.

 

Thanks,

Rick

[MortenSchmidt]

Why does unmenu hang like that? I've tried nohup'ing the command but no luck for me there - please help!

It hangs because unmenu is waiting for cache_dirs to return some output... which it will not because of the way it runs in the background.

 

The post below your, or above mine as the case were, will work to start cache_dirs.

 

I don't think that's accurate. When cache_dirs starts, it does output a string to the console. Looks like this on mine:

cache_dirs process ID 5317 started, To terminate it, type: cache_dirs -q

 

Further, I've now tried to invoke a script of mine that starts cache_dirs with my favorite arguments (that way it will always start with same arguments goth when called from go script and from unmenu). I've added an echo command - script looks like this:

cache_dirs_args='-w -s -d 5 -e "Backup" -e "Games" -e "MP3BACKUP"'
/boot/custom/cache_dirs/cache_dirs $cache_dirs_args
echo "cache_dirs started in background with argsuments" $cache_dirs_args

 

Still the same problem - unmenu hangs when I invoke my script (that I'm positively sure does output text).

 

I can use the AT workarround (Thnx sacretagent), just curious as to why this happens with the way I had done it. Trying to learn here 

[Joe L.]

Why does unmenu hang like that? I've tried nohup'ing the command but no luck for me there - please help!

It hangs because unmenu is waiting for cache_dirs to return some output... which it will not because of the way it runs in the background.

 

The post below your, or above mine as the case were, will work to start cache_dirs.

 

I don't think that's accurate. When cache_dirs starts, it does output a string to the console. Looks like this on mine:

cache_dirs process ID 5317 started, To terminate it, type: cache_dirs -q

 

Further, I've now tried to invoke a script of mine that starts cache_dirs with my favorite arguments (that way it will always start with same arguments goth when called from go script and from unmenu). I've added an echo command - script looks like this:

cache_dirs_args='-w -s -d 5 -e "Backup" -e "Games" -e "MP3BACKUP"'
/boot/custom/cache_dirs/cache_dirs $cache_dirs_args
echo "cache_dirs started in background with argsuments" $cache_dirs_args

 

Still the same problem - unmenu hangs when I invoke my script (that I'm positively sure does output text).

 

I can use the AT workarround (Thnx sacretagent), just curious as to why this happens with the way I had done it. Trying to learn here 

unMENU waits for all output.  If you want it to return you must re-direct both standard output and error output from file descriptor 2.

 

Try something like might work (I've not tried it):

 

cache_dirs -w 2>&1 >/dev/null

[arcane]

 

find /mnt/user/Movies -type f \( -name *.jpg -o -name *.xml \) -exec cat "{}" >/dev/null \;

 

 

Hi Joe.  Adding this line as suggested seems to constantly keep my disks spinning as it searches for the files.  As far as I can tell, almost all disks (since the movies are spread across almost all disks) have not been spun down for three days since the reboot after changes have been made.  Is this unusual or should it take a while?

 

Also, my movies are note under one user share but split into several alphabetical bins (e.g. E-J, etc.).  Is this the correct syntax to have to search all the Movie shares - find /mnt/user/Movies*?

 

Thanks Joe!

[Joe L.]

 

find /mnt/user/Movies -type f \( -name *.jpg -o -name *.xml \) -exec cat "{}" >/dev/null \;

 

 

Hi Joe.  Adding this line as suggested seems to constantly keep my disks spinning as it searches for the files.  As far as I can tell, almost all disks (since the movies are spread across almost all disks) have not been spun down for three days since the reboot after changes have been made.  Is this unusual or should it take a while?

 

Also, my movies are note under one user share but split into several alphabetical bins (e.g. E-J, etc.).  Is this the correct syntax to have to search all the Movie shares - find /mnt/user/Movies*?

 

Thanks Joe!

If your movies are not in a user-share named "Movies" then the sample command is the wrong syntax AND you will need to modify it.

Easiest is to probably say something like:

find /mnt/user/ -type f \( -name *.jpg -o -name *.xml \) -exec cat "{}" >/dev/null \;

 

If your share names have spaces, or square braces, as you indicated, the user-share names might need to be quoted to eliminate the shell from expanding the names.

 

Instead of adding the above line, type it on the command line, and see what it does.  Instead of "-exec cat "{} >/dev/null \;" you can put

find /mnt/user/ -type f \( -name *.jpg -o -name *.xml \) -print

while you experiment with how to format the find commands.

[arcane]

If your movies are not in a user-share named "Movies" then the sample command is the wrong syntax AND you will need to modify it.

Easiest is to probably say something like:

find /mnt/user/ -type f \( -name *.jpg -o -name *.xml \) -exec cat "{}" >/dev/null \;

 

If your share names have spaces, or square braces, as you indicated, the user-share names might need to be quoted to eliminate the shell from expanding the names.

 

Instead of adding the above line, type it on the command line, and see what it does.  Instead of "-exec cat "{} >/dev/null \;" you can put

find /mnt/user/ -type f \( -name *.jpg -o -name *.xml \) -print

while you experiment with how to format the find commands.

 

Thanks Joe for the tips.

 

I modified the shares to contain no brackets, but they do contain spaces.  My current syntax, tested at the command line seemed to locate the files correctly:

 

find "/mnt/user/Movies G-J" -type f \( -name *.jpg -o -name *.xml \) -exec cat "{}" >/dev/null \;

 

However, after running the script, it seems to be exhibiting the same behavior of spinning up the disks as it traverses the shares searching for the files; eventually, almost all disks are spinning.

 

I have attached an excerpt fro the syslog.  Thanks!

 

cache_excerpt.txt

[Joe L.]

If your movies are not in a user-share named "Movies" then the sample command is the wrong syntax AND you will need to modify it.

Easiest is to probably say something like:

find /mnt/user/ -type f \( -name *.jpg -o -name *.xml \) -exec cat "{}" >/dev/null \;

 

If your share names have spaces, or square braces, as you indicated, the user-share names might need to be quoted to eliminate the shell from expanding the names.

 

Instead of adding the above line, type it on the command line, and see what it does.  Instead of "-exec cat "{} >/dev/null \;" you can put

find /mnt/user/ -type f \( -name *.jpg -o -name *.xml \) -print

while you experiment with how to format the find commands.

It probably indicates you have more total space in all the combined .jpg files on your server than can fit in the available buffer cache.

 

[arcane]

It probably indicates you have more total space in all the combined .jpg files on your server than can fit in the available buffer cache.

 

As I recall, your collection is vastly larger than mine (with corresponding .jpg and .xml).  How does your server manage it?  Do you run across similar entries in your syslog?

[Joe L.]

It probably indicates you have more total space in all the combined .jpg files on your server than can fit in the available buffer cache.

 

As I recall, your collection is vastly larger than mine (with corresponding .jpg and .xml).  How does your server manage it?  Do you run across similar entries in your syslog?

I don't attempt to get them to not spin up.

[SilverRubicon]

I can't believe this hasn't been caught by anyone else, but there is an error on line 460 in the script

 

     mover_pid=`head -n 1 /ver/run/mover.pid`

 

it should be

 

     mover_pid=`head -n 1 /var/run/mover.pid`

 

As far as I know, no implementation of unraid has a /ver directory.