Jump to content
Joe L.

cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up

1086 posts in this topic Last Reply

Recommended Posts

Hello Joe.  I just reviewed my syslog after fixing the other "Missing Disk" problem that you helped with me.

 

I noticed that the Cache_Dir program does not seem to like folder names with spaces.  Is there a way other than eliminating the space to get around this?

 

Excerpt from syslog:

 

Dec 3 13:33:43 Tower cache_dirs: ==============================================

Dec 3 13:33:43 Tower cache_dirs: command-args=-w -i Movies -i TV Shows -i Backdrops

Dec 3 13:33:43 Tower cache_dirs: vfs_cache_pressure=1

Dec 3 13:33:43 Tower cache_dirs: max_seconds=10, min_seconds=1

Dec 3 13:33:43 Tower cache_dirs: max_depth=9999

Dec 3 13:33:43 Tower cache_dirs: command=find -noleaf

Dec 3 13:33:43 Tower cache_dirs: version=1.6.5

Dec 3 13:33:43 Tower cache_dirs: ---------- caching directories ---------------

Dec 3 13:33:43 Tower cache_dirs: Backdrops

Dec 3 13:33:43 Tower cache_dirs: TV Shows

Dec 3 13:33:43 Tower cache_dirs: Movies

Dec 3 13:33:43 Tower cache_dirs: ----------------------------------------------

Dec 3 13:33:43 Tower cache_dirs: ERROR: included directory "TV Shows" does not exist.

Dec 3 13:33:43 Tower cache_dirs: cache_dirs process ID 1915 started, To terminate it, type: cache_dirs -q

 

Try it with -i "TV Shows" or if that does not work try -i TV\ Shows

Share this post


Link to post

Thanks.  Placing the \ before the spaced worked perfectly.   :)

yes the backslash escaped space will work, but so will putting the name in quotes... In fact, you probably did not notice, but the VERY FIRST POST in this thread says exactly that:

 

If you have a folder or folders you wish to exclude, there is a -e option.  This option can be used multiple times.  To exclude a "data" and "old-stuff" directory, you would use

cache_dirs -w -e "data" -e "old-stuff"

Always use Quote marks around the folders you wish to exclude or include, this is especially important if the folder name contains a space or other special character that might be interpreted by the linux shell.

Share this post


Link to post

Possibly ignorant question here and I apologize as i am reading on my iPad but when is the cache updated? If I add a newmmovie would the cache reflect that immediately? Thanks!

Share this post


Link to post

Possibly ignorant question here and I apologize as i am reading on my iPad but when is the cache updated? If I add a newmmovie would the cache reflect that immediately? Thanks!

which of the "cache" are you referring to.

 

The "cache" disk that allow quick writes to an un-protected disk that is later moved to the protected array?  The cache disk is "merged" into the view of the share contents, so the file is instantly available for reading or writing, it is just not protected by parity until the file is moved into the protected array.    So.. you can add a movie, and have it on a "cache" disk, and still play it as if it was in the protected array.

 

or

 

The disk buffer cache, which holds in memory anything sent to , or read from the disks?

 

The disk buffer cache is will use as much memory as you have in the server.  It will re-use the least recently used block of memory when asked to read or write a disk block not already in its cache of recently used disks contents.  It is instantly updated as disks are accessed.  The least-recently used blocks are re-used if a new disk block is accessed.

 

Joe L.

Share this post


Link to post

I meant the cache for directory contents for this script?

It is not a separate cache maintained by this script.  It is the normal disk buffer cache keeping the most recently accessed blocks of data in memory.

 

The cache_dirs script simply does the equivalent of you requesting a directory listing (but very frequently), forcing the access of the data blocks representing the directory entries.  Since it does this frequently, they never end up as the least recently accessed blocks and do not get reused once they've been read the first time from the physical disks.  This minimizes the need to spin up a physical disk, so access is fast. 

Share this post


Link to post

Joel L. I just recently purchased another 2GB stick for my machine for a total of 4GB. Would that allow for more Cachin'g of files or is there a set limit when it begins to over right newer reads over older reads? I'm guessing that the cache of files doesn't take up much memory, but was a bit intriqued enough to ask. ;)

 

I personally picked up the extra memory because it was rather cheap and figured if I wanted to do more I would have the extra Ram.

Share this post


Link to post

Joel L. I just recently purchased another 2GB stick for my machine for a total of 4GB. Would that allow for more Cachin'g of files or is there a set limit when it begins to over right newer reads over older reads? I'm guessing that the cache of files doesn't take up much memory, but was a bit intriqued enough to ask. ;)

 

I personally picked up the extra memory because it was rather cheap and figured if I wanted to do more I would have the extra Ram.

Linux will use as much memory as it can for the buffering of disks.  Even when using 4Gig or RAM the first time you play a 5Gig movie all of it could get used.  Remember, the least-recently-accessed gets re-used.  So you can rewind a movie a few minutes really quick.

 

Joe L.

Share this post


Link to post

Thanks. Using XBMC I've had really good results with rewinding and fast forwarding, but sounds good to me. I'm glad to hear that the extra ram will not be a hindrance or a blessing with regards to caching. LOL

Share this post


Link to post

Hi Joe,

 

Cache-dirs is about the only thing I run on my server apart of the APC UPS Powerdown script.

 

I have only recently added it after upgrading to 4.5 series from 4.3.3.  Is there any time I should turn it off - like when I'm copying a lot to the server or running a parity check? I have 2Gb RAM and it seems to handle everything so far - but a couple of crashes (can't connect to the server) have started to plague me since I upgraded to the 4.5 series and I'm wondering if it running in the background while I do other tasks is having an affect? You did comment in another post how it can use all the RAM, but is there a situation it is better to turn it off?

 

Also...If I stop it via the "-q" do I start it again with "-W" Does this make it start again with all the features as in the "Go Script"?

 

And...as part of my trouble shooting, I'm considering downgrading, so I am assuming cache_dirs will work with 4.3.3, but that the double stop will take effect? Thx.

Share this post


Link to post

I meant the cache for directory contents for this script?

It is not a separate cache maintained by this script.  It is the normal disk buffer cache keeping the most recently accessed blocks of data in memory.

 

The cache_dirs script simply does the equivalent of you requesting a directory listing (but very frequently), forcing the access of the data blocks representing the directory entries.  Since it does this frequently, they never end up as the least recently accessed blocks and do not get reused once they've been read the first time from the physical disks.  This minimizes the need to spin up a physical disk, so access is fast.   

 

I am confused... I thought the purpose of this was the cache the directory listing so that when I request my movie list the drives wouldn't take time or have to spin up?  Is this not the case?  If it is then when I add a new movie when is the directory listing which I believe is cached at this point updated?

Share this post


Link to post

I meant the cache for directory contents for this script?

It is not a separate cache maintained by this script.  It is the normal disk buffer cache keeping the most recently accessed blocks of data in memory.

 

The cache_dirs script simply does the equivalent of you requesting a directory listing (but very frequently), forcing the access of the data blocks representing the directory entries.  Since it does this frequently, they never end up as the least recently accessed blocks and do not get reused once they've been read the first time from the physical disks.  This minimizes the need to spin up a physical disk, so access is fast.   

 

I am confused... I thought the purpose of this was the cache the directory listing so that when I request my movie list the drives wouldn't take time or have to spin up?  Is this not the case?  If it is then when I add a new movie when is the directory listing which I believe is cached at this point updated?

cache_dirs constantly accesses the directory listings, in an attempt to keep them in the disk buffer cache shared with ALL disk operations.

The hope is that normal playing of movies will not access disk blocks faster than the periodic scan of directories by cache_dirs causing them to be the least-recently accessed.

 

When you add a new movie it will be in the directory cache immediately, since it is the most recently accessed disk blocks.

 

The end effect is that drives should be able to so to sleep and still present a directory listing, since their directory blocks are in the in-memory disk buffer cache.

 

Joe L.

Share this post


Link to post

Joe,

 

I've been thinking of adding this to my server for some time, but just want to be sure I do it correctly.    Can you confirm that the following is the right process:

 

(1)  Copy cache_dirs to the root of the flash drive

 

(2)  Add the line

 

cache_dirs -w -i "DVDs"

 

to my "Go" script to cache ONLY the share named DVDs

 

(3)  Reboot the server

 

Question:  Can I eliminate #3 by logging in to the server and manually typing the 'cache_dirs -w -i "DVDs"' line?

I'd like to preserve my current Uptime if possible  :)

 

... also, is the download for cache_dirs in the first post of this thread the current version?

 

I basically run UnRAID solely as a file server for my media collection (the "DVDs" share) and to keep a "Backups" share (that doesn't need to be cached).    The only add-ons I have installed are UnMenu, the PowerDown and APC UPS packages.    Is cache_dirs likely to cause any issues with these?    I've read a bit about the need to sometimes increase the cache_pressure value -- does this make it "safer" ??    I like the idea of always-available directory listings; but do NOT want to do anything that may reduce the reliability of the server.    If the versions make any difference, I'm running 4.6 now (updated about a month ago) ... and will probably switch to 4.7 after it's past Beta so any future EARS drives will be supported without jumpers (and to eliminate the need for a ULimit parameter).

 

 

Share this post


Link to post

Joe,

 

I've been thinking of adding this to my server for some time, but just want to be sure I do it correctly.    Can you confirm that the following is the right process:

 

(1)  Copy cache_dirs to the root of the flash drive

 

(2)  Add the line

 

cache_dirs -w -i "DVDs"

 

to my "Go" script to cache ONLY the share named DVDs

Correct, but you'll need to include its full path

/boot/cache_dirs -w -i "DVDd"

(assuming you put it at the root of your flash drive)

(3)  Reboot the server

 

Question:   Can I eliminate #3 by logging in to the server and manually typing the 'cache_dirs -w -i "DVDs"' line?

I'd like to preserve my current Uptime if possible  :)

Yes, you can do exactly that.  Again you'll either need to change directory to /boot to invoke it, or, use the full path to it.

... also, is the download for cache_dirs in the first post of this thread the current version?

Yes it is current

I basically run UnRAID solely as a file server for my media collection (the "DVDs" share) and to keep a "Backups" share (that doesn't need to be cached).    The only add-ons I have installed are UnMenu, the PowerDown and APC UPS packages.    Is cache_dirs likely to cause any issues with these?    I've read a bit about the need to sometimes increase the cache_pressure value -- does this make it "safer" ??    I like the idea of always-available directory listings; but do NOT want to do anything that may reduce the reliability of the server.    If the versions make any difference, I'm running 4.6 now (updated about a month ago) ... and will probably switch to 4.7 after it's past Beta so any future EARS drives will be supported without jumpers (and to eliminate the need for a ULimit parameter).

You probably do not need to do anything, but cache_dirs will set the cache_pressure to 10.  (default is 100)

There is an option to cache_dirs to set it as you please.  Just add "-p 100"  (or whatever you like) as an additional argument when you invoke it.

/boot/cache_dirs -p 100 -w -i "DVDs"

 

Share this post


Link to post

Thanks ... I'll add that in a few minutes.

 

What impact does the "cache pressure" have?

Is 100 "safer" than the default 10?    ("safer" in the sense it's less likely to cause problems on the server)

It's not clear to me what this parameter means -- I presume it influences when linux releases the cache .. is that right?

 

What happens when copying new movies to the server?  Does the cache simply get overwritten -- and result in spinning up all the other drives to re-read it?    Or is the refresh rate of cache_dirs fast enough that the directories will still stay in the buffer?   [i have 4GB of RAM]    Is this influenced by "cache pressure" ??

 

I assume cache_dirs should be stopped before doing a parity check -- otherwise it would seem that it will be continuously reading directory info as the buffers get used by the parity check routine.  [i never use my server during a parity check anyway]

 

 

Share this post


Link to post

hi Joe,

 

is this normal ?

 

ERROR: List of process IDs must follow -p.
********* simple selection *********  ********* selection by list *********
-A all processes                      -C by command name
-N negate selection                   -G by real group ID (supports names)
-a all w/ tty except session leaders  -U by real user ID (supports names)
-d all except session leaders         -g by session OR by effective group name
-e all processes                      -p by process ID
T  all processes on this terminal     -s processes in the sessions given
a  all w/ tty, including other users  -t by tty
g  OBSOLETE -- DO NOT USE             -u by effective user ID (supports names)
r  only running processes             U  processes for specified users
x  processes w/o controlling ttys     t  by tty
*********** output format **********  *********** long options ***********
-o,o user-defined  -f full            --Group --User --pid --cols --ppid
-j,j job control   s  signal          --group --user --sid --rows --info
-O,O preloaded -o  v  virtual memory  --cumulative --format --deselect
-l,l long          u  user-oriented   --sort --tty --forest --version
-F   extra full    X  registers       --heading --no-heading --context
                    ********* misc options *********
-V,V  show version      L  list format codes  f  ASCII art forest
-m,m,-L,-T,H  threads   S  children in sum    -y change -l format
-M,Z  security data     c  true command name  -c scheduling class
-w,w  wide output       n  numeric WCHAN,UID  -H process hierarchy
head: cannot open `/ver/run/mover.pid' for reading: No such file or directory
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
ERROR: List of process IDs must follow -p.

 

this is my startup line

 

/boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e ".custom" -e ".mysql" -w

 

and this is the version

 

Version 1.6.5

 

this output break through in a terminal where i run top

 

Share this post


Link to post

hi Joe,

 

is this normal ?

 

ERROR: List of process IDs must follow -p.
********* simple selection *********  ********* selection by list *********
-A all processes                      -C by command name
-N negate selection                   -G by real group ID (supports names)
-a all w/ tty except session leaders  -U by real user ID (supports names)
-d all except session leaders         -g by session OR by effective group name
-e all processes                      -p by process ID
T  all processes on this terminal     -s processes in the sessions given
a  all w/ tty, including other users  -t by tty
g  OBSOLETE -- DO NOT USE             -u by effective user ID (supports names)
r  only running processes             U  processes for specified users
x  processes w/o controlling ttys     t  by tty
*********** output format **********  *********** long options ***********
-o,o user-defined  -f full            --Group --User --pid --cols --ppid
-j,j job control   s  signal          --group --user --sid --rows --info
-O,O preloaded -o  v  virtual memory  --cumulative --format --deselect
-l,l long          u  user-oriented   --sort --tty --forest --version
-F   extra full    X  registers       --heading --no-heading --context
                    ********* misc options *********
-V,V  show version      L  list format codes  f  ASCII art forest
-m,m,-L,-T,H  threads   S  children in sum    -y change -l format
-M,Z  security data     c  true command name  -c scheduling class
-w,w  wide output       n  numeric WCHAN,UID  -H process hierarchy
head: cannot open `/ver/run/mover.pid' for reading: No such file or directory
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
ERROR: List of process IDs must follow -p.

 

this is my startup line

 

/boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e ".custom" -e ".mysql" -w

 

and this is the version

 

Version 1.6.5

 

this output break through in a terminal where i run top

 

no... not normal... Looks like the -p option is not being applied properly.

You are probably better off just changing the value in the script by using these commands

cd directory_where_you_unzipped_cache_dirs

sed -i "s/^cache_pressure=10/cache_pressure=200/" cache_dirs

 

I don't have the time to debug today.

Share this post


Link to post

still have same issue

even with that command that you specified above

 

what i find worrysome is the amount of processes i open ?

 

7809 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7810 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7812 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7813 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7814 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7815 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7816 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7818 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7819 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7820 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7821 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7859 pts/1    S      0:49 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 

 

or is this normal ?

if i put cache dirs off then i don't have this...

and it seems only to log these messages when the mover is running

 

i have my mover set to run every 6 hours..

Share this post


Link to post

still have same issue

even with that command that you specified above

 

what i find worrysome is the amount of processes i open ?

 

7809 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7810 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7812 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7813 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7814 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7815 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7816 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7818 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7819 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7820 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7821 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7859 pts/1    S      0:49 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 

 

or is this normal ?

if i put cache dirs off then i don't have this...

and it seems only to log these messages when the mover is running

 

i have my mover set to run every 6 hours..

 

The above is normal.  Those are all child processes of the cache_dirs script.

Share this post


Link to post

still have same issue

even with that command that you specified above

 

what i find worrysome is the amount of processes i open ?

 

7809 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7810 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7812 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7813 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7814 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7815 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7816 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7818 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7819 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7820 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7821 pts/1    S      0:00 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 -e .
7859 pts/1    S      0:49 /bin/bash /boot/cache_dirs -d 4 -m 3 -M 5 -p 200 

 

or is this normal ?

if i put cache dirs off then i don't have this...

and it seems only to log these messages when the mover is running

 

i have my mover set to run every 6 hours..

 

The above is normal.  Those are all child processes of the cache_dirs script.

Correct.  They are child processes, one per disk in your array, to keep you from seeing false "unformatted" messages when you attempt to stop the array and cache_dirs is running.  As stated, they are normal.

 

Joe L.

Share this post


Link to post

Joe .

 

Before you start troubleshooting your script ... give me a few days (will be out of town again this weekend ) to check if it was not the mysql who caused it ...

i just upgraded my box with a few things and he first didn't want to come up

started to troubleshoot

by reverting back what i installed changed since last reboot

once i reverted the auto install from the mysql did i get back into the server

 

will see what the cache_dirs does now without mysql  installed

might try to reinstall mysql next week

Share this post


Link to post

update

 

it is still doing it ... mover starts running and a few minutes later i get these in my terminal

 

Try `grep --help' for more information.
head: cannot open `/ver/run/mover.pid' for reading: No such file or directory
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
ERROR: List of process IDs must follow -p.
********* simple selection *********  ********* selection by list *********
-A all processes                      -C by command name
-N negate selection                   -G by real group ID (supports names)
-a all w/ tty except session leaders  -U by real user ID (supports names)
-d all except session leaders         -g by session OR by effective group name
-e all processes                      -p by process ID
T  all processes on this terminal     -s processes in the sessions given
a  all w/ tty, including other users  -t by tty
g  OBSOLETE -- DO NOT USE             -u by effective user ID (supports names)
r  only running processes             U  processes for specified users
x  processes w/o controlling ttys     t  by tty
*********** output format **********  *********** long options ***********
-o,o user-defined  -f full            --Group --User --pid --cols --ppid
-j,j job control   s  signal          --group --user --sid --rows --info
-O,O preloaded -o  v  virtual memory  --cumulative --format --deselect
-l,l long          u  user-oriented   --sort --tty --forest --version
-F   extra full    X  registers       --heading --no-heading --context
                    ********* misc options *********
-V,V  show version      L  list format codes  f  ASCII art forest
-m,m,-L,-T,H  threads   S  children in sum    -y change -l format
-M,Z  security data     c  true command name  -c scheduling class
-w,w  wide output       n  numeric WCHAN,UID  -H process hierarchy

 

nothing in syslog though... so you really need to have a telnet session open to have to receive these

 

Share this post


Link to post

Hi,

 

I love using cache dirs, but I'm trying to track down an issue with my music steamer running on my unRaid server (Subsonic). When I add a new folder, it doesn't show up unless I reboot. I shouldn't need to do this. Its not the browser cache and since I rebuilt the search index for Subsonic and deleted the folderCache files it uses, I'm looking at other possibilities.

 

Stopping and starting the cache_dirs script doesn't help. Is there a way to force cache_dirs to rescan? Are files left over that could be deleted?

 

Many Thanks!

Share this post


Link to post

Just waiting a few seconds typically 30 seconds is more than enough will have cache_dirs rescan. It is continually rescanning your directories by using the "find" command. There is no need to stop and start cache_dirs to force rescans.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.