Jump to content
Joe L.

cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up

1086 posts in this topic Last Reply

Recommended Posts

Wont mediabrowser be accessing the mymovies.xml files in each folder to verify it is current?

 

That is corret. My media frontend (SageTV) also reads the cover images of all my videos.

So the caching script that I am using copies all .jpg files it finds to /dev/null

 

Purko

 

 

Purko,

 

Could you please explain how you modified this script?

I would love to have a way to give media browser instant access to the.xml and folder.jpg files so that all my drives don't have to spin up when browsing my library.

 

Thanks

Share this post


Link to post

I am noticing some strange phenomenom. When I call cache_dirs -w -B from the go script, my drives do not spin down. When I kill the process and start it manually on the cmdline, same syntax, they will properly spin down after the specified amount of time. Any hints as to what might be causing this behaviour? For what it matters I am calling cache_dirs -w -B right before calling unMenu in my go script.

Share this post


Link to post

I am noticing some strange phenomenom. When I call cache_dirs -w -B from the go script, my drives do not spin down. When I kill the process and start it manually on the cmdline, same syntax, they will properly spin down after the specified amount of time. Any hints as to what might be causing this behaviour? For what it matters I am calling cache_dirs -w -B right before calling unMenu in my go script.

I've got no idea...  One difference I can think of is the search PATH used for commands might be different. (Causing the shell to traverse different directories when looking for the commands to run)

 

 

My cache_dirs lives in /boot/custom/bin.  I invoke it like this in my "go" script.  my drives spin down just fine.  I add /boot/custom/bin to the execution path in the "go" script.

(This is my whole "go" script.)

#!/bin/bash

# Start the Management Utility

/usr/local/sbin/emhttp &

PATH=$PATH:/boot/custom/bin

cache_dirs -w -e Pictures -e data -d 3

fromdos < /boot/custom/bin/powerdown > /sbin/powerdown

chmod u+x /sbin/powerdown

fromdos < /boot/custom/etc/rc.d/S30-inittab-powerdown | sh

fromdos < /boot/custom/etc/rc.d/rc.local_startup | sh

echo "/boot/unmenu/uu" | at now + 1 minute

 

What version of cache_dirs are you running?  What version of unRAID are you running?

 

Other than that, something is either

1.  Accessing the disks, keeping them spinning

or

2.  Not finding the needed directory information in the buffer cache, so it has to spin up a drive to get the data.

or

3.  Your observation, although true, might have nothing to do with the actual issue.  It might be something entirely un-related to how you invoke it.

 

 

Share this post


Link to post

I have cache_dirs 1.6.4 and unRAID 4.5 final. fuser don't show any processes on the /mnt/disk*. Also, I have about 130k files and 4GB memory. I don't have $PATH in my go. I just start /boot/scripts/cache_dirs -w -B

 

Is it ok to have two cache_dir processes running? The one starts the other, it has its PID as parents PID in the second.

 

root 300 1 0 Jan07 pts/1 00:00:00 /bin/bash /boot/scripts/cache_dirs -w -B

root 3874 300 0 00:38 pts/1 00:00:00 /bin/bash /boot/scripts/cache_dirs -w -B

Share this post


Link to post

I have cache_dirs 1.6.4 and unRAID 4.5 final. fuser don't show any processes on the /mnt/disk*. Also, I have about 130k files and 4GB memory. I don't have $PATH in my go. I just start /boot/scripts/cache_dirs -w -B

 

Is it ok to have two cache_dir processes running? The one starts the other, it has its PID as parents PID in the second.

 

root       300     1  0 Jan07 pts/1    00:00:00 /bin/bash /boot/scripts/cache_dirs -w -B

root      3874   300  0 00:38 pts/1    00:00:00 /bin/bash /boot/scripts/cache_dirs -w -B

I'm running the same versions.  I only cache three levels deep.  I only have 512 meg of RAM in my server and under 5000 files in my Movies and Music folders.  You have lots more files you are attempting to cache, but far more memory too.

 

Yes, it is normal to see two processes. part of the script is run in a sub-shell.  (The while loop is run as a sub-shell)

 

 

Share this post


Link to post

It used to work when I first installed/started it by hand. I added it to the go script and then after time had to reboot adding a new drive. Then I noticed that the drives won't spin down. Killed, started by hand and it worked again. Restarted again the server, same - no spin down. However, since last manual restart drives are still spinning and I don't know why.

Share this post


Link to post

I have cache_dirs 1.6.4 and unRAID 4.5 final. fuser don't show any processes on the /mnt/disk*

fuser is a snapshot in time.  If at that instant in time no access was occurring, then nothing will show.   It is not able to show you things that happened 10 minutes ago and lasted a fraction of a second as a directory or file was accessed..

 

For that, we need something that can track all file and directory accesses, and log all of them.

Share this post


Link to post

lsof? It too doesn't show any processes being on /mnt/disk*

 

Edit: I was also wondering what is the default access mode for files on unRAID? I have ownership set for everything to root:root and access rights to 750 for dirs and 640 for files. Can remember that I was changing this lately... But shouldn't have to do anything with the strange behaviour described above?!

 

Edit2: Well, it is definitely cache_dirs spinning up the drives. I sync'ed on the cmdline a couple of times and them spun down the array from unRAID Main. After some time, the disks went up again, one at a time corresponding to the find processes started from cache_dirs...

 

Edit3: I am running unraid_notify, can this be the cause? It says it won't send notification if disks are spun down though, wouldn't do anything to them to spin them up.

Share this post


Link to post

I have cache_dirs 1.6.4 and unRAID 4.5 final. fuser don't show any processes on the /mnt/disk*. Also, I have about 130k files and 4GB memory. I don't have $PATH in my go. I just start /boot/scripts/cache_dirs -w -B

 

Is it ok to have two cache_dir processes running? The one starts the other, it has its PID as parents PID in the second.

 

root       300     1  0 Jan07 pts/1    00:00:00 /bin/bash /boot/scripts/cache_dirs -w -B

root      3874   300  0 00:38 pts/1    00:00:00 /bin/bash /boot/scripts/cache_dirs -w -B

I'm running the same versions.  I only cache three levels deep.  I only have 512 meg of RAM in my server and under 5000 files in my Movies and Music folders.   You have lots more files you are attempting to cache, but far more memory too.

 

Yes, it is normal to see two processes. part of the script is run in a sub-shell.  (The while loop is run as a sub-shell)

 

 

Hi Joe,

I also have lots of files and directories on my server and experience a lot of reading on the disks. I upgraded memory to make sure, it is sufficient to hold all dir-infos. Nevertheless my disks DO spindown (all), but often reread all contents. Might be caused, when I write to the array so the buffers are reused for writing?

So my question (I am Linux noob....): Is there a way/tools to check/monitor what's happening? currently I used "free" to check memory usage and the mgmt page to see read operations on the drives. So I can see how many reads I had for first run of cache_dirs - and after some days of operation I see 10 or 20 times the amount, although there was no "real" read on the disks.

Thanks, Guzzi

 

Edit: I use S3 and wake the server via WOL - does cache_dirs a reread after S3? Does UnRaid flush the buffers after S3-wake? Hmmm, that would explain it - how can I verify this and is there a way to avoid it?

 

Here is my "free" output:

            total      used      free    shared    buffers    cached

Mem:      4017452    2313480    1703972          0    248792    1391780

-/+ buffers/cache:    672908    3344544

Swap:            0          0          0

Do I read right, that mem extension from 2 to 4 GB wastn't necessary?

 

Share this post


Link to post

Hi Joe L.,

I have another question: What is the usecase for the "-u" parameter?  I understand it reads and caches the usershare - but I thought, this is already working, if the disk dirs are cached?

If -u is enabled - would the excludes still work? (Because I have mapped dirs in the usershare, that I do not want to cache)

Thanks, Guzzi

Share this post


Link to post

Hi Joe L.,

I have another question: What is the usecase for the "-u" parameter?  I understand it reads and caches the usershare - but I thought, this is already working, if the disk dirs are cached?

If -u is enabled - would the excludes still work? (Because I have mapped dirs in the usershare, that I do not want to cache)

Thanks, Guzzi

It would be used for Experimentation only.  User-shares are in memory already.  No need to cache since access is quick regardless.    It was made an option since user-shares are not "cached" specifically by cache_dirs by default.  When we were first trying to identify what we needed to do to keep from spinning up a disk some thought the user-shares mith need to be cached too.  They are not unless you use the -u option, but access is fast regardless.  If you have a lot of ram you can experiment with using the -u option or not.  General opinion is it will not make a difference (other than to cause the CPU to do more work).

 

Joe L.

Share this post


Link to post

Thanks for explanation - I currently have it installed and checked with htop - so CPU usage seems pretty high, but can't say if it was the same before without -u.

Will just let it run some days and see "how it feels".

rgds, Guzzi

Share this post


Link to post

Thanks for explanation - I currently have it installed and checked with htop - so CPU usage seems pretty high, but can't say if it was the same before without -u.

Will just let it run some days and see "how it feels".

rgds, Guzzi

A lot will depend on how many files & folders you have... basically, you are now doing a "find" on twice as many. 

 

It might even be worse, as the entries from the disk shares you want to stay in the cache, to keep them from spinning down, might be displaced by the extra ones read by scanning the user share's twin copies.

Share this post


Link to post

Thank you Joe for creating this!  I no longer have any pauses when using my media player to browse my unRaid server  ;D

 

I put the script in a folder like you indicated on an earlier post in this thread and added the appropriate lines (to set the path and kick off the script) in my go file.  After rebooting, the array automatically started a parity check.  Is that normal?  Once it completed, everything was working just fine.

Share this post


Link to post

Thank you Joe for creating this!  I no longer have any pauses when using my media player to browse my unRaid server  ;D

 

I put the script in a folder like you indicated on an earlier post in this thread and added the appropriate lines (to set the path and kick off the script) in my go file.  After rebooting, the array automatically started a parity check.  Is that normal?   Once it completed, everything was working just fine.

The parity check is only normal if you did not stop the array cleanly before rebooting.

 

You might want to post a syslog.  If you think you stopped the array before rebooting, and the parity check was unexpected, then you might want to post a syslog to see if there was a different reason for the parity check.

 

Joe L.

Share this post


Link to post

Thanks for explanation - I currently have it installed and checked with htop - so CPU usage seems pretty high, but can't say if it was the same before without -u.

Will just let it run some days and see "how it feels".

rgds, Guzzi

A lot will depend on how many files & folders you have... basically, you are now doing a "find" on twice as many. 

 

It might even be worse, as the entries from the disk shares you want to stay in the cache, to keep them from spinning down, might be displaced by the extra ones read by scanning the user share's twin copies.

 

so this is the result after running the box overnight:

All disks are spun down and memoryusage is as follows:

 

root@TOWER:/# free

            total      used      free    shared    buffers    cached

Mem:      4017452    3866908    150544          0    247984    2939772

-/+ buffers/cache:    679152    3338300

Swap:            0          0          0

 

I ususally access the server for files via the

[TOWER-Root] - path = /mnt/user

entry in smb-extra.conf to get all usershares on one drive; this access (from windows SMB1) is fast and no disks spin up.

 

Can you comment on how to read the output of free-command?

 

I check the same without -u parameter to compare later...

 

THanks, Guzzi

Share this post


Link to post

Thank you Joe for creating this!  I no longer have any pauses when using my media player to browse my unRaid server  ;D

 

I put the script in a folder like you indicated on an earlier post in this thread and added the appropriate lines (to set the path and kick off the script) in my go file.  After rebooting, the array automatically started a parity check.  Is that normal?   Once it completed, everything was working just fine.

The parity check is only normal if you did not stop the array cleanly before rebooting.

 

You might want to post a syslog.  If you think you stopped the array before rebooting, and the parity check was unexpected, then you might want to post a syslog to see if there was a different reason for the parity check.

 

Joe L.

 

That explains it then.  I did not stop the array via the web interface.  I only pressed the Reset button on my MD-1510.  Thanks!

Share this post


Link to post

Thank you Joe for creating this!  I no longer have any pauses when using my media player to browse my unRaid server  ;D

 

I put the script in a folder like you indicated on an earlier post in this thread and added the appropriate lines (to set the path and kick off the script) in my go file.  After rebooting, the array automatically started a parity check.  Is that normal?   Once it completed, everything was working just fine.

The parity check is only normal if you did not stop the array cleanly before rebooting.

 

You might want to post a syslog.  If you think you stopped the array before rebooting, and the parity check was unexpected, then you might want to post a syslog to see if there was a different reason for the parity check.

 

Joe L.

 

That explains it then.  I did not stop the array via the web interface.  I only pressed the Reset button on my MD-1510.  Thanks!

Don't know how to say this any other way, but don't just hit the reset button unless absolutely necessary and you have no way to reboot otherwise.  It might cause you to lose data... if the files being written to disks had not yet been flushed from the disk buffer cache to the physical disks they might be mangled/partially written, or lost/never written.

 

Always stop the array first using the "Stop" button on the web-interface.  Then you can reset/reboot/power down as needed.

When you power back up, the array will know it shut down cleanly and not need to perform a full parity check.

 

Joe L.

Share this post


Link to post

That explains it then.  I did not stop the array via the web interface.  I only pressed the Reset button on my MD-1510.  Thanks!

Don't know how to say this any other way, but don't just hit the reset button unless absolutely necessary and you have no way to reboot otherwise.  It might cause you to lose data... if the files being written to disks had not yet been flushed from the disk buffer cache to the physical disks they might be mangled/partially written, or lost/never written.

 

Always stop the array first using the "Stop" button on the web-interface.  Then you can reset/reboot/power down as needed.

When you power back up, the array will know it shut down cleanly and not need to perform a full parity check.

 

Joe L.

 

Thanks for the explanation.  Got it now! :) 

Share this post


Link to post

Can someone give me a standard command to cache everything on my unRAID server?

 

I'm currently using:

/boot/cache_dirs -d 10 -w

 

However, disk 1 likes to spin up when browsing certain directories (like pictures, or program installers). Is this normal?

Share this post


Link to post

Is it possible to use the 'exclude_dir' setting to exclude an entire drive? I have a non-array drive on the system that'd i'd rather not cache.

 

Thanks, Joe, for another useful tool!

Share this post


Link to post

Just tried this. It seemed to run as it spun up all my disks and they were grinding away for a while. Then I ran "cache_dirs -q" to stop it and started it with "cache_dirs -F" so I can see output but now it just sits there on a blank line forever, no output, nothing. It seems to run fine without the -F option so I guess it's no big deal. Just thought it was supposed to spit back info as it ran.

Share this post


Link to post

Just tried this. It seemed to run as it spun up all my disks and they were grinding away for a while. Then I ran "cache_dirs -q" to stop it and started it with "cache_dirs -F" so I can see output but now it just sits there on a blank line forever, no output, nothing. It seems to run fine without the -F option so I guess it's no big deal. Just thought it was supposed to spit back info as it ran.

You know... you are the first to describe this behavior since I posted the last version of cache_dirs.  

 

Guess what.  In adding the feature to suspend the script during the "mover" scripts operation, I broke cache_dirs.

 

It is supposed to spit_back info as it ran.  It is broken if you do not have a cache drive defined, as you discovered, a major part of the loop is skipped and no detail is printed.

 

Since last October, over 200 people have downloaded it, and you are the first to notice the problem. (indirectly, at least)

 

Thanks so much for the report of the problem you were having.  I'm just attached a newer version of cache_dirs to the first post in its thread.  I'll go hang my head in shame for my logic oversight.... for a few minutes anyway.  ;)

 

Version 1.6.5 is now attached to the first post in this thread.  It should work as you expected.

Joe L.

Share this post


Link to post

I really like this feature a lot - only problem I've found is that my array has grow to big to have everything in the 1GB of RAM my system holds....

 

Guess I need to buy 4GB instead.

Share this post


Link to post

I really like this feature a lot - only problem I've found is that my array has grow to big to have everything in the 1GB of RAM my system holds....

 

Guess I need to buy 4GB instead.

That is why you can limit the number of levels cached, and why you can exclude folders where the spin-up delay is not an issue.  I have only 512 Meg of ram in my server.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.