Jump to content
Joe L.

cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up

1086 posts in this topic Last Reply

Recommended Posts

OK, I just installed it and put cache_dirs -w in my go script and drives were still spinning up the same as before.

 

So, I stopped it and then ran it at the prompt like - cache_dirs -F -v

The below is the result;

 

Executing find /mnt/disk2/Car_Music -noleaf

Executing find /mnt/disk1/Movies -noleaf

Executing find /mnt/disk2/Music -noleaf

Executing find /mnt/disk1/New_Movies -noleaf

Executing find /mnt/disk2/New_Movies -noleaf

Executing find /mnt/disk2/Pictures -noleaf

Executing find /mnt/disk2/Storage -noleaf

Executing find /mnt/disk2/TV_Shows -noleaf

Executing find /mnt/disk3/TV_Shows -noleaf

Executed find in 0.287845 seconds, weighted avg=0.289340 seconds, now sleeping 6 seconds

 

So, it appears to be going through all the directories and it did not spin up any drives the first pass when run on the comand line so it also appears the version running from the go script was working. I left it last night and the drvies all spun down sometime early this morning and stayed spun down while I stopped it and ran it from the command line.

 

While it was running as above, the drives still spun up if I tried to navigate to almost any directory on them. I could go to the top level of the shares but that was about it. As I understand it, it will scan to a default maxium directory depth of 99 and I don't have anything beyond about 4 or 5 levels.

 

I am using W7 Windows Explorer. I specifically set the default view to "List" which should not try to gather any specific info about the file or create a thumbnail of the file. I also turned off the "Details" pane and the "Preview" pane as well.  Actually, the preview was off already.

 

I then tried the -M 1 option and it seems a little better but still spins up disks.also I also tried -p 1 option but not much change. However, after both of these options it seemed to me there was a higher chance the disk would spin up when I pressed the back button while surfing directories. But, I still didn't notice much if any pattern to it.

 

So, anyone have any ideas? Is this a Windows Explorer thing? Anyone else know if W7 Windows Explorer will still read file info no matter what.

 

Peter

 

Share this post


Link to post

Hi,

 

i start cache_dir with -w -B. All works good, but one of my datadisk and my parity hdd spin up after some hours. Did cache_dirs write anythin on my share or my disk when i start it with -w -B, and if yes how could i turn off this feature ?

 

Greets

Eisi

 

Share this post


Link to post

Hi,

 

i start cache_dir with -w -B. All works good, but one of my datadisk and my parity hdd spin up after some hours. Did cache_dirs write anythin on my share or my disk when i start it with -w -B, and if yes how could i turn off this feature ?

 

Greets

Eisi

 

cache_dirs writes nothing to the disks, your PC might be scanning for new files,  If parity spun up, then one of your disks was written to.  Perhaps it is making thumbnail images.

 

Joe L.

Share this post


Link to post

Have been using this for a while and I love it. Fast as hell.

 

Quick question: If I add a new folder to an existing share, will this acknowledge the newly added folder?

Share this post


Link to post

Have been using this for a while and I love it. Fast as hell.

 

Quick question: If I add a new folder to an existing share, will this acknowledge the newly added folder?

yes.

Share this post


Link to post

Hello all....I have a few questions that will build on each other....

 

I am using the Lime-Tech server with a MediaMax media server.  The mediamax scans the cover art every 10 minutes and this was causing skipping when watching a movie, especially blu-ray.  I installed the Cache-Dirs and edited my go script.  It seemed to take care of the problem until I added more movies.  I have throughly tested this and here is what is happening.

 

Whenever I put new movies on the server the skipping starts again, the more I put on the server without a restart the worse it gets.  If I restart the server, the problem goes away until i add more movies with out a restart.

 

Question 1--It seems that the cache-dirs is only ran during the start up, is this true?  That would explain why it skips after I add new movies, they wouldn't have been cached yet.

 

Question 2-- If that is the case, is there a way to automate the process.  I.E. have the server automatically , take the array off line, and reboot say at a specific time 4am or something.  This would completely eliminate the problem I believe.

 

As of right now, when ever I don't reboot the server after I add movies, the skipping starts.  After I reboot it, everything is fine.  Ideas?

 

Share this post


Link to post

Answer 1) No that is not the reason.

 

Answer 2) See answer 1.

 

Just to make things perfectly clear, after cache_dirs is invoked, it will be running in the background until the server is turned off. You only need to start cache_dirs once, typically when the array is online, or you need to tell it to wait until the array is online '-w' option. You never need to run cache_dirs again, even after adding files.

 

It only caches file and directory listings. When your media player is scanning for new movies, the new movie file and directory entries should be read from the cached entries. When your media player reads the contents of the files however, the discs must be spun up. The cache_dirs script can do nothing to cache actual file CONTENT. Spinning up idle disks can result in hiccups. This is the intent of spin-up groups, so you have your drives spun up as needed by functionality such as 'all movies'.

Share this post


Link to post

Ok, but that doesn't make sense.....or maybe I just don't understand what you are saying.....Why would a disc that is all ready spun up, all ready being read, and playing back a movie hiccup when a different drive is spun up?  I have tested this a ton....when I add new movies with out a reboot, it hiccups, when I reboot, the hiccups are gone.  Now I guess this could be because the discs are spun up on start up.... but even after all the discs spin down, as long as I don't add any new movies...no hiccups.  I apologize if I an not clear.

 

And if it is the spin up that is causing the issure, what do I do to prevent that from happening?

 

 

Answer 1) No that is not the reason.

 

Answer 2) See answer 1.

 

Just to make things perfectly clear, after cache_dirs is invoked, it will be running in the background until the server is turned off. You only need to start cache_dirs once, typically when the array is online, or you need to tell it to wait until the array is online '-w' option. You never need to run cache_dirs again, even after adding files.

 

It only caches file and directory listings. When your media player is scanning for new movies, the new movie file and directory entries should be read from the cached entries. When your media player reads the contents of the files however, the discs must be spun up. The cache_dirs script can do nothing to cache actual file CONTENT. Spinning up idle disks can result in hiccups. This is the intent of spin-up groups, so you have your drives spun up as needed by functionality such as 'all movies'.

Share this post


Link to post

Here is my go script if it helps

 

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp &
/boot/cache_dirs  -d  5  -m  3  -M  5  -w
[/cpde]

Share this post


Link to post

Why would a disc that is all ready spun up, all ready being read, and playing back a movie hiccup when a different drive is spun up? 

 

You must not have been around when we worked through the various cases of stuttering back in the 4.5 beta days.

 

Drives on the same controller just do that. If I/O is occurring on a spunup drive, and an I/O request is issued to a spundown drive on the same host controller, then I/O will freeze on the first drive until the spundown drive gets spunup. This is because the common host controller is busy spinning up a drive and can not process I/O requests for other drives it controls. It's just the way current controllers function.

 

Have a read: http://lime-technology.com/forum/index.php?topic=4782.0 Try to read the entire thread if you can, at least read everything in the first post.

 

 

Share this post


Link to post

BRIT, are there any controllers that prevents this? The freezing of a drive when spinning up other disks and implementing spinup groups as a solution to this, certainly neglects the idea of unRAID having independent disks and it's major benefit compared to traditional RAID arrays where all disks either sleep or spin.

 

I mean, one either must be single user to an unRAID server or put all disks in a single spinup group to avoid freezing the second disk when another is already being read?! 

Share this post


Link to post

So this is my problem?

 

3. An I/O stream via Windows PC freezes when same Windows PC is used to access data on a spundown disk.  This is a Windows problem and can not be fixed by unRAID OS.  From examining SMB connection traces, what I have determined is this: When a Windows PC (client) connects to a Server, it opens a single "SMB Connection".  Over this connection it sends it's requests: lookups, read, writes, etc, no matter which Share is being accessed (that is, all shares on the same server are accessed over a single SMB connection).  Hence, say for example, video is being read from the server and rendered on a Windows PC; if from the same Windows PC you now try to access another share with a spundown drive, Windows will use the same connection, resulting in "freeze" of the video while the I/O reqeust to the spundown drive is being serviced.

 

I have a windows based client (MediaMax) and I have 2 servers, Lime-Technology and MedaMax (windows based).  I am not getting any studdering when playing of the windows based server, just the Lime-Tech one.  If the above is the culprit, shouldn't this be happening on any server I use?

 

 

Share this post


Link to post

Joe I started reading about kernel tubales again.

 

And yet again if stumbled on http://kerneltrap.org/node/4462

 

Perhaps there are a couple of more tunables that could be of use in unRAID very special uage case:

 

/*

* "vm_passes" is the number of vm passes before failing the

* memory balancing. Take into account 3 passes are needed

* for a flush/wait/free cycle and that we only scan 1/vm_cache_scan_ratio

* of the inactive list at each pass.

*/

int vm_passes = 60;

/*

* "vm_cache_scan_ratio" is how much of the inactive LRU queue we will scan

* in one go. A value of 6 for vm_cache_scan_ratio implies that we'll

* scan 1/6 of the inactive lists during a normal aging round.

*/

int vm_cache_scan_ratio = 6;

/*

* "vm_mapped_ratio" controls the pageout rate, the smaller, the earlier

* we'll start to pageout.

*/

int vm_mapped_ratio = 100;

/*

* "vm_lru_balance_ratio" controls the balance between active and

* inactive cache. The bigger vm_balance is, the easier the

* active cache will grow, because we'll rotate the active list

* slowly. A value of 2 means we'll go towards a balance of

* 1/3 of the cache being inactive.

*/

int vm_lru_balance_ratio = 2;

/*

* "vm_vfs_scan_ratio" is what proportion of the VFS queues we will scan

* in one go. A value of 6 for vm_vfs_scan_ratio implies that 1/6th of

* the unused-inode, dentry and dquot caches will be freed during a normal

* aging round.

*/

int vm_vfs_scan_ratio = 6;

/*

* "vm_anon_lru" select if to immdiatly insert anon pages in the

* lru. Immediatly means as soon as they're allocated during the

* page faults.

*

* If this is set to 0, they're inserted only after the first

* swapout.

*

* Having anon pages immediatly inserted in the lru allows the

* VM to know better when it's worthwhile to start swapping

* anonymous ram, it will start to swap earlier and it should

* swap smoother and faster, but it will decrease scalability

* on the >16-ways of an order of magnitude. Big SMP/NUMA

* definitely can't take an hit on a global spinlock at

* every anon page allocation. So this is off by default.

*

* Low ram machines that swaps all the time want to turn

* this on (i.e. set to 1).

*/

 

 

Specifically vm_vfs_scan_ratio and vm_cache_scan_ratio may be useful.

 

I am suffering from lack of documentation. It seems when you get to this level you are expected to be able to speak in kernel code.

Share this post


Link to post

So should this addon be used in conjunction with sysctl vm.vfs_cache_pressure=1 off or not?  The wiki is confusing on that point.  Thanks.

Share this post


Link to post

So should this addon be used in conjunction with sysctl vm.vfs_cache_pressure=1 off or not?  The wiki is confusing on that point.  Thanks.

The wiki needs updating actually.  My personal experience is unless you have more than 512Meg you can run out of ram for processes as it is all horded for disk buffer cache.  (and I hav no idea how much is enough) 

 

Cache_dirs already sets cache_pressure.  By default it sets it to 10.  Even that is too low a value for me on my server if I'm trying to perform certain tasks. 

 

If you are running add-ons, either set it higher (200), or at least put it up to 100.

To do that you use the

-p NNN

option.

Share this post


Link to post

Thanks!  I've got 2GB.  I'm running a couple addons but nothing crazy yet.  I would like to try out ps3mediaserver or another media server, which I expect will be a resource hog.  How would I diagnose if cache_pressure needs to be increased?

 

The wiki needs lots of updating.  There's a ton of using links to the forum as a crutch.  Not very friendly noob-friendly :)

Share this post


Link to post

Thanks!  I've got 2GB.  I'm running a couple addons but nothing crazy yet.  I would like to try out ps3mediaserver or another media server, which I expect will be a resource hog.  How would I diagnose if cache_pressure needs to be increased?

 

The wiki needs lots of updating.  There's a ton of using links to the forum as a crutch.  Not very friendly noob-friendly :)

 

The User Contributed Wiki is editable by anyone.  Feel free to create an account on the wiki site and edit to your hearts content.  We update when and where we can, but it is just like any other documentation and can get out of date.

Share this post


Link to post

How would I diagnose if cache_pressure needs to be increased?

Easy.  If you run out  of memory and processes start getting killed off it is too low.  (Hint: emhttp and smbd tend to be some of the first to be killed off)

The wiki needs lots of updating.  There's a ton of using links to the forum as a crutch.  Not very friendly noob-friendly :)

As stated... pitch in and help, even if just pointing out what is out of date.

Share this post


Link to post

Oh sure, I didn't mean to sound whiny.  I think I'll go update that section with info from Joe L.'s reply.  I'm just saying as whole, the wiki can be pretty confusing once you get past the very basic stuff.

Share this post


Link to post

Oh sure, I didn't mean to sound whiny.  I think I'll go update that section with info from Joe L.'s reply.  I'm just saying as whole, the wiki can be pretty confusing once you get past the very basic stuff.

I already did, but you are welcome to edit and improve it even more.

Share this post


Link to post

I am running cache_dirs, and while it runs fantastically (and quick) i seem to be experiencing an issue not explicitly covered in this forum.

 

The very first time i need to copy a file onto unraid, it appears to max out the memory first (2GB), but when the memory is full the copy fails and says network is no longer available.

 

unraid then slowly writes the 2GB to disk (if i refresh it i can see the file size increasing in unraid) until it hits 2GB, then it displays the full file size (say 10GB).

 

once that whole process finishes, i can recopy that file (replacing the original) and everything works fine, including all subsequent file copies (regardless of size).

 

Not sure if anyone has experienced this or knows a solution? It doesnt pose a real problem since after the initial problem i can still copy, but would rather not have to go through that process everytime i boot my server.

 

Thanks

Share this post


Link to post

I am running cache_dirs, and while it runs fantastically (and quick) i seem to be experiencing an issue not explicitly covered in this forum.

 

The very first time i need to copy a file onto unraid, it appears to max out the memory first (2GB), but when the memory is full the copy fails and says network is no longer available.

 

unraid then slowly writes the 2GB to disk (if i refresh it i can see the file size increasing in unraid) until it hits 2GB, then it displays the full file size (say 10GB).

 

once that whole process finishes, i can recopy that file (replacing the original) and everything works fine, including all subsequent file copies (regardless of size).

 

Not sure if anyone has experienced this or knows a solution? It doesnt pose a real problem since after the initial problem i can still copy, but would rather not have to go through that process everytime i boot my server.

 

Thanks

Use the "-p NNN" option to cache_dirs to set the cache pressure higher than the value set by cache_dirs by default.

 

I would suggest a value of NNN somewhere between 100 and 200, perhaps higher.  It sounds as if the copy process is running out of memory and cannot get any since the disk buffer cache has been instructed to horde it.

 

Try

cache_dirs -q

to quit the currently running cache_dirs

followed by

cache_dirs -p 200

to re-start it with a new cache_pressure value.

 

Joe L.

Joe L.

Share this post


Link to post

Hello Joe.  I just reviewed my syslog after fixing the other "Missing Disk" problem that you helped with me.

 

I noticed that the Cache_Dir program does not seem to like folder names with spaces.  Is there a way other than eliminating the space to get around this?

 

Excerpt from syslog:

 

Dec 3 13:33:43 Tower cache_dirs: ==============================================

Dec 3 13:33:43 Tower cache_dirs: command-args=-w -i Movies -i TV Shows -i Backdrops

Dec 3 13:33:43 Tower cache_dirs: vfs_cache_pressure=1

Dec 3 13:33:43 Tower cache_dirs: max_seconds=10, min_seconds=1

Dec 3 13:33:43 Tower cache_dirs: max_depth=9999

Dec 3 13:33:43 Tower cache_dirs: command=find -noleaf

Dec 3 13:33:43 Tower cache_dirs: version=1.6.5

Dec 3 13:33:43 Tower cache_dirs: ---------- caching directories ---------------

Dec 3 13:33:43 Tower cache_dirs: Backdrops

Dec 3 13:33:43 Tower cache_dirs: TV Shows

Dec 3 13:33:43 Tower cache_dirs: Movies

Dec 3 13:33:43 Tower cache_dirs: ----------------------------------------------

Dec 3 13:33:43 Tower cache_dirs: ERROR: included directory "TV Shows" does not exist.

Dec 3 13:33:43 Tower cache_dirs: cache_dirs process ID 1915 started, To terminate it, type: cache_dirs -q

Share this post


Link to post

Hello Joe.  I just reviewed my syslog after fixing the other "Missing Disk" problem that you helped with me.

 

I noticed that the Cache_Dir program does not seem to like folder names with spaces.  Is there a way other than eliminating the space to get around this?

 

Excerpt from syslog:

 

Dec 3 13:33:43 Tower cache_dirs: ==============================================

Dec 3 13:33:43 Tower cache_dirs: command-args=-w -i Movies -i TV Shows -i Backdrops

Dec 3 13:33:43 Tower cache_dirs: vfs_cache_pressure=1

Dec 3 13:33:43 Tower cache_dirs: max_seconds=10, min_seconds=1

Dec 3 13:33:43 Tower cache_dirs: max_depth=9999

Dec 3 13:33:43 Tower cache_dirs: command=find -noleaf

Dec 3 13:33:43 Tower cache_dirs: version=1.6.5

Dec 3 13:33:43 Tower cache_dirs: ---------- caching directories ---------------

Dec 3 13:33:43 Tower cache_dirs: Backdrops

Dec 3 13:33:43 Tower cache_dirs: TV Shows

Dec 3 13:33:43 Tower cache_dirs: Movies

Dec 3 13:33:43 Tower cache_dirs: ----------------------------------------------

Dec 3 13:33:43 Tower cache_dirs: ERROR: included directory "TV Shows" does not exist.

Dec 3 13:33:43 Tower cache_dirs: cache_dirs process ID 1915 started, To terminate it, type: cache_dirs -q

 

Hello Mace, try a "\" between the space

 

For example TV\ shows

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.