cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up

[Joe L.]

I can't believe this hasn't been caught by anyone else, but there is an error on line 460 in the script

 

     mover_pid=`head -n 1 /ver/run/mover.pid`

 

it should be

 

     mover_pid=`head -n 1 /var/run/mover.pid`

 

As far as I know, no implementation of unraid has a /ver directory.

thank you.  you are right. /ver does not exist.

[WingmanNZ]

Hi Joe, just curious what the best settings for a setup like so would be?

 

(I have Split Level 0 and manually create the folders on the individual disks then allow high-water to fill the disks add i like having a single share )

 

Media

|_____Documentaries

|_____Fitness & Training

|         |_____Bluray

|         |         |_____1080p

|         |         |_____720p

|         |_____DVDR

|         |_____DVDRip

|_____Miniseries

|_____Movies

|         |_____Bluray

|         |         |_____1080p

|         |         |_____720p

|         |_____DVDR

|         |_____DVDRip

|         |_____HDDVD

|         |         |_____1080p

|         |         |_____720p

|         |_____HDTV

|         |         |_____1080p

|         |         |_____720p

|         |_____WEB-DL

|                   |_____1080p

|                   |_____720p

|_____Short Films

|         |_____Bluray

|         |         |_____1080p

|         |         |_____720p

|         |_____DVDRip

|         |_____WEB-DL

|                   |_____1080p

|                   |_____720p

|_____TV Shows

[Joe L.]

Hi Joe, just curious what the best settings for a setup like so would be?

 

(I have Split Level 0 and manually create the folders on the individual disks then allow high-water to fill the disks add i like having a single share )

 

Media

|_____Documentaries

|_____Fitness & Training

|         |_____Bluray

|         |         |_____1080p

|         |         |_____720p

|         |_____DVDR

|         |_____DVDRip

|_____Miniseries

|_____Movies

|         |_____Bluray

|         |         |_____1080p

|         |         |_____720p

|         |_____DVDR

|         |_____DVDRip

|         |_____HDDVD

|         |         |_____1080p

|         |         |_____720p

|         |_____HDTV

|         |         |_____1080p

|         |         |_____720p

|         |_____WEB-DL

|                   |_____1080p

|                   |_____720p

|_____Short Films

|         |_____Bluray

|         |         |_____1080p

|         |         |_____720p

|         |_____DVDRip

|         |_____WEB-DL

|                   |_____1080p

|                   |_____720p

|_____TV Shows

just

cache_dirs -w

 

[WingmanNZ]

Cheers for that mate, any known issues with 5.0b8d?

 

[Joe L.]

Cheers for that mate, any known issues with 5.0b8d?

 

no

[olympia]

Hi Joe L.,

 

not sure this has been already brought up by someone, but I think it hasn't.

 

When you stop cache_dirs with 'cache_dirs -q' it is only terminating the cache_dirs instance running in the background, but it is not killing any running (ongoing) 'find' command. So when you try to stop the array soon after you started up cache_dirs, the running 'find' command(s) can keep drive(s) busy even for longer time, resulting a long 'unmounting' on the unRAID interface unless you kill the 'find(s)' manually.

 

Could killing of 'find' command(s) be added to 'cache_dirs -q' so that it can shut down gracefully?

 

Thank you for your feedback in advance.

[methanoid]

Newbie question please (be gentle I'm not even running UnRAID yet!)

 

Does using this option consume much memory? I've got about 10Tb of media and would like to avoid spinning up drives just to look at what I have. I guess I am also wondering how much Ram UnRAID itself needs too.

 

 

[Joe L.]

I have 512 Meg of RAM in my older server and it has 6TB of media.  Been using it for nearly 6 years.

 

unRAID, needs at least 512 Meg.  It will use ALL you give it for disk buffer cache.  In other words, if you were to install 16Gig of RAM and watch 4 DVD ISO Images, all your RAM would be used as buffer cache.  the RAM used as cache is freed when there is no more "free" ram and more is needed.  The least recently used ram in the cache is reused first.  (In most cases, the movie/music you played least recently)

 

In my case, once I'm part way watching a DVD ISO image all my RAM is in use.  re-playing the previous chapter in the movie is typically done without needing to read the disk.  Obviously the entire movie does not fit in my RAM, so playing to the end and then going back to chapter 1 will access the disk once more.

 

So... using cache_dirs uses all the memory is needs to hold the directory listings.  That depends on how many files you have.  It works by listing the files on your server (keeping their directory listing most recently accessed and in the disk buffer cache)  The "memory" it uses is exactly the same as the buffer cache used when watching a movie.  It is available for re-use if it is the least-recently used.  (It however, constantly repeats its scan of the directories every few seconds, so usually, they are not the least recently used memory in the cache)

[methanoid]

Thanks Joe. If only other forums had members who gave such comprehensive and comprehendable answers. 

[neilt0]

As for how much RAM you need if you're running a bunch of add-ons, I run cache_dirs, sabnzbd, nzbget, SickBeard and a few other apps and my machine is currently using 844MB out of 5GB.

 

When I was running with 1GB, it wasn't enough RAM for all the add-ons I used, but 5GB is plenty.

 

[methanoid]

Hello Neil me old mucka. Now seriously you didnt need to post for me to appreciate it ;-)  And yes, I saw the Microserver post   But glad to see you remembered I'm a usenet guy!!

[neilt0]

Hello Neil me old mucka. Now seriously you didnt need to post for me to appreciate it ;-)  And yes, I saw the Microserver post   But glad to see you remembered I'm a usenet guy!!

 

Dude, stop following me around the internet! 

 

So, did you buy a MicroServer? I have a LinkStation LS-VL for sale if you're interested... I used it for about a week, I think.

[olympia]

Hi Joe L.,

 

not sure this has been already brought up by someone, but I think it hasn't.

 

When you stop cache_dirs with 'cache_dirs -q' it is only terminating the cache_dirs instance running in the background, but it is not killing any running (ongoing) 'find' command. So when you try to stop the array soon after you started up cache_dirs, the running 'find' command(s) can keep drive(s) busy even for longer time, resulting a long 'unmounting' on the unRAID interface unless you kill the 'find(s)' manually.

 

Could killing of 'find' command(s) be added to 'cache_dirs -q' so that it can shut down gracefully?

 

Thank you for your feedback in advance.

 

Hi Joe L.,

 

Was it that dumb question that you don't have any feedback on it or you missed this one?

 

Cheers!

[Joe L.]

Hi Joe L.,

 

not sure this has been already brought up by someone, but I think it hasn't.

 

When you stop cache_dirs with 'cache_dirs -q' it is only terminating the cache_dirs instance running in the background, but it is not killing any running (ongoing) 'find' command. So when you try to stop the array soon after you started up cache_dirs, the running 'find' command(s) can keep drive(s) busy even for longer time, resulting a long 'unmounting' on the unRAID interface unless you kill the 'find(s)' manually.

 

Could killing of 'find' command(s) be added to 'cache_dirs -q' so that it can shut down gracefully?

 

Thank you for your feedback in advance.

 

Hi Joe L.,

 

Was it that dumb question that you don't have any feedback on it or you missed this one?

 

Cheers!

Saw it, have not had time to investigate...

 

However... something is wrong in your use of the command.  Let me explain.

 

The very first "find" should take a while going through all your disks.  Subsequent "find" commands should just read from the buffer cache, since the blocks needed should not be the least recently used, and should not have been reused, and are all in RAM.    In that way, these subsequent "find" commands should be really quick... fractions of a second typically.

 

You can see what the cache_dirs program is doing by running it in the foreground by use of the "-F" option (possibly in combination with the "-v" option for even more information.)

 

On my server, with only 500 Meg of ram, I had to exclude my "data" directory and only cache my "Movies" and "Music" directories.  I simply use "-e data" when I invoke the command.

 

the reason I did not kill the "find" was that typically it was never there long enough to worry about.

 

Joe L.

[olympia]

Hi Joe L.,

 

thank you for your response.

I am also not doing any extraordinary hacking when I invoke the command, in fact I am also only doing a couple of excludes and nothing else.

 

However I have 4GB RAM and a diverse dir/file structure.

Cache_dirs is working perfectly, and the initial find stops after a while, and then there is no issue with shuting down the array. But this "after a while" for the initial scan takes about 5-15 minutes (never measured that) and at my friend who really has an insane structure it is more 15-25 minutes.

 

This means if I want to stop the array for whatever reason in this initial scan, then I have to telnet in and kill the find command(s) in order to do that. So based on this experience that this period can be even 25 minutes, I thought it would come more handy if cache_dirs -q was doing that kill in its own.

[Joe L.]

Hi Joe L.,

 

thank you for your response.

I am also not doing any extraordinary hacking when I invoke the command, in fact I am also only doing a couple of excludes and nothing else.

 

However I have 4GB RAM and a diverse dir/file structure.

Cache_dirs is working perfectly, and the initial find stops after a while, and then there is no issue with shuting down the array. But this "after a while" for the initial scan takes about 5-15 minutes (never measured that) and at my friend who really has an insane structure it is more 15-25 minutes.

 

This means if I want to stop the array for whatever reason in this initial scan, then I have to telnet in and kill the find command(s) in order to do that. So based on this experience that this period can be even 25 minutes, I thought it would come more handy if cache_dirs -q was doing that kill in its own.

OK, that explains the issue better.

 

I was not thinking you were trying to kill the initial "find", but subsequent ones.  In some cases, the initial find has to spin up drives.  I'm not sure it is killable while waiting for a drive to spin up.  (basically, the "find" process is in kernel space and not looking for user-signals at that time)

 

It would be easy if the initial "find" was a single command... It is not.  It is a "find" on each of the data drives in turn, for each of the user-shares in turn.  With 10 disks and 3 user-shares, it is 30 "find" commands.   

 

I'll give it some thought.  Might not be too difficult.

 

Joe L.

[Interstellar]

Should cache_dirs help when loading drives via AFP?

 

I only ask because via SMB my "Films" folder shows instantly but takes a few seconds for AFP to show anything.

 

Ideas?

 

Also, given I have 4Gb memory and don't really have anything running in the background apart from PS3 MS which uses a max of 600MB, can I get cache_dirs to use more of it?

 

Cheers!

[Joe L.]

Should cache_dirs help when loading drives via AFP?

 

I only ask because via SMB my "Films" folder shows instantly but takes a few seconds for AFP to show anything.

 

Ideas?

 

Also, given I have 4Gb memory and don't really have anything running in the background apart from PS3 MS which uses a max of 600MB, can I get cache_dirs to use more of it?

 

Cheers!

If AFP is reading anything other than the directory listing, no... since those disk blocks would not be in the recently used set of disk blocks in the cache.

 

[olympia]

I'll give it some thought.   Might not be too difficult.

 

Thank you for your time and efforts!

[HarryRosen]

I just installed this and I think it's working, but how do i check for sure?

[HarryRosen]

I figured it out and it's working but how to i get it to cache beyond a certain folder

 

my config is like this

 

/boot/cache_dirs -d 9 -m 3 -M 5 -p 200 -e ".custom" -e "mysql" -w

 

my share is like this

 

/TV Show/showname/season/file.mkv

/Movies/MovieName/file.mkv

 

and so on

 

but when i hit the tv show directory the drives start to spin up to show all the shows, how to I get it to cache all the way to the file.mkv

 

so if i click on the season folder the drives still don't spin up

[Johnm]

The -d 9 should tell it to go 9 folders in from root.

Do you have enough ram?

 

I would try a -i and do just the tv dir for testing.

Also it could be the space in the filename. Turn on the logging and look

 

Also at a console, restart it with -F -v I think watch in realtime what it does

[heffneil]

I figured it out and it's working but how to i get it to cache beyond a certain folder

 

my config is like this

 

/boot/cache_dirs -d 9 -m 3 -M 5 -p 200 -e ".custom" -e "mysql" -w

 

my share is like this

 

/TV Show/showname/season/file.mkv

/Movies/MovieName/file.mkv

 

and so on

 

but when i hit the tv show directory the drives start to spin up to show all the shows, how to I get it to cache all the way to the file.mkv

 

so if i click on the season folder the drives still don't spin up

 

Can you tell me how you figured it out?  I wanted to know the answer since I just installed and I needed to figure out if this is working properly.

 

Thanks!

 

Neil

 

[HarryRosen]

I have 8 gig of ram so i would think that would be enough. going to try and run this now

 

/boot/cache_dirs -d 9 -m 3 -M 5 -p 200 -i "TV\ Shows" -i "Movies" -i "Music" -i "Public" -i "Complete" -e ".custom" -e "mysql" -w -F -v

 

 

is there a way to see what dirs are in the cache?

[Joe L.]

I have 8 gig of ram so i would think that would be enough. going to try and run this now

 

/boot/cache_dirs -d 9 -m 3 -M 5 -p 200 -i "TV\ Shows" -i "Movies" -i "Music" -i "Public" -i "Complete" -e ".custom" -e "mysql" -w -F -v

 

 

is there a way to see what dirs are in the cache?

I'm not 100% sure, but the "TV Shows" option, because it is quoted, does not need the backslash. 

-i "TV Shows"

should do it.

 

Also, based on how the program works:

If no "-i" options are given, it will scan all the top level directories in /mnt/disk* and /mnt/cache and create an initial top-level list of directories to scan and cache  (those same top level directories represent your user-shares).

 

If "-i" entries are given, they will be the only top level directories to scan and cache.  Those will be the initial top-level list.

 

If "-e" (exclude) directories are given, they are then deleted from the initial top-level-list created by either the "-i" options, or the lack of "-i" options.

 

So, you probably do not need the -e options, since those are not in your list of "-i" entries.    They do not do any harm, but you've already excluded them by only including the other three "-i" options.

 

Joe L.