Jump to content
Joe L.

cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up

1086 posts in this topic Last Reply

Recommended Posts

Is it possible to use the 'exclude_dir' setting to exclude an entire drive? I have a non-array drive on the system that'd i'd rather not cache.

 

Thanks, Joe, for another useful tool!

cache_dirs only operates on /mnt/disk1 through /mnt/disk19 and /mnt/cache.  It does not need to exclude a disk not in the array since it does not cache them.

Share this post


Link to post

You know... you are the first to describe this behavior since I posted the last version of cache_dirs. 

 

Guess what.  In adding the feature to suspend the script during the "mover" scripts operation, I broke cache_dirs.

 

It is supposed to spit_back info as it ran.  It is broken if you do not have a cache drive defined, as you discovered, a major part of the loop is skipped and no detail is printed.

 

Since last October, over 200 people have downloaded it, and you are the first to notice the problem. (indirectly, at least)

Lol, well thanks for "fixing" it so fast. Either no one tests it before using it or they all have cache drives in their systems. Thanks again for this script! ;)

Share this post


Link to post

I had a problem with cache_dirs.  I was seeing some errors about duplicates show up on the console.  So I tried to shutdown but it locked up and I had to to a power off to get it to shutdown.  I tracked it down to having duplicate files in some shares (ok, my bad).  When I commented out cache_dirs in the go script the problem stopped. 

 

I'm guessing the duplicates which are mostly big iso files are there because I was doing some copy speed testing to the /mnt/drive#/... and inadvertantly created the issue myself.  But why the system wouldn't shut down I don't know.  Maybe too much memory was used up in a loop or something or who knows.  Something to look into Joe. 

 

#/boot/cache_dirs -w -u -d 3 -e "lost+found" -e "*iTunes*" -e "ftproot"

Share this post


Link to post

I had a problem with cache_dirs.  I was seeing some errors about duplicates show up on the console.  So I tried to shutdown but it locked up and I had to to a power off to get it to shutdown.  I tracked it down to having duplicate files in some shares (ok, my bad).  When I commented out cache_dirs in the go script the problem stopped. 

 

I'm guessing the duplicates which are mostly big iso files are there because I was doing some copy speed testing to the /mnt/drive#/... and inadvertantly created the issue myself.  But why the system wouldn't shut down I don't know.  Maybe too much memory was used up in a loop or something or who knows.  Something to look into Joe. 

 

#/boot/cache_dirs -w -u -d 3 -e "lost+found" -e "*iTunes*" -e "ftproot"

Cache dirs does nothing different than using your windows-explorer to list the files in the directories. 

Duplicate files create log file entries when directories containing them are browsed, no matter how they are browsed.

Filling up the syslog (using up all free ram) will eventually crash a system.

There is nothing cache_dirs can do to fix the cause of running out of memory. 

 

Perhaps I can think of a way to disable it when free memory gets low.  (But by then, you have other problems anyway)

Share this post


Link to post

Ok, I think I see what happened.  Cache_dirs browses the folders that have dups and they fill up the syslog.  So if I manually browse them it would do the same - right?  Are you saying that cache_dirs isn't aware that the duplicates exist and they messages are coming from unRAID or linux?

Share this post


Link to post

Ok, I think I see what happened.  Cache_dirs browses the folders that have dups and they fill up the syslog.  So if I manually browse them it would do the same - right?  Are you saying that cache_dirs isn't aware that the duplicates exist and they messages are coming from unRAID or linux?

Exactly.  They are coming from the user-file-system.  It is complaining since it can only let you access one of the duplicated files.  cache_dirs has no idea of the dupes existence.

Share this post


Link to post

Lets say I've got four user shares:  Me, You, Them, Everyone.

Under Me are folders Images, Video, Music

 

How can I get cache_dirs to cache everything in Me except the images folder (and it's subfolders)?

Share this post


Link to post

Cache Dirs doesn't seem to be working correctly for me anymore? I can't even access hard drives with 35 files on them without unraid spinning the drive up. It seems random, sometimes it spins them up and othertimes it doesn't. After a few hours cache_dir seems to have completely loss it's cache and every drive must be spun up when I access them.

 

I recently upgraded to 1.6.5, and it seems to have made it worse.. any ideas?

 

Heres my go file:

#!/bin/bash

# Start the Management Utility

/usr/local/sbin/emhttp &

/boot/unmenu/uu

/boot/cache_dirs -w

echo nameserver 192.168.0.1 >/etc/resolv.conf

echo 192.168.0.200 Server >>/etc/hosts

sleep 30; blockdev --setra 2048 /dev/md*

cd /boot/packages && find . -name '*.auto_install' -type f -print | sort | xargs -n1 sh -c

 

Share this post


Link to post

Cache Dirs doesn't seem to be working correctly for me anymore? I can't even access hard drives with 35 files on them without unraid spinning the drive up. It seems random, sometimes it spins them up and othertimes it doesn't. After a few hours cache_dir seems to have completely loss it's cache and every drive must be spun up when I access them.

 

I recently upgraded to 1.6.5, and it seems to have made it worse.. any ideas?

 

Heres my go file:

#!/bin/bash

# Start the Management Utility

/usr/local/sbin/emhttp &

/boot/unmenu/uu

/boot/cache_dirs -w

echo nameserver 192.168.0.1 >/etc/resolv.conf

echo 192.168.0.200 Server >>/etc/hosts

sleep 30; blockdev --setra 2048 /dev/md*

cd /boot/packages && find . -name '*.auto_install' -type f -print | sort | xargs -n1 sh -c

 

All that cache_dirs attempts to do is keep the directory entries in the buffer cache by accessing them periodically.

 

If you use your buffer cache by running other processes, or watching a movie, then it is easily possible for the least recently accessed entries to be displaced.

 

have you tried running cache_dirs in the foreground to see where it is spending its time?

 

Try the -F and -v options together like this

cache_dirs -F -v

 

You may have a larger hierarchy than it can handle, or you might be experiencing the side effects of spinup groups.

 

Share this post


Link to post

I'm apparently a noob or something because I can't get this to exclude a directory.

 

/boot/cache_dirs -w -e "Steam" is in my go file. It still caches my Steam folder.

 

Location of the folder:

\\SERVER\Other Files\Steam

 

Please help, this folder has around 350,000 files in it and takes around 30-40 minutes to cache.

Share this post


Link to post

I'm not sure if a nested/absolute path is needed or if the double-quotes are messing it up in your GO script. The double-quotes may need to be escaped or removed completely.

Share this post


Link to post

I'm not sure if a nested/absolute path is needed or if the double-quotes are messing it up in your GO script.

The double-quotes may need to be escaped or removed completely.

but then...

To exclude a "data" and "old-stuff" directory, you would use

cache_dirs -w -e "data" -e "old-stuff"

Always use Quote marks around the folders you wish to exclude

Share this post


Link to post

I'm not sure if a nested/absolute path is needed or if the double-quotes are messing it up in your GO script.

The double-quotes may need to be escaped or removed completely.

but then...

To exclude a "data" and "old-stuff" directory, you would use

cache_dirs -w -e "data" -e "old-stuff"

Always use Quote marks around the folders you wish to exclude

 

Yup, I did exactly how he shows it on the main post. My friend who has unraid also cannot get the exclude commands to work. I completely removed the mod for now because it's taking 30-40 minutes to cache everytime.

Share this post


Link to post

I'm apparently a noob or something because I can't get this to exclude a directory.

 

/boot/cache_dirs -w -e "Steam" is in my go file. It still caches my Steam folder.

 

Location of the folder:

\\SERVER\Other Files\Steam

 

Please help, this folder has around 350,000 files in it and takes around 30-40 minutes to cache.

Only "Top level" directories can be excluded using the -e option.  I never wrote it to be able to exclude directories further down the hierarchy.

 

Use

-e "Other Files"

 

or you can edit the program at around line 372 is this:

dir_list=`build_dir_list`

change it to

dir_list=`build_dir_list | grep -v "Steam"`

and then you don't need to use any -e options, unless there is an additional different top level directory you wish to exclude.

Make sure you use an editor that is linux friendly and does not add carriage returns to the ends of lines. 

(edit the file using "mc" and you'll be fine)

 

Share this post


Link to post

I'm apparently a noob or something because I can't get this to exclude a directory.

 

/boot/cache_dirs -w -e "Steam" is in my go file. It still caches my Steam folder.

 

Location of the folder:

\\SERVER\Other Files\Steam

 

Please help, this folder has around 350,000 files in it and takes around 30-40 minutes to cache.

Only "Top level" directories can be excluded using the -e option.  I never wrote it to be able to exclude directories further down the hierarchy.

 

Use

-e "Other Files"

 

or you can edit the program at around line 372 is this:

dir_list=`build_dir_list`

change it to

dir_list=`build_dir_list | grep -v "Steam"`

and then you don't need to use any -e options, unless there is an additional different top level directory you wish to exclude.

Make sure you use an editor that is linux friendly and does not add carriage returns to the ends of lines. 

(edit the file using "mc" and you'll be fine)

 

 

Thanks Joe.

Share this post


Link to post

I'm apparently a noob or something because I can't get this to exclude a directory.

 

/boot/cache_dirs -w -e "Steam" is in my go file. It still caches my Steam folder.

 

Location of the folder:

\\SERVER\Other Files\Steam

 

Please help, this folder has around 350,000 files in it and takes around 30-40 minutes to cache.

Only "Top level" directories can be excluded using the -e option.  I never wrote it to be able to exclude directories further down the hierarchy.

 

Use

-e "Other Files"

 

or you can edit the program at around line 372 is this:

dir_list=`build_dir_list`

change it to

dir_list=`build_dir_list | grep -v "Steam"`

and then you don't need to use any -e options, unless there is an additional different top level directory you wish to exclude.

Make sure you use an editor that is linux friendly and does not add carriage returns to the ends of lines. 

(edit the file using "mc" and you'll be fine)

 

 

Ahh, that answered something for me as well.  I have multiple backup folders and iTunes folders and thought  -e iTunes would skip them all.  But what I see is "No iTunes directory found" or something like that anyway because it's not at the top level.

Share this post


Link to post

I found a minor bug with the exclude directory command.

 

If I have this in my 'go' file:

/boot/cache_dirs -w -e "Other Files"

 

Syslog shows:

Apr 18 01:45:00 Server cache_dirs: ---------- caching directories ---------------

Apr 18 01:45:00 Server cache_dirs: Movies and TV Shows

Apr 18 01:45:00 Server cache_dirs: ----------------------------------------------

Apr 18 01:45:00 Server cache_dirs: ERROR: excluded directory "Other Files" does not exist.

 

----

 

If I have this in my 'go' file:

/boot/cache_dirs -w

 

Syslog shows:

Apr 18 01:45:00 Server cache_dirs: ---------- caching directories ---------------

Apr 18 01:45:00 Server cache_dirs: Movies and TV Shows

Apr 18 01:45:00 Server cache_dirs: Other Files

Apr 18 01:45:00 Server cache_dirs: ----------------------------------------------

 

 

It's removing "Other Files" from the cache directory but I still get a directory error. Just a minor thing, as it seems to be working correctly.

Share this post


Link to post

I found a minor bug with the exclude directory command.

 

If I have this in my 'go' file:

/boot/cache_dirs -w -e "Other Files"

 

Syslog shows:

Apr 18 01:45:00 Server cache_dirs: ---------- caching directories ---------------

Apr 18 01:45:00 Server cache_dirs: Movies and TV Shows

Apr 18 01:45:00 Server cache_dirs: ----------------------------------------------

Apr 18 01:45:00 Server cache_dirs: ERROR: excluded directory "Other Files" does not exist.

 

----

 

If I have this in my 'go' file:

/boot/cache_dirs -w

 

Syslog shows:

Apr 18 01:45:00 Server cache_dirs: ---------- caching directories ---------------

Apr 18 01:45:00 Server cache_dirs: Movies and TV Shows

Apr 18 01:45:00 Server cache_dirs: Other Files

Apr 18 01:45:00 Server cache_dirs: ----------------------------------------------

 

 

It's removing "Other Files" from the cache directory but I still get a directory error. Just a minor thing, as it seems to be working correctly.

Thanks... I'll see if I can figure out what is the root cause.

Share this post


Link to post

Hi,

 

i try to use your cache dir. But something wents wrong :(

My dir structure ist one Share 5 Datadisk

under nmt/user is the server share

unter server are some dirs calles movie, mp3,serie

under movie is the structure so

movie

- 0-9

- A-C

- D - .... and so on

 

in the 0-9 dir are the movie files

 

i star cache_dir -w

 

but my disk spin up wenn i go with my windows explorer on the dirs. i´m using unraid 4.5.3

 

could you give me any help ? or does cache_dir realy only cache dir names and note the filenames inthere ?

 

Greets

Eisi

 

Share this post


Link to post

does cache_dir realy only cache dir names and note the filenames inthere ?

Only the directory and file names. 

And, only if you have enough memory to hold the entire list of directory and file names

And, only if they are not displaced by something else needing memory.

 

To see what it is doing, kill the cache_dirs process you have running by typing

cache_dirs -q

and instead invoke it in the foreground with

cache_dirs  -F -v 

 

Type "Control-C" to exit it when run in the foreground.

Share this post


Link to post

Hi,

 

after i make cache_dirs -F -v i become these message

 

 

root@Server:/boot/custom# cache_dirs -F -v

Executing find /mnt/disk1/UMC -noleaf

Executing find /mnt/disk1/server -noleaf

Executing find /mnt/disk2/server -noleaf

Executing find /mnt/disk3/server -noleaf

Executing find /mnt/disk4/server -noleaf

Executed find in 0.107251 seconds, weighted avg=0.107251 seconds, now sleeping 5 seconds

Executing find /mnt/disk1/UMC -noleaf

Executing find /mnt/disk1/server -noleaf

Executing find /mnt/disk2/server -noleaf

Executing find /mnt/disk3/server -noleaf

Executing find /mnt/disk4/server -noleaf

Executed find in 0.089995 seconds, weighted avg=0.095747 seconds, now sleeping 6 seconds

Executing find /mnt/disk1/UMC -noleaf

Executing find /mnt/disk1/server -noleaf

Executing find /mnt/disk2/server -noleaf

Executing find /mnt/disk3/server -noleaf

Executing find /mnt/disk4/server -noleaf

 

and so on

 

Greets

Eisi

 

 

 

Share this post


Link to post

Hi,

 

after i make cache_dirs -F -v i become these message

 

 

root@Server:/boot/custom# cache_dirs -F -v

Executing find /mnt/disk1/UMC -noleaf

Executing find /mnt/disk1/server -noleaf

Executing find /mnt/disk2/server -noleaf

Executing find /mnt/disk3/server -noleaf

Executing find /mnt/disk4/server -noleaf

Executed find in 0.107251 seconds, weighted avg=0.107251 seconds, now sleeping 5 seconds

Executing find /mnt/disk1/UMC -noleaf

Executing find /mnt/disk1/server -noleaf

Executing find /mnt/disk2/server -noleaf

Executing find /mnt/disk3/server -noleaf

Executing find /mnt/disk4/server -noleaf

Executed find in 0.089995 seconds, weighted avg=0.095747 seconds, now sleeping 6 seconds

Executing find /mnt/disk1/UMC -noleaf

Executing find /mnt/disk1/server -noleaf

Executing find /mnt/disk2/server -noleaf

Executing find /mnt/disk3/server -noleaf

Executing find /mnt/disk4/server -noleaf

 

and so on

 

Greets

Eisi

 

 

 

looks like it is doing exactly what it is supposed to be doing.

the "find" command on each of the directories will read into the disk buffer cache the names of the files.

 

Since each "find" is occurring very quickly, I'd guess all the requests are being handles from RAM and not going to the disks at all.

 

In any case, now you know exactly what cache_dirs is doing.

 

You can type

find /mnt/disk1/server -noleaf

etc,

to see the files and folders being read into the disks cache.  cache_dirs is throwing away the listings, sending the listing to /dev/null

 

Now, if your "browsing" of the directory is writing to them (MAC OS writes ._DS files) or opening files to show thumbnail views (windows will do this) then yes, the disks will spin up, since the file contents are not cached.

 

Joe L.

 

Share this post


Link to post

I'm apparently a noob or something because I can't get this to exclude a directory.

 

/boot/cache_dirs -w -e "Steam" is in my go file. It still caches my Steam folder.

 

Location of the folder:

\\SERVER\Other Files\Steam

 

Please help, this folder has around 350,000 files in it and takes around 30-40 minutes to cache.

Only "Top level" directories can be excluded using the -e option.  I never wrote it to be able to exclude directories further down the hierarchy.

 

Use

-e "Other Files"

 

or you can edit the program at around line 372 is this:

dir_list=`build_dir_list`

change it to

dir_list=`build_dir_list | grep -v "Steam"`

and then you don't need to use any -e options, unless there is an additional different top level directory you wish to exclude.

Make sure you use an editor that is linux friendly and does not add carriage returns to the ends of lines. 

(edit the file using "mc" and you'll be fine)

 

 

Doing this still only excludes top level directories - right?  I'm looking for a way to exclude non-top level directories.  I tried hacking the script a little but didn't get too far.  What's actually keeping the directories in memory?  Is it just the find command causes them to be cached when it runs?

Share this post


Link to post

  What's actually keeping the directories in memory?  Is it just the find command causes them to be cached when it runs?

Exactly, it is just that the "find" command accesses all the directories in turn, keeping them from being the least recently used cache blocks available for re-use.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.