cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up

[arcane]

Would you mind sharing this private email with us?

At least the cache_dirs part

 

cirkator - Sure.  Joe suggested playing with the cache pressure and running it in the foreground:

 

"Best you can do is experiment with other cache_pressure values.  (the tunable used by the kernel to try to free buffers as needed) 

cache_dirs -p 100 -w is as good as any to try.  (and even it may not help)  If not specified, it defaults to 10."

 

"run cache_dirs -F -v in the foreground, to see where it is spending its time.  you might have a file-system that needs attention, or can be excluded."

 

Weebotech -

 

Thanks for the suggestion.  I'll remove the shares with the largest amount of files and see if that helps things any.

 

Joe L. -

 

Again, thanks for all your help and troubleshooting!

[sacretagent]

Joe

 

any way to include killing the different find threads when you do a cache-dirs -q ?

now when you stop cache dirs the find threads go on for sometimes like 15 minutes.. depending in which dir they are...

if we can kill these finds then we can stop the array quicker ?

or is there something i don't know that makes it dangerous to kill those finds ?

 

OH yeah using cache dirs

with these arguments via SF

Jun 17 19:27:09 P8H67 cache_dirs: version=1.6.7

Jun 17 19:27:09 P8H67 cache_dirs: command-args=-w -B -m 5 -M 20 -d 10 -e mysql -e plex -e headphones -e couchpotatov2 -e Crashplan -e lazylibrarian -e Thumbnails

[Joe L.]

Joe

 

any way to include killing the different find threads when you do a cache-dirs -q ?

now when you stop cache dirs the find threads go on for sometimes like 15 minutes.. depending in which dir they are...

if we can kill these finds then we can stop the array quicker ?

or is there something i don't know that makes it dangerous to kill those finds ?

 

OH yeah using cache dirs

with these arguments via SF

Jun 17 19:27:09 P8H67 cache_dirs: version=1.6.7

Jun 17 19:27:09 P8H67 cache_dirs: command-args=-w -B -m 5 -M 20 -d 10 -e mysql -e plex -e headphones -e couchpotatov2 -e Crashplan -e lazylibrarian -e Thumbnails

I do not support cache_dirs insalled  via SimpleFeatures.  It is not the program I have attached to the cache_dirs thread.  It is a copy modified for the plugin.  (yes, it is confusing, since they did not change the version number)

 

Try running it in the foreground with verbose mode enabled to see where it is spending its time.  Personally, I would NEVER use the -m or -M options or ask it to go 10 directories deep.

 

[Ice_Black]

Would you consider implementing cache xml and jpg files? For movies use.

 

[Joe L.]

Would you consider implementing cache xml and jpg files? For movies use.

It is more than just those two files people are interested in... (I use XBMC, and it also has .nfo, .tbn)

 

To keep them in memory requires reading them and sending the contents to /dev/null

 

You can add a lines in the main loop

find "$i"/"$share_dir"  -type f -name "*.jpg" -exec cat {} > /dev/null \;

find "$i"/"$share_dir" -type f -name "*.xml" -exec cat {} > /dev/null \;

[Ice_Black]

Would you consider implementing cache xml and jpg files? For movies use.

It is more than just those two files people are interested in... (I use XBMC, and it also has .nfo, .tbn)

 

To keep them in memory requires reading them and sending the contents to /dev/null

 

You can add a lines in the main loop

find "$i"/"$share_dir"  -type f -name "*.jpg" -exec cat {} > /dev/null \;

find "$i"/"$share_dir" -type f -name "*.xml" -exec cat {} > /dev/null \;

 

Oh cool.

 

Why send the contents to /dev/null \;?

[Joe L.]

Would you consider implementing cache xml and jpg files? For movies use.

It is more than just those two files people are interested in... (I use XBMC, and it also has .nfo, .tbn)

 

To keep them in memory requires reading them and sending the contents to /dev/null

 

You can add a lines in the main loop

find "$i"/"$share_dir"  -type f -name "*.jpg" -exec cat {} > /dev/null \;

find "$i"/"$share_dir" -type f -name "*.xml" -exec cat {} > /dev/null \;

 

Oh cool.

 

Why send the contents to /dev/null \;?

The contents must be read to have them in cache.  (it is not enough to just have the file names in cache)  You must do something with the blocks read, so /dev/null is best. 

 

Now, this will take a lot more RAM if you have a lot of large .jpg files.  Each file occupies (potentially) 100k or more. It you have 1000 thumbnails, then 100k * 1000  = 1Gig of additional RAM needed for cache.

[WeeboTech]

Now, this will take a lot more RAM if you have a lot of large .jpg files.  Each file occupies (potentially) 100k or more. It you have 1000 thumbnails, then 100k * 1000  = 1Gig of additional RAM needed for cache.

 

Plus there is no guarantee they will stay in ram and/or be flushed.

If there are that many small files that you want to be read at high speed, using an SSD would be better.

other choice is to put allot of ram in the system and/or use a tmpfs.

 

SSD is probably better.  While investigating today, I found a some cool products that can use mSATA in a PCIe slot. (if you don't have enough room for a 2.5" ssd).

[Guzzi]

Hi JoeL., is there any command parameter/setting, that would make sure, writing to the array does not clear/loose the cachedirs buffers?

Currently, after I copied something to the array, cachedirs has to reread a lot from disk again.

And one more question: Is there a way to see, how much memory cachedirs (ot "the  populates cache buffers") occupy - to see, if there is not enough memory?

 

[Joe L.]

Hi JoeL., is there any command parameter/setting, that would make sure, writing to the array does not clear/loose the cachedirs buffers?

Currently, after I copied something to the array, cachedirs has to reread a lot from disk again.

And one more question: Is there a way to see, how much memory cachedirs (ot "the  populates cache buffers") occupy - to see, if there is not enough memory?

no,  that is expected, the most recently used disk buffers re-use the least recently used ones, and no.

Other than to look at the output of

free -l

there are no commands.  Nothing can pin the directory entries in memory.

 

Joe L.

[Guzzi]

Thanks JoeL. for the quick answer - so back to "try and error" ;-) - I think, I have too many files there in some areas, that just flood the buffers and flush old ones (I noticed a LOT of reads on stat page after some time with cachedirs active recently - and only a few without, so I think that is a clear indicator...).

Is there a rough formula how much (buffer) memory is used per directory and per file/dir entry - that would allow to check, if there is a general "sizing" problem?

[WeeboTech]

It's the dentry cache. On my other linux system here's a snipped.

 

rcotrone@rgclws:~ $ grep -i dentry /var/log/dmesg

Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)

sizeof(dentry)=192 bytes

 

It's probably different for unRAID.

 

FWIW, I tried expanding it once, it sorta worked, then I received an OOM condition and the machine started crashing. It uses low memory and low memory was exhausted.

 

maybe this will change when unRAID goes 64bit.

 

[Guzzi]

good hint ... google gave some more info, e.g.

available memory (not only dentry)

grep -i slab /proc/meminfo

 

memory details:

awk '/dentry|inode/ { print $1,$2,$3,$4}' /proc/slabinfo

 

live memor usage

slabtop -s c

 

JoeL, Weebotech - would it help to check dentry usage there, while cachedirs fills buffers/dentries? (I hardly understand what all this means and where/what to check .....)

[Guzzi]

It uses low memory and low memory was exhausted.

 

maybe this will change when unRAID goes 64bit.

So is there a known limitation in terms of "files/directories" that can be cached with current 32bit architecture?

[WeeboTech]

It doesn't help to check there as I thought it would. I know what has happened in the past.

I booted with a parameter to double dentry and my machine still spun up the drives (because I have so many files).

Later on the machine had failures and crashed due to being out of low memory.

I then tripled the memory, being a glutton for punishment, and the machine crashed soon after being out of low memory.

 

Granted my bittorrent drive had tons of files, but so did my ftp pub folder, my mp3 folder and my backup folder.

There was no way to get around the sheer magnitude of files I was dealing with.

 

There was a kernel boot parameter you can use. I forget what it was, that could resize the dentry cache.

You still have to adjust a kernel parameter to prevent it from shrinking the dentry cache upon cache pressure.

 

This may be different with the later releases of the kernel. I don't know, I cannot tell at the current time.

 

[WeeboTech]

It uses low memory and low memory was exhausted.

 

maybe this will change when unRAID goes 64bit.

So is there a known limitation in terms of "files/directories" that can be cached with current 32bit architecture?

 

There is, and when I expanded it, bad things started to happen.

I have no idea if 64 bit will help this.

[Guzzi]

There is [...]

Yeah, I understood that - I was asking for a real number (maybe during your experiments) with stock settings - as I could then do a count (assuming the size per entry is known) on the data and get an idea how much is possilbe at the maximum - and then exclude/include only some areas for cachedirs script ...

[sacretagent]

HI joe

 

you said me to look what is keeping my finds busy

by running verbose and in the foreground

 

From commandline

root@p5bplus:~# cache_dirs -w -F -v -B -M 20 -d 5 -e .custom -e plex -e NZB -e .dropbox -e BIN -e Crashplan -e Thai_Torrents_complete -p 200

 

the -p 200 you told me one time to put that in a LOOOONG time ago... not sure if it is still needed...

 

it returns these after going through the dirs

Executed find in 1198.110905 seconds, weighted avg=1198.110905 seconds, now sleeping 10 seconds

Executed find in 1232.366907 seconds, weighted avg=1220.948239 seconds, now sleeping 9 seconds

Executed find in 1305.310749 seconds, weighted avg=1263.129494 seconds, now sleeping 8 seconds

Executed find in 1163.732992 seconds, weighted avg=1223.370893 seconds, now sleeping 9 seconds

 

no clue what this means though

between last 2 lines there was a mover action going on  just fyi

 

[Joe L.]

HI joe

 

you said me to look what is keeping my finds busy

by running verbose and in the foreground

 

From commandline

root@p5bplus:~# cache_dirs -w -F -v -B -M 20 -d 5 -e .custom -e plex -e NZB -e .dropbox -e BIN -e Crashplan -e Thai_Torrents_complete -p 200

 

the -p 200 you told me one time to put that in a LOOOONG time ago... not sure if it is still needed...

 

it returns these after going through the dirs

Executed find in 1198.110905 seconds, weighted avg=1198.110905 seconds, now sleeping 10 seconds

Executed find in 1232.366907 seconds, weighted avg=1220.948239 seconds, now sleeping 9 seconds

Executed find in 1305.310749 seconds, weighted avg=1263.129494 seconds, now sleeping 8 seconds

Executed find in 1163.732992 seconds, weighted avg=1223.370893 seconds, now sleeping 9 seconds

 

no clue what this means though

between last 2 lines there was a mover action going on  just fyi

It indicates any given "find" command is taking over 1000 seconds.  (that is pretty bad )  It must be going to the physical disks to get data not in disk buffer cache in memory.    That is over 16 minutes, so if you've set your spin-down timer shorter, it might have to spin up the disks every time.  I'd drop all the extra options other than the "-e" options to start. 

 

On my system, it looks more like this, with the "find" operating in RAM and returning in a fraction of a second.

cache_dirs -F  -e data

Executed find in 0.294915 seconds, weighted avg=0.294915 seconds, now sleeping 5 seconds

Executed find in 0.295294 seconds, weighted avg=0.295168 seconds, now sleeping 4 seconds

Executed find in 0.295314 seconds, weighted avg=0.295241 seconds, now sleeping 4 seconds

Executed find in 0.335031 seconds, weighted avg=0.311157 seconds, now sleeping 3 seconds

Executed find in 0.296573 seconds, weighted avg=0.306296 seconds, now sleeping 4 seconds

Executed find in 0.297404 seconds, weighted avg=0.303755 seconds, now sleeping 5 seconds

Executed find in 0.296930 seconds, weighted avg=0.302049 seconds, now sleeping 6 seconds

Executed find in 0.296885 seconds, weighted avg=0.300901 seconds, now sleeping 7 seconds

Executed find in 0.370029 seconds, weighted avg=0.314727 seconds, now sleeping 6 seconds

 

[WeeboTech]

Is there anyway to put the count of files read in the dir_cache program?

maybe that could provide a benchmark of when it exceeds it's usefulness.

 

I know I couldn't run it on my server. I had over 250,000 files on some drives.

[sacretagent]

Hi Joe

 

I found the culprit

 

i put a -e for the sabnzbd and utorrent and sickbeard program folder and now we are seeing seconds returns

 

no real need to keep those in memory i guess ... especially the utorrent one

gonna see now a bit if the sickbeard page with the banners is slow or not

seems to be OK ...

 

anyway THX for the help

disks are spinning down much more now

Executed find in 1.824769 seconds, weighted avg=2.731875 seconds, now sleeping 10 seconds

Executed find in 1.821371 seconds, weighted avg=1.764779 seconds, now sleeping 9 seconds

Executed find in 1.695226 seconds, weighted avg=1.749911 seconds, now sleeping 10 seconds

[garycase]

Joe => I have no idea how "doable" this is ... but is there a way to have CacheDirs "suspend itself" if a parity check is in process?    Not shut itself down (which would then require re-starting it) ... but just basically ask "Is parity check in process?" before it kicks off the threads to read the directories.

 

Logically it seems like that'd be fairly simple, but I'm just not a Linux guy, so I don't really know.

 

[Guzzi]

It indicates any given "find" command is taking over 1000 seconds.  (that is pretty bad )  It must be going to the physical disks to get data not in disk buffer cache in memory.    That is over 16 minutes, so if you've set your spin-down timer shorter, it might have to spin up the disks every time.  I'd drop all the extra options other than the "-e" options to start. 

Hi JoeL.,

what do you think about an output to the log file, if cachedirs takes longer than a certain amout of time repeatedly (let's say more than twice - this filters the first time reading and any rereading when the array was accessed and dcahes flushed) ?

I think that would help a lot to identify the case, when it is not working properly and "overloaded" - instead of trying to observe that by checking spindown behaviour. If it is not taking too much cpu cycles, such an output could include the number of files read by find - so user get's an info with what number of files that case happens - what do you think? Doable?

[Joe L.]

It indicates any given "find" command is taking over 1000 seconds.  (that is pretty bad )  It must be going to the physical disks to get data not in disk buffer cache in memory.    That is over 16 minutes, so if you've set your spin-down timer shorter, it might have to spin up the disks every time.  I'd drop all the extra options other than the "-e" options to start. 

Hi JoeL.,

what do you think about an output to the log file, if cachedirs takes longer than a certain amout of time repeatedly (let's say more than twice - this filters the first time reading and any rereading when the array was accessed and dcahes flushed) ?

I think that would help a lot to identify the case, when it is not working properly and "overloaded" - instead of trying to observe that by checking spindown behaviour. If it is not taking too much cpu cycles, such an output could include the number of files read by find - so user get's an info with what number of files that case happens - what do you think? Doable?

some interesting ideas...  I'll give them some thought.  More diagnostic output is certainly better when trying to understand what is happening.

Might eliminate a post or two saying "why is my array spinning up"   I'm running cache_dirs.

[garycase]

some interesting ideas...  I'll give them some thought.

 

Does this comment also apply to my note r.e. suspending Cache_Dirs during parity checks? 

 

It sure seems that would be a VERY useful/nice feature.    Seems like a simple "If parity check in progress, don't start the next Find" check in Cache_Dirs  would basically suspend it => at least after the current Find completed.    No reason to shut down Cache_Dirs ... it would simply keep checking at the current intervals, but as long as the parity check was in progress, wouldn't initiate any more Finds ... thus not interfering with the check.

 

... and of course once the parity check was over, the next time Cache_Dirs checked all the disks would still be spinning, so it'd be very quickly up-to-date.