cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up


Recommended Posts

Its purpose is to keep directory entries in RAM so drives don't have to keep spinning up.

Aha, I understand! Many thanks for this sentance.

"Dynamix Cache Directories" only keeps directory structure of users shares in directory entries on cache drive. Copy structure of users share.

It keeps the directory entries in RAM as RobJ said, not on cache drive.

Can I set this opt to "Prefer" or "Only" to prevention move the share-directory to users share?

Only will only write to cache and will not move it from cache.

 

If you have a cache drive, Prefer will only write to cache, but if any of the share is already on other drives it will move to cache. This allows you to add a cache later and then get the share to use it as if it were cache-only.

 

Turn on Help in the upper right of the webUI.

Link to comment

It keeps the directory entries in RAM as RobJ said, not on cache drive.

 

If you have a cache drive, Prefer will only write to cache, but if any of the share is already on other drives it will move to cache. This allows you to add a cache later and then get the share to use it as if it were cache-only.

 

Ok. Many thanks, trurl!  ;)

Link to comment
  • 1 month later...

Hi all,

 

What does the Minimum interval between folder scans (sec) and Maximum interval between folder scans (sec) do?

 

Does cache_dir physically scan based on these parameters?

 

So will it spin up the drives every 10 seconds (default maximum interval) to search directories for new entries?

 

Thanks.

Link to comment

So will it spin up the drives every 10 seconds (default maximum interval) to search directories for new entries?

 

Thanks.

The reason for the seemingly quick rescan time is so that it won't have to spin up the drives.  Cache-dirs attempts to keep the directory structures in memory by rescanning all the time.  (Linux if the directory structure isn't used in a while will automatically drop it from memory) 

 

However, there are times where Cache Dirs will be unable to keep the entries in memory, and under those circumstances a spin up will happen (but it shouldn't happen again until those circumstances repeat themselves)

Link to comment

So will it spin up the drives every 10 seconds (default maximum interval) to search directories for new entries?

 

Thanks.

The reason for the seemingly quick rescan time is so that it won't have to spin up the drives.  Cache-dirs attempts to keep the directory structures in memory by rescanning all the time.  (Linux if the directory structure isn't used in a while will automatically drop it from memory) 

 

Ah - so it's primarily a memory scan to keep the data in RAM and prevent a flush. If the data is flushed, cache_dir will then do a physical scan and reload the data.

 

Thanks for the clarification, I was wondering how it worked.

Link to comment
  • 1 month later...

Hi Zin105,

 

You can limit the cache_dirs to only scan some shares, and you can also limit it to only scan to a certain depth. I have attached a program you can use to count where the bulk of your files are, it runs the following command at various depth

find $f -type f -maxdepth ${DEPTH} | wc -l

 

You can also you my modified cache-dirs which I link at this post: 

 

I modified it to stop scanning disks forever, if it cannot have files in cache, it now adaptively adjust its scan-depth according to how much it can have in memory, which is judged by the scan duration. When my server is pressed with other stuff, cache_dirs would lose its cache, and start scanning, causing my server to be even more pressed, which seemed silly. My adjustments makes cache_dirs scan only to a low depth in such a pressed situation and then stop after a couple of minutes attempt. You are welcome to try it out, but don't expect support, because I'm to busy elsewhere... :) But good luck with whatever you choose.

cache_dirs_file_count

Link to comment

I recently removed a number of unused shares.  When these cached shares are removed from the system, it appears that CacheDirs still tries to start them up.  Where can I correct this error?  Here's a log snippet of CacheDirs.  

 

Mar  1 06:37:39 HunterNAS cache_dirs: ==============================================
Mar  1 06:37:39 HunterNAS cache_dirs: Starting cache_dirs:
Mar  1 06:37:39 HunterNAS cache_dirs: Arguments=-u -i Backup -i Dropbox -i Goodsync -i ISO\ Files -i Pydio -i appdata -i backups -i cachebackup -i documents -i flashbkup -i home\ movies -i jwhbackup -i movies -i music -i ndh -i pictures -i tv
Mar  1 06:37:39 HunterNAS cache_dirs: Cache Pressure=10
Mar  1 06:37:39 HunterNAS cache_dirs: Max Scan Secs=10, Min Scan Secs=1
Mar  1 06:37:39 HunterNAS cache_dirs: Scan Type=adaptive
Mar  1 06:37:39 HunterNAS cache_dirs: Max Scan Depth=none
Mar  1 06:37:39 HunterNAS cache_dirs: Use Command='find -noleaf'
Mar  1 06:37:39 HunterNAS cache_dirs: Version=2.1.1
Mar  1 06:37:39 HunterNAS cache_dirs: ---------- Caching Directories ---------------
Mar  1 06:37:39 HunterNAS cache_dirs: D..x
Mar  1 06:37:39 HunterNAS cache_dirs: I..s
Mar  1 06:37:39 HunterNAS cache_dirs: P..o
Mar  1 06:37:39 HunterNAS cache_dirs: a..a
Mar  1 06:37:39 HunterNAS cache_dirs: c..p
Mar  1 06:37:39 HunterNAS cache_dirs: h..s
Mar  1 06:37:39 HunterNAS cache_dirs: j..p
Mar  1 06:37:39 HunterNAS cache_dirs: m..s
Mar  1 06:37:39 HunterNAS cache_dirs: m..c
Mar  1 06:37:39 HunterNAS cache_dirs: p..s
Mar  1 06:37:39 HunterNAS cache_dirs: t..v
Mar  1 06:37:39 HunterNAS cache_dirs: ----------------------------------------------
Mar  1 06:37:39 HunterNAS cache_dirs: ERROR: included directory "Backup" does not exist.
Mar  1 06:37:39 HunterNAS cache_dirs: ERROR: included directory "Goodsync" does not exist.
Mar  1 06:37:39 HunterNAS cache_dirs: ERROR: included directory "backups" does not exist.
Mar  1 06:37:39 HunterNAS cache_dirs: ERROR: included directory "documents" does not exist.
Mar  1 06:37:39 HunterNAS cache_dirs: ERROR: included directory "flashbkup" does not exist.
Mar  1 06:37:39 HunterNAS cache_dirs: ERROR: included directory "ndh" does not exist.
Mar  1 06:37:39 HunterNAS cache_dirs: cache_dirs process ID 13515 started

In particular, the Arguments appear to not match the setting in the GUI (see image).

Syslog shows (Highlighted are the deleted shares):

Mar  1 06:37:39 HunterNAS cache_dirs: Arguments=-u -i Backup -i Dropbox -i Goodsync -i ISO\ Files -i Pydio -i appdata -i backups -i cachebackup -i documents -i flashbkup -i home\ movies -i jwhbackup -i movies -i music -i ndh -i pictures -i tv

 

WebGUI shows:

Dropbox, ISO Files, Pydio, appdata, cachebackup, home movies, jwhbackup, movies, music, pictures, tv

 

Where can I edit the startup to correct this?

 

Cache_dirs problem.jpg

Link to comment

Most likely, it just needs to write the new list to its config.  Just make a change and save.  Or change something, save, then change it back and save.  You need the corrected list saved to the config file, removing the old entries.

 

I would also turn off "Scan user shares", because it's usually redundant.  Unless you have clearly found something that only gets cached with that setting on.  Otherwise it's wasting memory and time.

Link to comment
  • 2 weeks later...

I've been using cache_dirs for some time now and thought it was working properly until today. I went to pull up my main media directory which houses pretty much everything and Windows Explorer just hung there for a few seconds before showing the directory at the top level. I had unRAID up in the browser and sure enough all 4 of my drives spun up just to get a listing.

 

On unRAID 6.3.2 and cache_dirs 2.1.1. I don't see anything in the logs to indicate a problem. Anyone care to shed some light?

 

Settings attached.

 

cache_dirs-settings.PNG

Link to comment
17 minutes ago, acurcione said:

I've been using cache_dirs for some time now and thought it was working properly until today. I went to pull up my main media directory which houses pretty much everything and Windows Explorer just hung there for a few seconds before showing the directory at the top level. I had unRAID up in the browser and sure enough all 4 of my drives spun up just to get a listing.

 

See this post for some explanation of how cache_dirs works and how Windows Explorer might cause disks to spin up anyway.

 

Link to comment
  • 4 weeks later...

Hi, and sorry if this was asked and answered before.

i have all disk spin-ups with cache dirs enabled and massive read from one of the disks. for example, i have a cron job that backups some huge VM's files to my backup server. after some time all other disks spun up too. i think it's related to cache dirs, since when i disable it, all other disks spun down after 30 min. i tried RobJ's suggested vm.dirty* setting change mentioned here with no luck.. are there another suggestion's how to prevent disk spin-ups? i changed RAM size for unRAID to 8GB recently and still no joy.. 

Link to comment
  • 3 months later...

I had problems with something "thrashing" my disk with reads causing one disk to always stay spun up. After some troubleshooting I found that it was cache_dirs plugin.

It seems that one of my backup folders contains a large amount of directories, apparently too many to fit into RAM, so cache_dirs read the structure from disk every 10 seconds because the cached structure got evicted in the mean time. This resulted in endless directories reads from disk to memory. Not great.

 

Excluding that folder worked for the most part but some reads were still not cached; now I doubled my RAM, included the folder again and had zero read/write from any disk in the past few hours.

Is there any way to figure out how much memory the cached directory structure currently uses?

Link to comment
  • 2 weeks later...

I've been having an issue since the 6.x release series where cache_dirs will report 100% CPU usage after I use my time machine share (via AFP).

 

This share is explicitly excluded from cache_dirs since it contains a billion little files.

cache_dirs sits at 100% in top and uses one full cpu core

 

 3241 root      20   0   10244   2824   2100 R 100.0  0.1  80291:02 cache_dirs

 

the disks are not spinning up, as far as I can tell.
lsof doesn't report what it's actually doing (it's apparently not disk accesses as the numbers don't increase. Edit: Duh. Should have been looking for the "find" process, but still...):

 

cache_dir  3241             root  cwd       DIR                0,2      400          2 /
cache_dir  3241             root  rtd       DIR                0,2      400          2 /
cache_dir  3241             root  txt       REG                0,2  1094752       3371 /bin/bash
cache_dir  3241             root  mem       REG                0,2   174816       4468 /lib64/ld-2.24.so
cache_dir  3241             root  mem       REG                0,2    18436       4425 /lib64/libtermcap.so.2.0.8
cache_dir  3241             root  mem       REG                0,2    18808       4480 /lib64/libdl-2.24.so
cache_dir  3241             root  mem       REG                0,2  2067512       4474 /lib64/libc-2.24.so
cache_dir  3241             root    0r      CHR                1,3      0t0       2051 /dev/null
cache_dir  3241             root    1w      CHR                1,3      0t0       2051 /dev/null
cache_dir  3241             root    2w      CHR                1,3      0t0       2051 /dev/null
cache_dir  3241             root    6r      REG                0,2     2727       8292 /usr/local/emhttp/update.htm

 

I thought it may be related to internal linux file caching of the files on the AFP share, but if there was an issue with cache_dirs needing to recache drives, I'd think it would show disk accesses/spin up the disks.

 

restarting cache_dirs seems to fix it, but any idea how to debug what causes it so I don't need to manually do this every time?

 

I've attached diagnostics, any ideas?

tower-diagnostics-20170811-0212.zip

Edited by SnickySnacks
Link to comment
On 11-8-2017 at 9:20 AM, SnickySnacks said:

I've been having an issue since the 6.x release series where cache_dirs will report 100% CPU usage after I use my time machine share (via AFP).

 

This share is explicitly excluded from cache_dirs since it contains a billion little files.

cache_dirs sits at 100% in top and uses one full cpu core

 


 3241 root      20   0   10244   2824   2100 R 100.0  0.1  80291:02 cache_dirs

 

the disks are not spinning up, as far as I can tell.
lsof doesn't report what it's actually doing (it's apparently not disk accesses as the numbers don't increase. Edit: Duh. Should have been looking for the "find" process, but still...):

 


cache_dir  3241             root  cwd       DIR                0,2      400          2 /
cache_dir  3241             root  rtd       DIR                0,2      400          2 /
cache_dir  3241             root  txt       REG                0,2  1094752       3371 /bin/bash
cache_dir  3241             root  mem       REG                0,2   174816       4468 /lib64/ld-2.24.so
cache_dir  3241             root  mem       REG                0,2    18436       4425 /lib64/libtermcap.so.2.0.8
cache_dir  3241             root  mem       REG                0,2    18808       4480 /lib64/libdl-2.24.so
cache_dir  3241             root  mem       REG                0,2  2067512       4474 /lib64/libc-2.24.so
cache_dir  3241             root    0r      CHR                1,3      0t0       2051 /dev/null
cache_dir  3241             root    1w      CHR                1,3      0t0       2051 /dev/null
cache_dir  3241             root    2w      CHR                1,3      0t0       2051 /dev/null
cache_dir  3241             root    6r      REG                0,2     2727       8292 /usr/local/emhttp/update.htm

 

I thought it may be related to internal linux file caching of the files on the AFP share, but if there was an issue with cache_dirs needing to recache drives, I'd think it would show disk accesses/spin up the disks.

 

restarting cache_dirs seems to fix it, but any idea how to debug what causes it so I don't need to manually do this every time?

 

I've attached diagnostics, any ideas?

tower-diagnostics-20170811-0212.zip

 

I just noticed by accident the same issue on my server.  I saw 1 cpu pegged at 100% and it turned out to be cache_dirs.

 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 7270 root      20   0   10760   3036   2048 R 100.0  0.0 795:53.95 cache_dirs
 

lsof showed me

 

# lsof |grep -i 7270
cache_dir  7270             root  cwd       DIR                0,2      380          2 /
cache_dir  7270             root  rtd       DIR                0,2      380          2 /
cache_dir  7270             root  txt       REG                0,2  1094752        298 /bin/bash
cache_dir  7270             root  mem       REG                0,2   174816       2419 /lib64/ld-2.24.so
cache_dir  7270             root  mem       REG                0,2    18436       2376 /lib64/libtermcap.so.2.0.8
cache_dir  7270             root  mem       REG                0,2    18808       2431 /lib64/libdl-2.24.so
cache_dir  7270             root  mem       REG                0,2  2067512       2425 /lib64/libc-2.24.so
cache_dir  7270             root    0r     FIFO                0,8      0t0   59140270 pipe
cache_dir  7270             root    1w      CHR                1,3      0t0       1029 /dev/null
cache_dir  7270             root    2w      CHR                1,3      0t0       1029 /dev/null
cache_dir  7270             root    6r      REG                0,2     2727       6243 /usr/local/emhttp/update.htm
cache_dir  7270             root   10r      CHR                1,3      0t0       1029 /dev/null
cache_dir  7270             root   63r     FIFO                0,8      0t0   59140270 pipe
 

Running unraid 6.3.5

 

This is the first time I had this problem, and the only thing that change in my environment is

- upgrade of my docker plex image

- switch my htpc to using LibreELEC (Krypton) v8.0.2 from gotham

I have now reverted my htpc back to gotham and will see if the problem returns or not..

Link to comment

It doesn't seem to be related to file access or time machine. It happened again when I haven't used time machine in weeks.

 

Looks like it may be stuck waiting for a timeout or something.

 

I'm not really sure.

 

root@tower:/usr/local/emhttp/plugins/dynamix.cache.dirs/scripts# lsof | grep cache_dir
cache_dir 19058             root  cwd       DIR                0,2      220       7660 /usr/local/emhttp
cache_dir 19058             root  rtd       DIR                0,2      400          2 /
cache_dir 19058             root  txt       REG                0,2  1094752       3371 /bin/bash
cache_dir 19058             root  mem       REG                0,2   174816       4468 /lib64/ld-2.24.so
cache_dir 19058             root  mem       REG                0,2    18436       4425 /lib64/libtermcap.so.2.0.8
cache_dir 19058             root  mem       REG                0,2    18808       4480 /lib64/libdl-2.24.so
cache_dir 19058             root  mem       REG                0,2  2067512       4474 /lib64/libc-2.24.so
cache_dir 19058             root    0r     FIFO                0,8      0t0  105856711 pipe
cache_dir 19058             root    1w      CHR                1,3      0t0       2051 /dev/null
cache_dir 19058             root    2w      CHR                1,3      0t0       2051 /dev/null
cache_dir 19058             root   10r      CHR                1,3      0t0       2051 /dev/null
cache_dir 19058             root   63r     FIFO                0,8      0t0  105856711 pipe

root@tower:/usr/local/emhttp/plugins/dynamix.cache.dirs/scripts# lsof | grep find
root@tower:/usr/local/emhttp/plugins/dynamix.cache.dirs/scripts# pstree -pl 19058
cache_dirs(19058)───timeout(6371)
root@tower:/usr/local/emhttp/plugins/dynamix.cache.dirs/scripts# ps awx | grep timeout
 6371 ?        Z      0:00 [timeout] <defunct>
 9118 pts/0    S+     0:00 grep timeout

 

Link to comment
  • 2 weeks later...

Is Cache_Dir incompatible with OSX? It seems OSX will do a disk read no matter what, leading to it bypassing the directory cache?

 

I wonder if OSX is trying to read the .ds_store file containing the folder meta data.

 

If so, it would be good if unRAID has some way of keeping the .ds_store file on a SSD cache, thereby negating the need to spin up the whole array to read these small files. Does anyone know if there is a way to do that?

  • Like 1
Link to comment
  • 1 month later...

I'm not sure if I'm using this plugin correctly.  After installing the plugin does it work out of the box, or do I need to make the changes listed in the wiki?

 

https://wiki.lime-technology.com/index.php?title=Improving_unRAID_Performance#Keep_directory_entries_cached

Edit your go script to include the following line, near the bottom:
/boot/cache_dirs -w
This will go into effect on your next boot, and will cache all files and folders, with a scanning interval of between 1 and 10 seconds
The cache_dirs command is commonly customized, for example like this:
/boot/cache_dirs -d 5 -m 3 -M 5 -w

confused

Link to comment
14 minutes ago, DZMM said:

I'm not sure if I'm using this plugin correctly.  After installing the plugin does it work out of the box, or do I need to make the changes listed in the wiki?

 

https://wiki.lime-technology.com/index.php?title=Improving_unRAID_Performance#Keep_directory_entries_cached


Edit your go script to include the following line, near the bottom:
/boot/cache_dirs -w
This will go into effect on your next boot, and will cache all files and folders, with a scanning interval of between 1 and 10 seconds
The cache_dirs command is commonly customized, for example like this:
/boot/cache_dirs -d 5 -m 3 -M 5 -w

confused

Should basically work out of the box.  You can fine tune the parameters as needed.

Link to comment
  • 1 month later...

Every time I reboot my server I get an error that /boot/cache_dir: Does not exist.

 

I'm just guessing that the file has something to do with this plugin. It's being called from my go file-

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp &
/boot/cache_dirs -w
beep -l 300 -f 220.00 -n -l 300 -f 293.66 -D 100 -n -l 150 -f 220.00 -n -l 150 -f 293.66 -D 100 -n -l 500 -f 349.23 -D 100 -n -l 500 -f 293.66 -D 100 -n -l 300 -f 349.23 -D 100 -n -l 150 -f 293.66 -D 100 -n -l 150 -f 349.23 -D 100 -n -l 500 -f 440.00 -D 100 -n -l 500 -f 349.23 -D 100 -n -l 300 -f 440.00 -D 100 -n -l 150 -f 349.23 -D 100 -n -l 150 -f 440.00 -D 100 -n -l 500 -f 523.25 -D 100 -n -l 500 -f 261.63 -D 100 -n -l 300 -f 349.23 -D 100 -n -l 150 -f 261.63 -D 100 -n -l 150 -f 349.23 -D 100 -n -l 500 -f 440.00 -D 300 -n -l 300 -f 261.63 -n -l 300 -f 349.23 -D 100 -n -l 150 -f 261.63 -n -l 150 -f 349.23 -D 100 -n -l 500 -f 440.00 -D 100 -n -l 500 -f 349.23 -D 100 -n -l 300 -f 440.00 -D 100 -n -l 150 -f 349.23 -D 100 -n -l 150 -f 440.00 -D 100 -n -l 500 -f 523.25 -D 100 -n -l 500 -f 440.00 -D 100 -n -l 300 -f 523.25 -D 100 -n -l 150 -f 440.00 -D 100 -n -l 150 -f 523.25 -D 100 -n -l 500 -f 659.26 -D 100 -n -l 500 -f 329.63 -D 100 -n -l 300 -f 440.00 -D 100 -n -l 150 -f 329.63 -D 100 -n -l 150 -f 440.00 -D 100 -n -l 500 -f 554.00 -D 300 -n -l 300 -f 220.00 -D 100 -n -l 300 -f 233.00 -D 100 -n -l 150 -f 196.00 -D 100 -n -l 300 -f 223.00 -D 100 -n -l 750 -f 293.66 -D 300 -n -l 300 -f 220.00 -D 100 -n -l 300 -f 233.00 -D 100 -n -l 150 -f 196.00 -D 100 -n -l 300 -f 223.00 -D 100 -n -l 750 -f 311.00 -D 100 -n -l 300 -f 293.66 -D 100 -n -l 150 -f 220.00 -D 100 -n -l 150 -f 293.66 -D 100 -n -l 500 -f 349.23 -D 100 -n -l 300 -f 220.00 -D 100 -n -l 300 -f 233.00 -D 100 -n -l 150 -f 196.00 -D 100 -n -l 150 -f 233.00 -D 100 -n -l 500 -f 293.66 -D 300 -n -l 300 -f 220.00 -D 100 -n -l 300 -f 233.00 -D 100 -n -l 150 -f 196.00 -D 100 -n -l 150 -f 233.00 -D 100 -n -l 500 -f 329.63 -D 100 -n -l 300 -f 220.00 -D 100 -n -l 300 -f 220.00 -D 100 -n -l 150 -f 293.66 -D 100 -n -l 150 -f 370.00 -D 100 -n -l 500 -f 440.00 -D 100
# force iptable mangle module to load (required for *vpn dockers)
/sbin/modprobe iptable_mangle
# Enlarge the LOG partition
mount -o remount,size=384m /var/log

Should I just delete the line for my go file or am I missing a file?

Link to comment
  • 1 month later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.