trurl Posted November 14, 2016 Share Posted November 14, 2016 Its purpose is to keep directory entries in RAM so drives don't have to keep spinning up. Aha, I understand! Many thanks for this sentance. "Dynamix Cache Directories" only keeps directory structure of users shares in directory entries on cache drive. Copy structure of users share. It keeps the directory entries in RAM as RobJ said, not on cache drive. Can I set this opt to "Prefer" or "Only" to prevention move the share-directory to users share? Only will only write to cache and will not move it from cache. If you have a cache drive, Prefer will only write to cache, but if any of the share is already on other drives it will move to cache. This allows you to add a cache later and then get the share to use it as if it were cache-only. Turn on Help in the upper right of the webUI. Quote Link to comment
Ropo Posted November 14, 2016 Share Posted November 14, 2016 It keeps the directory entries in RAM as RobJ said, not on cache drive. If you have a cache drive, Prefer will only write to cache, but if any of the share is already on other drives it will move to cache. This allows you to add a cache later and then get the share to use it as if it were cache-only. Ok. Many thanks, trurl! Quote Link to comment
coolspot Posted January 13, 2017 Share Posted January 13, 2017 Hi all, What does the Minimum interval between folder scans (sec) and Maximum interval between folder scans (sec) do? Does cache_dir physically scan based on these parameters? So will it spin up the drives every 10 seconds (default maximum interval) to search directories for new entries? Thanks. Quote Link to comment
Squid Posted January 13, 2017 Share Posted January 13, 2017 So will it spin up the drives every 10 seconds (default maximum interval) to search directories for new entries? Thanks. The reason for the seemingly quick rescan time is so that it won't have to spin up the drives. Cache-dirs attempts to keep the directory structures in memory by rescanning all the time. (Linux if the directory structure isn't used in a while will automatically drop it from memory) However, there are times where Cache Dirs will be unable to keep the entries in memory, and under those circumstances a spin up will happen (but it shouldn't happen again until those circumstances repeat themselves) Quote Link to comment
coolspot Posted January 13, 2017 Share Posted January 13, 2017 So will it spin up the drives every 10 seconds (default maximum interval) to search directories for new entries? Thanks. The reason for the seemingly quick rescan time is so that it won't have to spin up the drives. Cache-dirs attempts to keep the directory structures in memory by rescanning all the time. (Linux if the directory structure isn't used in a while will automatically drop it from memory) Ah - so it's primarily a memory scan to keep the data in RAM and prevent a flush. If the data is flushed, cache_dir will then do a physical scan and reload the data. Thanks for the clarification, I was wondering how it worked. Quote Link to comment
Alex R. Berg Posted February 22, 2017 Share Posted February 22, 2017 Hi Zin105, You can limit the cache_dirs to only scan some shares, and you can also limit it to only scan to a certain depth. I have attached a program you can use to count where the bulk of your files are, it runs the following command at various depth find $f -type f -maxdepth ${DEPTH} | wc -l You can also you my modified cache-dirs which I link at this post: I modified it to stop scanning disks forever, if it cannot have files in cache, it now adaptively adjust its scan-depth according to how much it can have in memory, which is judged by the scan duration. When my server is pressed with other stuff, cache_dirs would lose its cache, and start scanning, causing my server to be even more pressed, which seemed silly. My adjustments makes cache_dirs scan only to a low depth in such a pressed situation and then stop after a couple of minutes attempt. You are welcome to try it out, but don't expect support, because I'm to busy elsewhere... But good luck with whatever you choose. cache_dirs_file_count Quote Link to comment
jeffreywhunter Posted March 1, 2017 Share Posted March 1, 2017 I recently removed a number of unused shares. When these cached shares are removed from the system, it appears that CacheDirs still tries to start them up. Where can I correct this error? Here's a log snippet of CacheDirs. Mar 1 06:37:39 HunterNAS cache_dirs: ============================================== Mar 1 06:37:39 HunterNAS cache_dirs: Starting cache_dirs: Mar 1 06:37:39 HunterNAS cache_dirs: Arguments=-u -i Backup -i Dropbox -i Goodsync -i ISO\ Files -i Pydio -i appdata -i backups -i cachebackup -i documents -i flashbkup -i home\ movies -i jwhbackup -i movies -i music -i ndh -i pictures -i tv Mar 1 06:37:39 HunterNAS cache_dirs: Cache Pressure=10 Mar 1 06:37:39 HunterNAS cache_dirs: Max Scan Secs=10, Min Scan Secs=1 Mar 1 06:37:39 HunterNAS cache_dirs: Scan Type=adaptive Mar 1 06:37:39 HunterNAS cache_dirs: Max Scan Depth=none Mar 1 06:37:39 HunterNAS cache_dirs: Use Command='find -noleaf' Mar 1 06:37:39 HunterNAS cache_dirs: Version=2.1.1 Mar 1 06:37:39 HunterNAS cache_dirs: ---------- Caching Directories --------------- Mar 1 06:37:39 HunterNAS cache_dirs: D..x Mar 1 06:37:39 HunterNAS cache_dirs: I..s Mar 1 06:37:39 HunterNAS cache_dirs: P..o Mar 1 06:37:39 HunterNAS cache_dirs: a..a Mar 1 06:37:39 HunterNAS cache_dirs: c..p Mar 1 06:37:39 HunterNAS cache_dirs: h..s Mar 1 06:37:39 HunterNAS cache_dirs: j..p Mar 1 06:37:39 HunterNAS cache_dirs: m..s Mar 1 06:37:39 HunterNAS cache_dirs: m..c Mar 1 06:37:39 HunterNAS cache_dirs: p..s Mar 1 06:37:39 HunterNAS cache_dirs: t..v Mar 1 06:37:39 HunterNAS cache_dirs: ---------------------------------------------- Mar 1 06:37:39 HunterNAS cache_dirs: ERROR: included directory "Backup" does not exist. Mar 1 06:37:39 HunterNAS cache_dirs: ERROR: included directory "Goodsync" does not exist. Mar 1 06:37:39 HunterNAS cache_dirs: ERROR: included directory "backups" does not exist. Mar 1 06:37:39 HunterNAS cache_dirs: ERROR: included directory "documents" does not exist. Mar 1 06:37:39 HunterNAS cache_dirs: ERROR: included directory "flashbkup" does not exist. Mar 1 06:37:39 HunterNAS cache_dirs: ERROR: included directory "ndh" does not exist. Mar 1 06:37:39 HunterNAS cache_dirs: cache_dirs process ID 13515 started In particular, the Arguments appear to not match the setting in the GUI (see image). Syslog shows (Highlighted are the deleted shares): Mar 1 06:37:39 HunterNAS cache_dirs: Arguments=-u -i Backup -i Dropbox -i Goodsync -i ISO\ Files -i Pydio -i appdata -i backups -i cachebackup -i documents -i flashbkup -i home\ movies -i jwhbackup -i movies -i music -i ndh -i pictures -i tv WebGUI shows: Dropbox, ISO Files, Pydio, appdata, cachebackup, home movies, jwhbackup, movies, music, pictures, tv Where can I edit the startup to correct this? Quote Link to comment
RobJ Posted March 1, 2017 Share Posted March 1, 2017 Most likely, it just needs to write the new list to its config. Just make a change and save. Or change something, save, then change it back and save. You need the corrected list saved to the config file, removing the old entries. I would also turn off "Scan user shares", because it's usually redundant. Unless you have clearly found something that only gets cached with that setting on. Otherwise it's wasting memory and time. Quote Link to comment
jeffreywhunter Posted March 1, 2017 Share Posted March 1, 2017 That did it! Making a change caused it to correct the config. Thanks! Quote Link to comment
acurcione Posted March 15, 2017 Share Posted March 15, 2017 I've been using cache_dirs for some time now and thought it was working properly until today. I went to pull up my main media directory which houses pretty much everything and Windows Explorer just hung there for a few seconds before showing the directory at the top level. I had unRAID up in the browser and sure enough all 4 of my drives spun up just to get a listing. On unRAID 6.3.2 and cache_dirs 2.1.1. I don't see anything in the logs to indicate a problem. Anyone care to shed some light? Settings attached. Quote Link to comment
trurl Posted March 15, 2017 Share Posted March 15, 2017 17 minutes ago, acurcione said: I've been using cache_dirs for some time now and thought it was working properly until today. I went to pull up my main media directory which houses pretty much everything and Windows Explorer just hung there for a few seconds before showing the directory at the top level. I had unRAID up in the browser and sure enough all 4 of my drives spun up just to get a listing. See this post for some explanation of how cache_dirs works and how Windows Explorer might cause disks to spin up anyway. Quote Link to comment
uldise Posted April 9, 2017 Share Posted April 9, 2017 Hi, and sorry if this was asked and answered before. i have all disk spin-ups with cache dirs enabled and massive read from one of the disks. for example, i have a cron job that backups some huge VM's files to my backup server. after some time all other disks spun up too. i think it's related to cache dirs, since when i disable it, all other disks spun down after 30 min. i tried RobJ's suggested vm.dirty* setting change mentioned here with no luck.. are there another suggestion's how to prevent disk spin-ups? i changed RAM size for unRAID to 8GB recently and still no joy.. Quote Link to comment
lionceau Posted July 27, 2017 Share Posted July 27, 2017 I had problems with something "thrashing" my disk with reads causing one disk to always stay spun up. After some troubleshooting I found that it was cache_dirs plugin. It seems that one of my backup folders contains a large amount of directories, apparently too many to fit into RAM, so cache_dirs read the structure from disk every 10 seconds because the cached structure got evicted in the mean time. This resulted in endless directories reads from disk to memory. Not great. Excluding that folder worked for the most part but some reads were still not cached; now I doubled my RAM, included the folder again and had zero read/write from any disk in the past few hours. Is there any way to figure out how much memory the cached directory structure currently uses? Quote Link to comment
SnickySnacks Posted August 11, 2017 Share Posted August 11, 2017 (edited) I've been having an issue since the 6.x release series where cache_dirs will report 100% CPU usage after I use my time machine share (via AFP). This share is explicitly excluded from cache_dirs since it contains a billion little files. cache_dirs sits at 100% in top and uses one full cpu core 3241 root 20 0 10244 2824 2100 R 100.0 0.1 80291:02 cache_dirs the disks are not spinning up, as far as I can tell. lsof doesn't report what it's actually doing (it's apparently not disk accesses as the numbers don't increase. Edit: Duh. Should have been looking for the "find" process, but still...): cache_dir 3241 root cwd DIR 0,2 400 2 / cache_dir 3241 root rtd DIR 0,2 400 2 / cache_dir 3241 root txt REG 0,2 1094752 3371 /bin/bash cache_dir 3241 root mem REG 0,2 174816 4468 /lib64/ld-2.24.so cache_dir 3241 root mem REG 0,2 18436 4425 /lib64/libtermcap.so.2.0.8 cache_dir 3241 root mem REG 0,2 18808 4480 /lib64/libdl-2.24.so cache_dir 3241 root mem REG 0,2 2067512 4474 /lib64/libc-2.24.so cache_dir 3241 root 0r CHR 1,3 0t0 2051 /dev/null cache_dir 3241 root 1w CHR 1,3 0t0 2051 /dev/null cache_dir 3241 root 2w CHR 1,3 0t0 2051 /dev/null cache_dir 3241 root 6r REG 0,2 2727 8292 /usr/local/emhttp/update.htm I thought it may be related to internal linux file caching of the files on the AFP share, but if there was an issue with cache_dirs needing to recache drives, I'd think it would show disk accesses/spin up the disks. restarting cache_dirs seems to fix it, but any idea how to debug what causes it so I don't need to manually do this every time? I've attached diagnostics, any ideas? tower-diagnostics-20170811-0212.zip Edited August 11, 2017 by SnickySnacks Quote Link to comment
themaxxz Posted August 18, 2017 Share Posted August 18, 2017 On 11-8-2017 at 9:20 AM, SnickySnacks said: I've been having an issue since the 6.x release series where cache_dirs will report 100% CPU usage after I use my time machine share (via AFP). This share is explicitly excluded from cache_dirs since it contains a billion little files. cache_dirs sits at 100% in top and uses one full cpu core 3241 root 20 0 10244 2824 2100 R 100.0 0.1 80291:02 cache_dirs the disks are not spinning up, as far as I can tell. lsof doesn't report what it's actually doing (it's apparently not disk accesses as the numbers don't increase. Edit: Duh. Should have been looking for the "find" process, but still...): cache_dir 3241 root cwd DIR 0,2 400 2 / cache_dir 3241 root rtd DIR 0,2 400 2 / cache_dir 3241 root txt REG 0,2 1094752 3371 /bin/bash cache_dir 3241 root mem REG 0,2 174816 4468 /lib64/ld-2.24.so cache_dir 3241 root mem REG 0,2 18436 4425 /lib64/libtermcap.so.2.0.8 cache_dir 3241 root mem REG 0,2 18808 4480 /lib64/libdl-2.24.so cache_dir 3241 root mem REG 0,2 2067512 4474 /lib64/libc-2.24.so cache_dir 3241 root 0r CHR 1,3 0t0 2051 /dev/null cache_dir 3241 root 1w CHR 1,3 0t0 2051 /dev/null cache_dir 3241 root 2w CHR 1,3 0t0 2051 /dev/null cache_dir 3241 root 6r REG 0,2 2727 8292 /usr/local/emhttp/update.htm I thought it may be related to internal linux file caching of the files on the AFP share, but if there was an issue with cache_dirs needing to recache drives, I'd think it would show disk accesses/spin up the disks. restarting cache_dirs seems to fix it, but any idea how to debug what causes it so I don't need to manually do this every time? I've attached diagnostics, any ideas? tower-diagnostics-20170811-0212.zip I just noticed by accident the same issue on my server. I saw 1 cpu pegged at 100% and it turned out to be cache_dirs. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7270 root 20 0 10760 3036 2048 R 100.0 0.0 795:53.95 cache_dirs lsof showed me # lsof |grep -i 7270 cache_dir 7270 root cwd DIR 0,2 380 2 / cache_dir 7270 root rtd DIR 0,2 380 2 / cache_dir 7270 root txt REG 0,2 1094752 298 /bin/bash cache_dir 7270 root mem REG 0,2 174816 2419 /lib64/ld-2.24.so cache_dir 7270 root mem REG 0,2 18436 2376 /lib64/libtermcap.so.2.0.8 cache_dir 7270 root mem REG 0,2 18808 2431 /lib64/libdl-2.24.so cache_dir 7270 root mem REG 0,2 2067512 2425 /lib64/libc-2.24.so cache_dir 7270 root 0r FIFO 0,8 0t0 59140270 pipe cache_dir 7270 root 1w CHR 1,3 0t0 1029 /dev/null cache_dir 7270 root 2w CHR 1,3 0t0 1029 /dev/null cache_dir 7270 root 6r REG 0,2 2727 6243 /usr/local/emhttp/update.htm cache_dir 7270 root 10r CHR 1,3 0t0 1029 /dev/null cache_dir 7270 root 63r FIFO 0,8 0t0 59140270 pipe Running unraid 6.3.5 This is the first time I had this problem, and the only thing that change in my environment is - upgrade of my docker plex image - switch my htpc to using LibreELEC (Krypton) v8.0.2 from gotham I have now reverted my htpc back to gotham and will see if the problem returns or not.. Quote Link to comment
SnickySnacks Posted August 25, 2017 Share Posted August 25, 2017 It doesn't seem to be related to file access or time machine. It happened again when I haven't used time machine in weeks. Looks like it may be stuck waiting for a timeout or something. I'm not really sure. root@tower:/usr/local/emhttp/plugins/dynamix.cache.dirs/scripts# lsof | grep cache_dir cache_dir 19058 root cwd DIR 0,2 220 7660 /usr/local/emhttp cache_dir 19058 root rtd DIR 0,2 400 2 / cache_dir 19058 root txt REG 0,2 1094752 3371 /bin/bash cache_dir 19058 root mem REG 0,2 174816 4468 /lib64/ld-2.24.so cache_dir 19058 root mem REG 0,2 18436 4425 /lib64/libtermcap.so.2.0.8 cache_dir 19058 root mem REG 0,2 18808 4480 /lib64/libdl-2.24.so cache_dir 19058 root mem REG 0,2 2067512 4474 /lib64/libc-2.24.so cache_dir 19058 root 0r FIFO 0,8 0t0 105856711 pipe cache_dir 19058 root 1w CHR 1,3 0t0 2051 /dev/null cache_dir 19058 root 2w CHR 1,3 0t0 2051 /dev/null cache_dir 19058 root 10r CHR 1,3 0t0 2051 /dev/null cache_dir 19058 root 63r FIFO 0,8 0t0 105856711 pipe root@tower:/usr/local/emhttp/plugins/dynamix.cache.dirs/scripts# lsof | grep find root@tower:/usr/local/emhttp/plugins/dynamix.cache.dirs/scripts# pstree -pl 19058 cache_dirs(19058)───timeout(6371) root@tower:/usr/local/emhttp/plugins/dynamix.cache.dirs/scripts# ps awx | grep timeout 6371 ? Z 0:00 [timeout] <defunct> 9118 pts/0 S+ 0:00 grep timeout Quote Link to comment
coolspot Posted September 7, 2017 Share Posted September 7, 2017 Is Cache_Dir incompatible with OSX? It seems OSX will do a disk read no matter what, leading to it bypassing the directory cache? I wonder if OSX is trying to read the .ds_store file containing the folder meta data. If so, it would be good if unRAID has some way of keeping the .ds_store file on a SSD cache, thereby negating the need to spin up the whole array to read these small files. Does anyone know if there is a way to do that? 1 Quote Link to comment
DZMM Posted October 15, 2017 Share Posted October 15, 2017 I'm not sure if I'm using this plugin correctly. After installing the plugin does it work out of the box, or do I need to make the changes listed in the wiki? https://wiki.lime-technology.com/index.php?title=Improving_unRAID_Performance#Keep_directory_entries_cached Edit your go script to include the following line, near the bottom: /boot/cache_dirs -w This will go into effect on your next boot, and will cache all files and folders, with a scanning interval of between 1 and 10 seconds The cache_dirs command is commonly customized, for example like this: /boot/cache_dirs -d 5 -m 3 -M 5 -w confused Quote Link to comment
Squid Posted October 15, 2017 Share Posted October 15, 2017 14 minutes ago, DZMM said: I'm not sure if I'm using this plugin correctly. After installing the plugin does it work out of the box, or do I need to make the changes listed in the wiki? https://wiki.lime-technology.com/index.php?title=Improving_unRAID_Performance#Keep_directory_entries_cached Edit your go script to include the following line, near the bottom: /boot/cache_dirs -w This will go into effect on your next boot, and will cache all files and folders, with a scanning interval of between 1 and 10 seconds The cache_dirs command is commonly customized, for example like this: /boot/cache_dirs -d 5 -m 3 -M 5 -w confused Should basically work out of the box. You can fine tune the parameters as needed. Quote Link to comment
DZMM Posted October 15, 2017 Share Posted October 15, 2017 32 minutes ago, Squid said: Should basically work out of the box. You can fine tune the parameters as needed. Thanks. I'm on a mission to reduce server noise and spin ups this week Quote Link to comment
wgstarks Posted November 21, 2017 Share Posted November 21, 2017 Every time I reboot my server I get an error that /boot/cache_dir: Does not exist. I'm just guessing that the file has something to do with this plugin. It's being called from my go file- #!/bin/bash # Start the Management Utility /usr/local/sbin/emhttp & /boot/cache_dirs -w beep -l 300 -f 220.00 -n -l 300 -f 293.66 -D 100 -n -l 150 -f 220.00 -n -l 150 -f 293.66 -D 100 -n -l 500 -f 349.23 -D 100 -n -l 500 -f 293.66 -D 100 -n -l 300 -f 349.23 -D 100 -n -l 150 -f 293.66 -D 100 -n -l 150 -f 349.23 -D 100 -n -l 500 -f 440.00 -D 100 -n -l 500 -f 349.23 -D 100 -n -l 300 -f 440.00 -D 100 -n -l 150 -f 349.23 -D 100 -n -l 150 -f 440.00 -D 100 -n -l 500 -f 523.25 -D 100 -n -l 500 -f 261.63 -D 100 -n -l 300 -f 349.23 -D 100 -n -l 150 -f 261.63 -D 100 -n -l 150 -f 349.23 -D 100 -n -l 500 -f 440.00 -D 300 -n -l 300 -f 261.63 -n -l 300 -f 349.23 -D 100 -n -l 150 -f 261.63 -n -l 150 -f 349.23 -D 100 -n -l 500 -f 440.00 -D 100 -n -l 500 -f 349.23 -D 100 -n -l 300 -f 440.00 -D 100 -n -l 150 -f 349.23 -D 100 -n -l 150 -f 440.00 -D 100 -n -l 500 -f 523.25 -D 100 -n -l 500 -f 440.00 -D 100 -n -l 300 -f 523.25 -D 100 -n -l 150 -f 440.00 -D 100 -n -l 150 -f 523.25 -D 100 -n -l 500 -f 659.26 -D 100 -n -l 500 -f 329.63 -D 100 -n -l 300 -f 440.00 -D 100 -n -l 150 -f 329.63 -D 100 -n -l 150 -f 440.00 -D 100 -n -l 500 -f 554.00 -D 300 -n -l 300 -f 220.00 -D 100 -n -l 300 -f 233.00 -D 100 -n -l 150 -f 196.00 -D 100 -n -l 300 -f 223.00 -D 100 -n -l 750 -f 293.66 -D 300 -n -l 300 -f 220.00 -D 100 -n -l 300 -f 233.00 -D 100 -n -l 150 -f 196.00 -D 100 -n -l 300 -f 223.00 -D 100 -n -l 750 -f 311.00 -D 100 -n -l 300 -f 293.66 -D 100 -n -l 150 -f 220.00 -D 100 -n -l 150 -f 293.66 -D 100 -n -l 500 -f 349.23 -D 100 -n -l 300 -f 220.00 -D 100 -n -l 300 -f 233.00 -D 100 -n -l 150 -f 196.00 -D 100 -n -l 150 -f 233.00 -D 100 -n -l 500 -f 293.66 -D 300 -n -l 300 -f 220.00 -D 100 -n -l 300 -f 233.00 -D 100 -n -l 150 -f 196.00 -D 100 -n -l 150 -f 233.00 -D 100 -n -l 500 -f 329.63 -D 100 -n -l 300 -f 220.00 -D 100 -n -l 300 -f 220.00 -D 100 -n -l 150 -f 293.66 -D 100 -n -l 150 -f 370.00 -D 100 -n -l 500 -f 440.00 -D 100 # force iptable mangle module to load (required for *vpn dockers) /sbin/modprobe iptable_mangle # Enlarge the LOG partition mount -o remount,size=384m /var/log Should I just delete the line for my go file or am I missing a file? Quote Link to comment
BRiT Posted November 21, 2017 Share Posted November 21, 2017 I would switch over to the dynamix cache dirs. plugin and remove the line from your go file. Quote Link to comment
wgstarks Posted November 21, 2017 Share Posted November 21, 2017 Just now, BRiT said: I would switch over to the dynamix cache dirs. plugin and remove the line from your go file. Actually thats the one I have installed. Guess that line is just an old leftover. Quote Link to comment
cholzer Posted November 23, 2017 Share Posted November 23, 2017 Hi there! I just installed the plugin an noticed that "Folder caching function:" is set to "disabled". I guess I have to enable that for the plugin to work, right? Or is that some additional feature to let the system cache "more" folders / deeper into the folder tree? Quote Link to comment
Necrotic Posted January 13, 2018 Share Posted January 13, 2018 Did anyone else have issues with cachedirs going nuts and pegging one cpu to 100% forever? It happens rarely but over the past year or something had it happen twice. I went into settings, disabled it and re-enabled and it fixed it. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.