interwebtech Posted May 23, 2019 Share Posted May 23, 2019 cache_dirs appears to not be working properly with the latest 6.7 stable. Opening a cached share in windows exploder almost always lights up a drive on the array which delays the display of folders until its spun up. I've tried adjusting "Cache pressure of directories on system" from 10 to 1 but no change in behavior. logs attached tower-diagnostics-20190523-1807.zip Quote Link to comment
gacpac Posted May 25, 2019 Share Posted May 25, 2019 I'm new with this plugin and just set it up. Is it possible that using this plugin will prevent dockers to spin disks randomly? I'm having a bit of an issue with the Transmission docker. Quote Link to comment
trurl Posted May 27, 2019 Share Posted May 27, 2019 On 5/24/2019 at 9:27 PM, gacpac said: I'm new with this plugin and just set it up. Is it possible that using this plugin will prevent dockers to spin disks randomly? I'm having a bit of an issue with the Transmission docker. Go to User Shares, Compute All, post screenshot. Quote Link to comment
gacpac Posted May 27, 2019 Share Posted May 27, 2019 Go to User Shares, Compute All, post screenshot.Sorry for the late reply. I fixed itSent from my Pixel 2 XL using Tapatalk Quote Link to comment
harleywastaken Posted May 28, 2019 Share Posted May 28, 2019 (edited) On 5/23/2019 at 11:13 AM, interwebtech said: cache_dirs appears to not be working properly with the latest 6.7 stable. Opening a cached share in windows exploder almost always lights up a drive on the array which delays the display of folders until its spun up. I've tried adjusting "Cache pressure of directories on system" from 10 to 1 but no change in behavior. logs attached tower-diagnostics-20190523-1807.zipUnavailable i'm not using windows, however i'm also seeing drives spin up when i would not expect them to. in your case i'd guess Explorer is trying to read some file metadata (thumbnails for picture files, etc) which cache_dirs cannot help with. in my case i _suspect_ it's sonarr/radarr running in Docker containers doing the same thing as the spin-ups appear to coincide with either app starting a library scan, as well as the Settings > Media Management > File Management > Analyze Video Files option being enabled... hopefully i'll have time to debug further soon. edit: duplicate episodes/movies were my issue - on each scan the duplicate was found and it was scanned for metadata during the process to compare it with the existing episode/movie to see if it should be replaced. once the dupes were removed there was no comparison to be made and the disks do not spin up. Edited May 28, 2019 by harleywastaken additional info Quote Link to comment
gacpac Posted May 28, 2019 Share Posted May 28, 2019 How can I test if the cache plugin is working as supposed to?If the drive is not spinning and the cache is working. I should be able to browse my shares. At least the top folder, right? Sent from my Pixel 2 XL using Tapatalk Quote Link to comment
JonathanM Posted May 29, 2019 Share Posted May 29, 2019 5 hours ago, gacpac said: If the drive is not spinning and the cache is working. I should be able to browse my shares. At least the top folder, right? Depends on your definition of browse. If you mean list the file names, yes. Quote Link to comment
Alex R. Berg Posted June 23, 2019 Share Posted June 23, 2019 On 5/23/2019 at 8:13 PM, interwebtech said: cache_dirs appears to not be working properly with the latest 6.7 stable. Opening a cached share in windows exploder almost always lights up a drive on the array which delays the display of folders until its spun up. I've tried adjusting "Cache pressure of directories on system" from 10 to 1 but no change in behavior. logs attached tower-diagnostics-20190523-1807.zip 159.48 kB · 1 download Its working for me on unRaid 6.7.0. I tried accessing a folder on my disk2 both from windows, unRaid itself, in both cases both user and disk share. Disk didn't spin up. I'm filled up at the moment, so I probably don't investigate much, but if you want to supply logs, if it is to be useful, the logs from cache-dirs are needed Cache_dirs has a command to collect them. It would be awesome it that could somehow be collected automatically by unRaid, for instance by some kind of plugin-event. I assume that is not possible, but if it is, let me know. If not, it might make sense to ask Tom @ Limetech if that is a reasonable feature to add. Maybe you feel like doing that interwebtech, or tell me where to post? Best Alex Quote Link to comment
papnikol Posted June 24, 2019 Share Posted June 24, 2019 (edited) Before updating to 6.7.1 I run the update assistant and I get the following: Checking for plugin compatibility Issue Found: dynamix.cache.dirs.plg is not known to Community Applications. Compatibility for this plugin CANNOT be determined and it may cause you issues. I have the latest cache dirs version (2018.12.04). Does anyone have problems with cache dirs and 6.7.1? Edited June 24, 2019 by papnikol Quote Link to comment
Alex R. Berg Posted June 24, 2019 Share Posted June 24, 2019 Hmm, its just an annoying warning, putting a scare into users. Unfortunately some GitHub merge stuff went badly between my developing fork and the Community app version by Bergware. Hence the warning, I think. If asked Bergware if he'll fix it, or if perhaps we should switch community apps to this repo. It works fine on unRaid 6.7.1 Quote Link to comment
papnikol Posted June 25, 2019 Share Posted June 25, 2019 9 hours ago, Alex R. Berg said: Hmm, its just an annoying warning, putting a scare into users. Unfortunately some GitHub merge stuff went badly between my developing fork and the Community app version by Bergware. Hence the warning, I think. If asked Bergware if he'll fix it, or if perhaps we should switch community apps to this repo. It works fine on unRaid 6.7.1 Thanks for your response, I though so, I just wanted to double check Quote Link to comment
comet424 Posted December 15, 2019 Share Posted December 15, 2019 hi I have this cahce dir installed and left the defaults... is it supposed ot load entire hard drives directories into ram? If so I find it not working.. it will do the first Folder listtins so what I mean is \\tower\ it will show those file folders if I click into one its fast but to go into anything after that is slow as it waits for the drives to boot up to see whats in the directories.. whats the proper format to load everything and how much ram you need Quote Link to comment
pyrater Posted April 10, 2020 Share Posted April 10, 2020 (edited) Is there anyway to trouble shoot this plugin? I have it running but none of my drives spun down after 3 hours. The only thing being accessed is my SSD / Cache pool for plex. Edited April 10, 2020 by pyrater Quote Link to comment
Frank1940 Posted April 10, 2020 Share Posted April 10, 2020 How much are you caching and how much memory is installed. I seem to recall that as memory is filled with cached data, It will discard the earlier stuff to store the later data. So you can get caught in a round-robin. Try caching only the shares (and depth) you need at a regular basis to see if that helps. Quote Link to comment
comet424 Posted April 10, 2020 Share Posted April 10, 2020 how do you guys cache more then 1 folder the main shares if you click on them you see the folders inside them quickly but when you go it another folder its not cacheced so example \\tower Music country hip hop so Music Folder is cached it seems but not country or hip hop as example how do you cache the entire file directories? and im using 32gb of ram or do I need more.. as its not using it.. Quote Link to comment
pyrater Posted April 10, 2020 Share Posted April 10, 2020 (edited) 24 minutes ago, Frank1940 said: How much are you caching and how much memory is installed. I seem to recall that as memory is filled with cached data, It will discard the earlier stuff to store the later data. So you can get caught in a round-robin. Try caching only the shares (and depth) you need at a regular basis to see if that helps. .....Frank that may very well be it.... I am caching 83 TBs on 16GB of memory with 8GB of that dedicated to a plex transcode ramdisk. Is it possible to check this? top - 12:01:28 up 2 days, 22:15, 3 users, load average: 6.91, 6.34, 4.82 Tasks: 388 total, 2 running, 386 sleeping, 0 stopped, 0 zombie %Cpu(s): 90.3 us, 2.1 sy, 4.2 ni, 3.4 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st MiB Mem : 15732.1 total, 156.5 free, 5446.3 used, 10129.3 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 6932.5 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11058 nobody 20 0 1195292 359328 18088 R 370.0 2.2 23:59.63 Plex Transcoder 6044 nobody 20 0 17932 1444 672 S 2.3 0.0 1:08.15 EasyAudioEncode 5440 nobody 20 0 753832 50128 9584 S 1.7 0.3 4:13.90 Plex Transcoder 4001 nobody 20 0 2779608 1.4g 14600 S 1.3 9.1 37:09.63 Plex Media Serv 32029 root 20 0 839180 31592 17996 S 1.3 0.2 54:03.57 containerd 24305 root 20 0 1596456 348108 992 S 1.0 2.2 2370:41 shfs 11340 root 20 0 152380 33148 27360 S 0.7 0.2 0:00.02 docker 32013 root 20 0 1260828 73996 43840 S 0.7 0.5 34:43.80 dockerd 10 root 20 0 0 0 0 I 0.3 0.0 2:21.84 rcu_sched 2465 root 20 0 109104 7948 5184 S 0.3 0.0 5:00.19 containerd-shim 2610 root 20 0 109104 9960 4864 S 0.3 0.1 0:56.35 containerd-shim 3305 root 20 0 109104 9960 4800 S 0.3 0.1 1:01.88 containerd-shim 8343 root 20 0 105276 13532 7504 S 0.3 0.1 0:00.03 php-fpm 11292 root 20 0 6724 3204 2432 R 0.3 0.0 0:00.01 top 23070 root 20 0 349176 4260 3404 S 0.3 0.0 8:11.76 emhttpd 23836 root 20 0 0 0 0 S 0.3 0.0 2:05.27 unraidd1 23951 root 20 0 3792 2736 2464 S 0.3 0.0 0:47.52 diskload 31975 root 0 -20 0 0 0 S 0.3 0.0 8:15.51 loop2 32727 root 20 0 30276 17024 1988 S 0.3 0.1 0:24.80 supervisord 1 root 20 0 2468 1732 1620 S 0.0 0.0 0:24.17 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.04 kthreadd 3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp 6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H-kblockd 8 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq 9 root 20 0 0 0 0 S 0.0 0.0 0:31.88 ksoftirqd/0 11 root 20 0 0 0 0 I 0.0 0.0 0:00.00 rcu_bh 12 root rt 0 0 0 0 S 0.0 0.0 0:01.06 migration/0 13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0 14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/1 15 root rt 0 0 0 0 S 0.0 0.0 0:01.17 migration/1 16 root 20 0 0 0 0 S 0.0 0.0 0:31.58 ksoftirqd/1 Edited April 10, 2020 by pyrater Quote Link to comment
Frank1940 Posted April 10, 2020 Share Posted April 10, 2020 Go to Settings >>> Folder Caching Look carefully at the options-- turn on the Help system-- the (?) icon on the toolbar. There are a number of settings to address many specific issues. BTW, Folder Caching seems to work fine for some folks (including myself) but not for many others. I do only one share which has about 1300 items. Remember that it only caches file info. If you are using thumbprints or anything else, it will require disc access to display those. Quote Link to comment
comet424 Posted April 10, 2020 Share Posted April 10, 2020 oh ok ill re look.. all I ever done was load installed. it and set it to enable.. I left it as default I don't use thumbnails I leave wndows file explorer to Details view or List view.. so its just the plain.. but after going into the first folder then it jjust hangs till drive spins up Quote Link to comment
adoucette Posted April 14, 2020 Share Posted April 14, 2020 What have I set wrong with this plugin? I see two issues with my setup currently: 1) I still have to wait for disks to spin up when I'm browsing network shares that should be cached, and 2) the FolderCaching plugin keeps cycling my CPU seems to run for 30 seconds, takes a break, then runs another 30 seconds. I had thought that the plugin allow me to browse the network shares to see folder and files names without spinning up disks unless I actually transfer files and also would use some CPU initially to make a list of the folders and files to cache, and then stop using a lot of system resources until those folders/files are updated. I'm running unraid 6.8.3 and foldercaching 2.2.7. Here is my CPU output from NetData. You can see that I disabled the Folder Caching plugin after 08:44:00 in the graph. As soon as I turn off the plugin, the CPU goes back to normal. Here are my settings for the plugin: I've attached the size of the shares below in case needed. However, I do not have the total number of files in each share. How should I configure Folder Caching to 1) allow me to browse folders and filenames in the included shares without spinning up disks until I need a file and 2) stop using so many CPU resources? Thanks for any assistance. Ari Quote Link to comment
xPliZit_xs Posted July 23, 2020 Share Posted July 23, 2020 Hmm that's some significant system load. I can reproduce this as well. Who would have thought this task was so resource heavy. Quote Link to comment
Alex R. Berg Posted July 24, 2020 Share Posted July 24, 2020 Hi Ari, Here's some help that i wrote in the plugin. ``` Folder Caching Info The folder caching (cache_dirs) reads directories continously to keep them in memory. The program does not know whether a directory is already in memory or need to be reread from disk. Therefore it cannot avoid spinning up disks when it happens that the linux memory cache evicts a directory and cache_dirs rescans the directoy. If a large number of files are being kept in memory then it seems inevitably that some diretories get evicted from the cache when the system is under load. The program by default uses an adaptive mode where it reduces the depth of the directory structure which it tries to keep in memory. When it detects a cache-miss (slow scan time) it will reduce the depth until disks are idle again, but will still be at risk of putting load on the disks for a minute or two until the program gives up and waits for idle disks. The less files the cache needs to hold, the less likely it is that the program spins up disks. ``` It doesn't quite address your issue directly, but consider it spinning up disks, but its part of the same issue, that directories get evicted from memory. Try reducing folders it caches. Don't expect it'll ever be perfect, as its a hack we use, it doesn't tell the OS to never evict dirs. Oh also, quite important, maybe increase the cache-pressure. There's also info on that in the help, by clicking the ? of unraid gui while in folder caching gui Best Alex Quote Link to comment
interwebtech Posted July 25, 2020 Share Posted July 25, 2020 (edited) I'm baaaack. I think I figured out my problem or at least my fix. I set "Minimum level depth (for adaptive depth)" to 2 from the default of 4. I also set "Cache pressure of directories on system" back to 10 from my previous 1 (I think the first one is the one that fixed it tho) Then I let the disks all spin down. Opened the share (from windows pc) with the largest number of folders in it (20k+) and it displayed instantly. It used to take for-freakin-ever to load them all up as it spun up each disk in the share. The thing that finally clicked for me (lightbulb moment) was in the help for "Minimum level depth": Sets the minimum folder level for the adaptive scan (user-share > child folder > grand child is two levels). Default is 4. For some reason I thought my depth was erm... deeper. My shares are pretty basic: Share (e.g., Movies) movie 1 movie 2 movie 3 etc. Question: was the problem related to it trying to cache the file names within the "movie n" folders. If so I have one share with 20K+ folders each with a file in it for 40K right there. I suspect not tho because the issue I had existed before that share grew so large. Edited July 25, 2020 by interwebtech finish the thought Quote Link to comment
interwebtech Posted July 25, 2020 Share Posted July 25, 2020 Disregard the celebration. I just tried to open my TV share and it lit up one drive on the server (activity light) and didn't display anything until it had spun up. Going to try setting "Cache pressure of directories on system:" back to 1. Quote Link to comment
Alex R. Berg Posted July 25, 2020 Share Posted July 25, 2020 (edited) Interwebtech: Darn annoying that it doesn't just work! I doubt that minimum dept should matter, if you go shallower. It does as it says on the box, it always scans each share, but then it does 'find -maxdepth n` where n is your depth. With minimum level n is at least 4. So it'll scan /mnt/disk1/share/X/Y/Z/A - though I might be off by one. I've also noticed lately that my system is slow to display folders which is exactly what cache-dirs should prevent, but I've been ignoring it, not wanting to reopen that can of worms called hacking linux with cache-dirs. I remember people having issues with disks spinning up, but I didn't. And therefore I added the scan of /mnt/user, its in the option panel: `Scan user shares (/mnt/user):` I just tested on my unraid 6.8.3, disks spun up when accessing /mnt/user/media/Film, which it definitely shouldn't. I had scan users folder=false. I now set scan users folder =true. Then set max level to 4, to get a quick test, spun down a disk, and accessed share from windows. Now it does not spin up. adoucette: This does not fix the CPU issues mentioned by adoucette, as that is surely caused by loss of cache and a following rescan. Oh, adoucette, you might want to enable logging, I always have logging enabled, so I can check my tail cache log when I want to verify the behaviour of cach-dirs. alias tailcachelog='tail -n 2000 -f /var/log/cache_dirs.log' though beware that log rotation is needed if keeping logging on always. There are files included but I'm not sure if the logrotation is active. See /etc/logrotate.d PS: I use cache-pressure of 1. I think 0 means never release cache, which might cause out-of-memory. It's system wide, not just cache-dirs. Edited July 25, 2020 by Alex R. Berg Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.