May 23, 20197 yr cache_dirs appears to not be working properly with the latest 6.7 stable. Opening a cached share in windows exploder almost always lights up a drive on the array which delays the display of folders until its spun up. I've tried adjusting "Cache pressure of directories on system" from 10 to 1 but no change in behavior. logs attached tower-diagnostics-20190523-1807.zip
May 25, 20197 yr I'm new with this plugin and just set it up. Is it possible that using this plugin will prevent dockers to spin disks randomly? I'm having a bit of an issue with the Transmission docker.
May 27, 20197 yr On 5/24/2019 at 9:27 PM, gacpac said: I'm new with this plugin and just set it up. Is it possible that using this plugin will prevent dockers to spin disks randomly? I'm having a bit of an issue with the Transmission docker. Go to User Shares, Compute All, post screenshot.
May 27, 20197 yr Go to User Shares, Compute All, post screenshot.Sorry for the late reply. I fixed itSent from my Pixel 2 XL using Tapatalk
May 28, 20197 yr On 5/23/2019 at 11:13 AM, interwebtech said: cache_dirs appears to not be working properly with the latest 6.7 stable. Opening a cached share in windows exploder almost always lights up a drive on the array which delays the display of folders until its spun up. I've tried adjusting "Cache pressure of directories on system" from 10 to 1 but no change in behavior. logs attached tower-diagnostics-20190523-1807.zipUnavailable i'm not using windows, however i'm also seeing drives spin up when i would not expect them to. in your case i'd guess Explorer is trying to read some file metadata (thumbnails for picture files, etc) which cache_dirs cannot help with. in my case i _suspect_ it's sonarr/radarr running in Docker containers doing the same thing as the spin-ups appear to coincide with either app starting a library scan, as well as the Settings > Media Management > File Management > Analyze Video Files option being enabled... hopefully i'll have time to debug further soon. edit: duplicate episodes/movies were my issue - on each scan the duplicate was found and it was scanned for metadata during the process to compare it with the existing episode/movie to see if it should be replaced. once the dupes were removed there was no comparison to be made and the disks do not spin up. Edited May 28, 20197 yr by harleywastaken additional info
May 28, 20197 yr How can I test if the cache plugin is working as supposed to?If the drive is not spinning and the cache is working. I should be able to browse my shares. At least the top folder, right? Sent from my Pixel 2 XL using Tapatalk
May 29, 20197 yr 5 hours ago, gacpac said: If the drive is not spinning and the cache is working. I should be able to browse my shares. At least the top folder, right? Depends on your definition of browse. If you mean list the file names, yes.
June 23, 20197 yr On 5/23/2019 at 8:13 PM, interwebtech said: cache_dirs appears to not be working properly with the latest 6.7 stable. Opening a cached share in windows exploder almost always lights up a drive on the array which delays the display of folders until its spun up. I've tried adjusting "Cache pressure of directories on system" from 10 to 1 but no change in behavior. logs attached tower-diagnostics-20190523-1807.zip 159.48 kB · 1 download Its working for me on unRaid 6.7.0. I tried accessing a folder on my disk2 both from windows, unRaid itself, in both cases both user and disk share. Disk didn't spin up. I'm filled up at the moment, so I probably don't investigate much, but if you want to supply logs, if it is to be useful, the logs from cache-dirs are needed Cache_dirs has a command to collect them. It would be awesome it that could somehow be collected automatically by unRaid, for instance by some kind of plugin-event. I assume that is not possible, but if it is, let me know. If not, it might make sense to ask Tom @ Limetech if that is a reasonable feature to add. Maybe you feel like doing that interwebtech, or tell me where to post? Best Alex
June 24, 20197 yr Before updating to 6.7.1 I run the update assistant and I get the following: Checking for plugin compatibility Issue Found: dynamix.cache.dirs.plg is not known to Community Applications. Compatibility for this plugin CANNOT be determined and it may cause you issues. I have the latest cache dirs version (2018.12.04). Does anyone have problems with cache dirs and 6.7.1? Edited June 24, 20197 yr by papnikol
June 24, 20197 yr Hmm, its just an annoying warning, putting a scare into users. Unfortunately some GitHub merge stuff went badly between my developing fork and the Community app version by Bergware. Hence the warning, I think. If asked Bergware if he'll fix it, or if perhaps we should switch community apps to this repo. It works fine on unRaid 6.7.1
June 25, 20197 yr 9 hours ago, Alex R. Berg said: Hmm, its just an annoying warning, putting a scare into users. Unfortunately some GitHub merge stuff went badly between my developing fork and the Community app version by Bergware. Hence the warning, I think. If asked Bergware if he'll fix it, or if perhaps we should switch community apps to this repo. It works fine on unRaid 6.7.1 Thanks for your response, I though so, I just wanted to double check
December 15, 20196 yr hi I have this cahce dir installed and left the defaults... is it supposed ot load entire hard drives directories into ram? If so I find it not working.. it will do the first Folder listtins so what I mean is \\tower\ it will show those file folders if I click into one its fast but to go into anything after that is slow as it waits for the drives to boot up to see whats in the directories.. whats the proper format to load everything and how much ram you need
April 10, 20206 yr Is there anyway to trouble shoot this plugin? I have it running but none of my drives spun down after 3 hours. The only thing being accessed is my SSD / Cache pool for plex. Edited April 10, 20206 yr by pyrater
April 10, 20206 yr How much are you caching and how much memory is installed. I seem to recall that as memory is filled with cached data, It will discard the earlier stuff to store the later data. So you can get caught in a round-robin. Try caching only the shares (and depth) you need at a regular basis to see if that helps.
April 10, 20206 yr how do you guys cache more then 1 folder the main shares if you click on them you see the folders inside them quickly but when you go it another folder its not cacheced so example \\tower Music country hip hop so Music Folder is cached it seems but not country or hip hop as example how do you cache the entire file directories? and im using 32gb of ram or do I need more.. as its not using it..
April 10, 20206 yr 24 minutes ago, Frank1940 said: How much are you caching and how much memory is installed. I seem to recall that as memory is filled with cached data, It will discard the earlier stuff to store the later data. So you can get caught in a round-robin. Try caching only the shares (and depth) you need at a regular basis to see if that helps. .....Frank that may very well be it.... I am caching 83 TBs on 16GB of memory with 8GB of that dedicated to a plex transcode ramdisk. Is it possible to check this? top - 12:01:28 up 2 days, 22:15, 3 users, load average: 6.91, 6.34, 4.82 Tasks: 388 total, 2 running, 386 sleeping, 0 stopped, 0 zombie %Cpu(s): 90.3 us, 2.1 sy, 4.2 ni, 3.4 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st MiB Mem : 15732.1 total, 156.5 free, 5446.3 used, 10129.3 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 6932.5 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11058 nobody 20 0 1195292 359328 18088 R 370.0 2.2 23:59.63 Plex Transcoder 6044 nobody 20 0 17932 1444 672 S 2.3 0.0 1:08.15 EasyAudioEncode 5440 nobody 20 0 753832 50128 9584 S 1.7 0.3 4:13.90 Plex Transcoder 4001 nobody 20 0 2779608 1.4g 14600 S 1.3 9.1 37:09.63 Plex Media Serv 32029 root 20 0 839180 31592 17996 S 1.3 0.2 54:03.57 containerd 24305 root 20 0 1596456 348108 992 S 1.0 2.2 2370:41 shfs 11340 root 20 0 152380 33148 27360 S 0.7 0.2 0:00.02 docker 32013 root 20 0 1260828 73996 43840 S 0.7 0.5 34:43.80 dockerd 10 root 20 0 0 0 0 I 0.3 0.0 2:21.84 rcu_sched 2465 root 20 0 109104 7948 5184 S 0.3 0.0 5:00.19 containerd-shim 2610 root 20 0 109104 9960 4864 S 0.3 0.1 0:56.35 containerd-shim 3305 root 20 0 109104 9960 4800 S 0.3 0.1 1:01.88 containerd-shim 8343 root 20 0 105276 13532 7504 S 0.3 0.1 0:00.03 php-fpm 11292 root 20 0 6724 3204 2432 R 0.3 0.0 0:00.01 top 23070 root 20 0 349176 4260 3404 S 0.3 0.0 8:11.76 emhttpd 23836 root 20 0 0 0 0 S 0.3 0.0 2:05.27 unraidd1 23951 root 20 0 3792 2736 2464 S 0.3 0.0 0:47.52 diskload 31975 root 0 -20 0 0 0 S 0.3 0.0 8:15.51 loop2 32727 root 20 0 30276 17024 1988 S 0.3 0.1 0:24.80 supervisord 1 root 20 0 2468 1732 1620 S 0.0 0.0 0:24.17 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.04 kthreadd 3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp 6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H-kblockd 8 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq 9 root 20 0 0 0 0 S 0.0 0.0 0:31.88 ksoftirqd/0 11 root 20 0 0 0 0 I 0.0 0.0 0:00.00 rcu_bh 12 root rt 0 0 0 0 S 0.0 0.0 0:01.06 migration/0 13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0 14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/1 15 root rt 0 0 0 0 S 0.0 0.0 0:01.17 migration/1 16 root 20 0 0 0 0 S 0.0 0.0 0:31.58 ksoftirqd/1 Edited April 10, 20206 yr by pyrater
April 10, 20206 yr Go to Settings >>> Folder Caching Look carefully at the options-- turn on the Help system-- the (?) icon on the toolbar. There are a number of settings to address many specific issues. BTW, Folder Caching seems to work fine for some folks (including myself) but not for many others. I do only one share which has about 1300 items. Remember that it only caches file info. If you are using thumbprints or anything else, it will require disc access to display those.
April 10, 20206 yr oh ok ill re look.. all I ever done was load installed. it and set it to enable.. I left it as default I don't use thumbnails I leave wndows file explorer to Details view or List view.. so its just the plain.. but after going into the first folder then it jjust hangs till drive spins up
April 14, 20206 yr What have I set wrong with this plugin? I see two issues with my setup currently: 1) I still have to wait for disks to spin up when I'm browsing network shares that should be cached, and 2) the FolderCaching plugin keeps cycling my CPU seems to run for 30 seconds, takes a break, then runs another 30 seconds. I had thought that the plugin allow me to browse the network shares to see folder and files names without spinning up disks unless I actually transfer files and also would use some CPU initially to make a list of the folders and files to cache, and then stop using a lot of system resources until those folders/files are updated. I'm running unraid 6.8.3 and foldercaching 2.2.7. Here is my CPU output from NetData. You can see that I disabled the Folder Caching plugin after 08:44:00 in the graph. As soon as I turn off the plugin, the CPU goes back to normal. Here are my settings for the plugin: I've attached the size of the shares below in case needed. However, I do not have the total number of files in each share. How should I configure Folder Caching to 1) allow me to browse folders and filenames in the included shares without spinning up disks until I need a file and 2) stop using so many CPU resources? Thanks for any assistance. Ari
July 23, 20205 yr Hmm that's some significant system load. I can reproduce this as well. Who would have thought this task was so resource heavy.
July 24, 20205 yr Hi Ari, Here's some help that i wrote in the plugin. ``` Folder Caching Info The folder caching (cache_dirs) reads directories continously to keep them in memory. The program does not know whether a directory is already in memory or need to be reread from disk. Therefore it cannot avoid spinning up disks when it happens that the linux memory cache evicts a directory and cache_dirs rescans the directoy. If a large number of files are being kept in memory then it seems inevitably that some diretories get evicted from the cache when the system is under load. The program by default uses an adaptive mode where it reduces the depth of the directory structure which it tries to keep in memory. When it detects a cache-miss (slow scan time) it will reduce the depth until disks are idle again, but will still be at risk of putting load on the disks for a minute or two until the program gives up and waits for idle disks. The less files the cache needs to hold, the less likely it is that the program spins up disks. ``` It doesn't quite address your issue directly, but consider it spinning up disks, but its part of the same issue, that directories get evicted from memory. Try reducing folders it caches. Don't expect it'll ever be perfect, as its a hack we use, it doesn't tell the OS to never evict dirs. Oh also, quite important, maybe increase the cache-pressure. There's also info on that in the help, by clicking the ? of unraid gui while in folder caching gui Best Alex
July 25, 20205 yr I'm baaaack. I think I figured out my problem or at least my fix. I set "Minimum level depth (for adaptive depth)" to 2 from the default of 4. I also set "Cache pressure of directories on system" back to 10 from my previous 1 (I think the first one is the one that fixed it tho) Then I let the disks all spin down. Opened the share (from windows pc) with the largest number of folders in it (20k+) and it displayed instantly. It used to take for-freakin-ever to load them all up as it spun up each disk in the share. The thing that finally clicked for me (lightbulb moment) was in the help for "Minimum level depth": Sets the minimum folder level for the adaptive scan (user-share > child folder > grand child is two levels). Default is 4. For some reason I thought my depth was erm... deeper. My shares are pretty basic: Share (e.g., Movies) movie 1 movie 2 movie 3 etc. Question: was the problem related to it trying to cache the file names within the "movie n" folders. If so I have one share with 20K+ folders each with a file in it for 40K right there. I suspect not tho because the issue I had existed before that share grew so large. Edited July 25, 20205 yr by interwebtech finish the thought
July 25, 20205 yr Disregard the celebration. I just tried to open my TV share and it lit up one drive on the server (activity light) and didn't display anything until it had spun up. Going to try setting "Cache pressure of directories on system:" back to 1.
July 25, 20205 yr Interwebtech: Darn annoying that it doesn't just work! I doubt that minimum dept should matter, if you go shallower. It does as it says on the box, it always scans each share, but then it does 'find -maxdepth n` where n is your depth. With minimum level n is at least 4. So it'll scan /mnt/disk1/share/X/Y/Z/A - though I might be off by one. I've also noticed lately that my system is slow to display folders which is exactly what cache-dirs should prevent, but I've been ignoring it, not wanting to reopen that can of worms called hacking linux with cache-dirs. I remember people having issues with disks spinning up, but I didn't. And therefore I added the scan of /mnt/user, its in the option panel: `Scan user shares (/mnt/user):` I just tested on my unraid 6.8.3, disks spun up when accessing /mnt/user/media/Film, which it definitely shouldn't. I had scan users folder=false. I now set scan users folder =true. Then set max level to 4, to get a quick test, spun down a disk, and accessed share from windows. Now it does not spin up. adoucette: This does not fix the CPU issues mentioned by adoucette, as that is surely caused by loss of cache and a following rescan. Oh, adoucette, you might want to enable logging, I always have logging enabled, so I can check my tail cache log when I want to verify the behaviour of cach-dirs. alias tailcachelog='tail -n 2000 -f /var/log/cache_dirs.log' though beware that log rotation is needed if keeping logging on always. There are files included but I'm not sure if the logrotation is active. See /etc/logrotate.d PS: I use cache-pressure of 1. I think 0 means never release cache, which might cause out-of-memory. It's system wide, not just cache-dirs. Edited July 25, 20205 yr by Alex R. Berg
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.