Alex R. Berg

Cache pressure 0 is dangerous. I've used it for some months with my 1.584.929 files as reported in the cache-dirs log and 16 GB memory. I now go for cache-pressure of 1. Works very well, and no crash due to out-of-memory. With zero I think it will never release cached-memory. With cache-pressure of 1 the dirs are almost never lost now it seems, but sometimes cache_dirs decreases depth because the system is under load, but when the load vanishes it scans the increased levels so quickly that I think they were already under caching. It which cache the adaptiveness doesn't matter when it works that well, but now that I have the adaptive feature I kind of prefer it.

Ah I think maybe markswift was unlocky to be super fast to update if it works for alturismo. I pushed the plg-to the github repo, thinking that I had plenty time announcing it. But the new plg in the github repo, contain a link to the package which was not yet released. A chicken and egg-problem of the way I did the release.

Cool, that's what I thought would happen, when I linked the plugin to my github repo. I saw the same thing happen with my 2.2.2 version. @markswift Ah thanks for reporting, I must have messed up something. Let me check again.

I am happy to announce that CacheDirs 2.2.2 is out. I have added the minDepth parameter to the adaptive scanning, so we can limit it its adaptiveness between two set levels min and max. It seems to work, but I may have missed something, so report back if its log says it still goes to level 1 sometimes. Its so cool, I have published on my forked github, and just tested my tower, and it updates the plugin nicely. So if you have installed by 2.2.1 version above you can update via the unRaid Web-GUI > Plugins. Or you can install this new version at the same link: Replace the existing plg in /boot/config/plugins with (just like I mentioned before for 2.2.1) https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg This also means, if we, that mean sme and dynamix/Bergware/bonienl, decides to publish an updated version of the plugin, I can if I so choose push that update to you all via an update on my github fork. Though it seems easier for the time being if we stay at my version for a little bit, to make sure no other things pop up. Best Alex PS: @highdefinitelyI'm really glad it worked for you, and that you appreciate it. You can reenable auto-update I think, but I do think it is wise to do it manually for the next month or so with all the issues there's been, so you know when it updates.

Thanks for helping out sorting all this Fireball. alturismo: It looks good. Regarding plugin: 2018.10.14: That's the broken one. Regarding script version shown on the plugin-page of 'Folder-caching'. This is the version of cache-dirs. I messed up the versioning. The old version in the 2018.10.14 and previous is script version 2.2.0j. I started labeling from 2.2.0a in the above thread in my debugging session with Fireball3 so 2.2.0a - 2.2.0d is NEWER than 2.2.0j. But ignore all that and just go with the new version I linked above on my Fork of dynamix, it should read script version 2.2.1 when installed, just like @alturismo showed.

Ups, forgot attachment as Frank mentioned. Sorry. I havn't read other comments yet. Download this file: https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg Future updates will be posted to this github archive, till Bergware gets his up2date. I suspect that when I push updates to this location, you will automatically get update to your plugin just like dynamix 'official' plugin which I forked. It does also mean if you install my version of the above plugin, I think I can actually automatically update your servers and point you back to Bergwares plugin when his is up2date, if I feel like it. But maybe dis dynamix package is being deprecated because parts are incorporated directly into unRaid, i don't know. Anyway update to above, report back, and stay tuned for future updates. Best Alex

I seem to be getting an overload of info for me to handle here. I will not be offering support on the official plugin of cache-dirs at the moment from dynamix, because its not up2date. I don't want to spend my time helping others going over their logs regarding issues I might already have fixed. I have created (temporary) release of the cache-dirs plugin in my fork of dynamix plugin, the PLG is attached and will download the archive itself. Place the plg in /boot/config/plugins (\\flash\config\plugins) and reboot. It is still the cache_dirs 2.2.1 version, but does add a logrotate script so cache_dirs logging won't waste your mem-mounted root-diskspace too much. I think all problems above are because of lack of scan of user-share. I don't need it to avoid disk spin-up, but others do. 2018.10.14 does not contain this feature, as its based on old script 2.0.0j. @NAS: I suspect your problem is not the adaptiveness, but the lack of scan of user-share, as reported by others users. Check attached release. Also if you don't find love for the new adaptive feature just disable it. I added it because I hated seeing cache_dirs absoluty thrashing my disks when they where otherwise occupied with writing huge files or scanning for md5's, and that was the moment when the linux filesystem disaded to use the file cache for something else than directories causing cache-dirs to thrash the disks. I'm also find sometimes that the adaptiveness does not seem at all perfect. It does seem to work with me with cache-pressure of 1 though and enough memory or few enough files that it works. @NAS I can add a global minimum depth that is adjustable. I already have that in the code, its just not user-modifiable. Actually just checking the code now, it looks like I don't have a minimum depth. Maybe I removed that by mistake. But definitely that is easy and a good idea. I'll add it later. I also thought it would be cool to have filters, but its to difficult to add into the bash-script as cache_dirs is implemented in. It certainly possible and not extremely difficult, as find does support excludes, but it's a pain to work with bash. I've considered re-implementing in scala, but don't feel like it. I have discovered in my process of working with it that its impossible to make it really good because cache_dirs is a hack. It scans the dirs repeatedly in the hope of making linux keep the dirs in memory. Sometimes linux will decide to evict the dirs, and there is no way for us to tell whether linux have evicted the dirs. I try to determine this by checking scan-duration and if its long, I kill the scan procses and back-off, to avoid thrashing the disks, when my system use them for other stuff. But that strategy is never going to be perfect, so I don't feel like messing that much more with it. If you feel like adding it to the script, go for it. Actually its a dead simple scan, so implementing in scala seems super easy, but then people would need to download jvm, and might not want it. It might be helpful, if others can chime in helping out, if I already helped them through some issues. Read further up to see diagnostics check, something about running cache_dirs -L on the new version attached, if my memory serves me. I think it was Fireball3 I helped.

I may offer free unrewarded support on cache_dirs in the cache_dirs thread, but will not be offering support at the moment on the official dynamix plugin, since it's is not up2date. Go here:

Try the new version i posted in the dynamix thread, or wait for 'official' release. It at least fixes the user share, adding it back in, and the start on boot issue.

Thanks for the info. Did you try without cache-dirs plugin, so we know for sure that the broken pipe comes from cache_dirs plugin? Best Alex

Regarding cache_dirs: I pulled the changes from the dynamix github page when I built the plg, and fixed a bug in the 'official' plugin. I don't even know if the broken pipe is coming from cache_dirs. Best way to check that is to remove cache-dirs plg. and reboot and check the log. If its still there I would remove all plugins reboot, and if its not there add half of them back in and reboot thus doing a kind of binary search for the problematic plugin. Since interwebtech also has the broken pipe error you and beg and plead him to do the digging, and see if its the official cache-dirs causing it, and whether my version fixes it. Or wait til you have direct access to machine again.

I don't know if it'll fix that because, I don't know the cause of the broken pipe. It might fix it though, its worth a shot.

I just noticed in linked this thread above. I meant to link the dynamix thread for the discussion of the plugin-error with startup:

I've pushed the new version to the repository. There was a bug in the event-script that I think caused it to fail to start on boot. The new version 2.2.1 also has the option to scan user mount again, which fixed disks spinning up for FIreball3. If you are in a hurry you can use this attached version. Overwrite existing plg in config\plugins and txz in config\plugins\dynamix.cache.dirs and reboot. If you try it, let us know if it causes problems. Best Alex dynamix.cache.dirs.plg dynamix.cache.dirs.txz

It looks good interwebotech, so your problem is not the line i noticed had changed. After I sent the message I noticed I completely had missed pulling all the changes from the release-repository of the plugin. I found the bug causing the plugin not to start, which was surely also Fireballs problem. I didn't have that bug in the plugin I attached above to Fireball, so that's why it suddenly worked, with regards to being started on reboot. I have pushed a change request to the repository, so the new plugin should be released when bergware gets around to it. Best Alex

Ah, thank you for the info. I didn't know that bergware/bonienl was messing with it. His change is in the plugin. Lets take the discussion there, so boniel knows what's going on:

Regarding cache_dirs plugin update Reply to question in other thread : try this and report in this dynamix thread what you see (as root, ie with 'sudo -i') grep -Po "^mdState=\K.*" /proc/mdstat and cat cat /proc/mdstat Probably boniel is on top of it, but I don't know

Are you referring to a test of the plugin I attached 3 posts up, or the one published on dynamix github? Best Alex

I have added a -u option to scan user share, and updated the plugin. Try it out and see if that helps. It shouldn't be necessary to scan the user share, but it seems I'm wrong. Let me know whether or not it makes a difference. Best Alex dynamix.cache.dirs.plg dynamix.cache.dirs.txz

I've just added a -P switch that will do the scan and count files. It can also be used to swiftly check if files are under caching, by running it manually just like the find command I gave. Eg. sudo ./cache_dirs -P -i Video this just gives you number of files. The find command is better at checking which disks aren't cached, becaues you can target it on the disk you choose. cache_dirs

If you in-comment line 601 of /usr/local/bin/cache_dirs (or the version I sent+attached) it becomes a lot more verbose in the log file. Incomment by removing the leading # # (( $DEBUG_THREAD )) && log "scanning $depth_num $dir_to_scan - remaining_time=$remaining_time - pid=$BASHPID" The best way I can think of to check what is cached, is by running find yourself. If it returns immediately it was probably cached, if its slow, it wasn't cached. find /mnt/disk*/share > /dev/null where 'share' is the share you want to check Interestingly one of my shares with 800.000 files is scanned in 1.5 secs when scanning /mnt/disk* but takes 30 secs when scanning /mnt/user/. All with disks spun down. So unRaid is pretty slow at converting user to disk share, but that shouldn't matter. Note I do not scan the user share in the program, only /mnt/cache and /mnt/disk*. It shouldn't matter though, as /mnt/user consist of cache+disk* If you spin down the disks, and run the find on that disk, and it does not spin up, then definitely its cached. If you don't have enough memory and you have many files, it might not be able to cache all. I doubt filesystem matters, but I don't know. I use XFS for my disks and it works well with cache-Dirs. cache_dirs

No its not available yet, I prefer to wait a few weeks to see if it continues to work for you. Maybe I'll commit it one of these days, and post send it along to dynamix to create a new plugin. If you just send him the plg and the txz he can do the same as you, but he can also wait. Best Alex

Great to hear its running now. I guess that means it somehow died with the old version. I just noticed your are only caching your share 'Videos'. The excludes have no effect, because you use include. This is the list of directories maching your include/exclude criteria, as seen in syslog. This is probably what you want, I just wanted to let you know, now that I saw it. Sep 30 23:26:00 Tuerke cache_dirs: ---------- Caching Directories --------------- Sep 30 23:26:00 Tuerke cache_dirs: Video Sep 30 23:26:00 Tuerke cache_dirs: ---------------------------------------------- Best Alex

It is very surprising to me if the 2.2.0a version I sent does not give the same issues when executed without the -T switch. The only change is how I report idle time of disks, I cap the reporting at 9999s when to avoid the crazy numbers you got when all disks where spun down. I wonder it its somehow related to the spun-down situation. I've create a test-plugin for you and added logging during startup. Copy txz to \\tower\flash\config\plugins\dynamix.cache.dirs and replace the plugins\dynamix.cache.dirs.plg (and backup the originals so you can roll back). The updated plugin has the multithreaded param in the plugin-page http://tower/Settings/FolderCachingSettings I've also attached a history of cache-dirs scripts, so you can test them individually for the defunct tests in case you don't have easy access to the cache_dirs via plugins. Best Alex dynamix.cache.dirs.txz dynamix.cache.dirs.plg cache_dirs-scripts.zip

Yeah, looks good so far. You ran the cache_dirs -L diagnostics immediately after starting cache_dirs it seems. That's to early to get much info, because cache_dirs is just started. But anyway, you can report back when you have more details. Best Alex