cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up

[NAS]

Having a hard time learning to love the new adaptive feature.

 

The idea sounds great in practice but I have found that effectively cache dirs doesn't work any more, with all lower level folder access waking up some disks.

 

Certainly this will be user specific but without feedback on how the adaptive caching is working, ideally showing depth per top level folder, timings etc it seems to me people would be better served with a default manual setup.

 

On this note I don't know how other people do this but my ideal use case is to be able to set a global minimum depth and exceptions per folder. Currently I exclude 30% of my array as the data is rarely accessed but very large numbers of files.... however whilst this helps cache dirs a single stray click on one of these top level folders busts the cache and disks spin up.

 

I would love to get into a discussion on how we could enhance this feature as without a word of a lie, it is the most critical one I operate.

[garycase]

1 hour ago, NAS said:

Having a hard time learning to love the new adaptive feature.

 

The idea sounds great in practice but I have found that effectively cache dirs doesn't work any more, with all lower level folder access waking up some disks.

 

Agree … the latest version doesn't seem to be working.    I upgraded my storage server to 6.6.3 and updated to the latest version of Dynamix Cache Directories;  then decided to revert to 6.5.3 due to a massive increase in parity check time (nearly 3 hours) -- but Cache Directories is still on the latest version (2018.10.14).

 

After reading the last couple posts here, I checked to see if it's working -- I have it set to only cache ONE of my shares ("Backups"), but if I access that share ALL of the disks spin up and it takes several seconds to read and display the info.

 

Prior to the upgrade, accessing that share was virtually instant, as the directory info was cached.

[craigr]

Sorry if this has been covered, but I searched and can't find the answer.  In Folder Caching I cannot change any of the following parameters:

Disk Idle Timer (sec): 60
Scan timeout for a scan during disk-idle period (sec):150
Scan timeout for a scan during initial startup or less-idle period (sec):30
Scan timeout for a scan when all directories have been succesfully cached (sec):20
Minimum interval between folder scans (sec):1
Maximum interval between folder scans (sec):10
Cache pressure of directories on system:10
Limit the memory available to the shell in which the script runs (increase if it dies):50000

I've had used cache_dirs for some time before it was renamed Folder Caching, but for a long time now I have had trouble with discs spinning up when the directories I am browsing should be cached or even all the discs spinning up when a user share uses all of the discs.  Discs are constantly being spun up when file browsing and in the past I didn't have this issue.  I would like to try and tweak the settings, but can't change any of the values.  I've tried disabling Folder Caching and then changing, but no joy there either.  This has to be something simple right?

 

Thanks

craigr

Edited October 26, 2018 by craigr

[craigr]

18 minutes ago, craigr said:

Sorry if this has been covered, but I searched and can't find the answer.  In Folder Caching I cannot change any of the following parameters:

Disk Idle Timer (sec): 60
Scan timeout for a scan during disk-idle period (sec):150
Scan timeout for a scan during initial startup or less-idle period (sec):30
Scan timeout for a scan when all directories have been succesfully cached (sec):20
Minimum interval between folder scans (sec):1
Maximum interval between folder scans (sec):10
Cache pressure of directories on system:10
Limit the memory available to the shell in which the script runs (increase if it dies):50000

I've had used cache_dirs for some time before it was renamed Folder Caching, but for a long time now I have had trouble with discs spinning up when the directories I am browsing should be cached or even all the discs spinning up when a user share uses all of the discs.  Discs are constantly being spun up when file browsing and in the past I didn't have this issue.  I would like to try and tweak the settings, but can't change any of the values.  I've tried disabling Folder Caching and then changing, but no joy there either.  This has to be something simple right?

 

Thanks

craigr

Never mind.  Uninstalled and reinstalled and now it works.

 

craigr 

[NAS]

Something weird is going on. I can see cache dirs running and even though I now have

 

/bin/bash /usr/local/emhttp/plugins/dynamix.cache.dirs/scripts/cache_dirs -e appdata -l off -p 0 -D 9999

 

the cache seems to be perpetually needing disks spun up.

 

How would you like me to debug this further?

 

Update: it is this obvious, have all my cache includes been removed so I end with exclusions only and no inclusions  or is there there an implied * include?

[Frank1940]

The observation that I made here might shed some additional light on what is going with the Dynamix ch ache folders plugin.

     

 

 

Not sure how this observation fits into the larger picture of what is happening with the script but thought it might provide some useful information...

[Alex R. Berg]

I seem to be getting an overload of info for me to handle here. I will not be offering support on the official plugin of cache-dirs at the moment from dynamix, because its not up2date. I don't want to spend my time helping others going over their logs regarding issues I might already have fixed.

 

I have created (temporary) release of the cache-dirs plugin in my fork of dynamix plugin, the PLG is attached and will download the archive itself. Place the plg in /boot/config/plugins (\\flash\config\plugins) and reboot. It is still the cache_dirs 2.2.1 version, but does add a logrotate script so cache_dirs logging won't waste your mem-mounted root-diskspace too much.

I think all problems above are because of lack of scan of user-share. I don't need it to avoid disk spin-up, but others do. 2018.10.14 does not contain this feature, as its based on old script 2.0.0j.

@NAS: I suspect your problem is not the adaptiveness, but the lack of scan of user-share, as reported by others users. Check attached release. Also if you don't find love for the new adaptive feature just disable it. I added it because I hated seeing cache_dirs absoluty thrashing my disks when they where otherwise occupied with writing huge files or scanning for md5's, and that was the moment when the linux filesystem disaded to use the file cache for something else than directories causing cache-dirs to thrash the disks. I'm also find sometimes that the adaptiveness does not seem at all perfect. It does seem to work with me with cache-pressure of 1 though and enough memory or few enough files that it works.

 

@NAS I can add a global minimum depth that is adjustable. I already have that in the code, its just not user-modifiable. Actually just checking the code now, it looks like I don't have a minimum depth. Maybe I removed that by mistake. But definitely that is easy and a good idea. I'll add it later. I also thought it would be cool to have filters, but its to difficult to add into the bash-script as cache_dirs is implemented in. It certainly possible and not extremely difficult, as find does support excludes, but it's a pain to work with bash. I've considered re-implementing in scala, but don't feel like it. I have discovered in my process of working with it that its impossible to make it really good because cache_dirs is a hack. It scans the dirs repeatedly in the hope of making linux keep the dirs in memory. Sometimes linux will decide to evict the dirs, and there is no way for us to tell whether linux have evicted the dirs. I try to determine this by checking scan-duration and if its long, I kill the scan procses and back-off, to avoid thrashing the disks, when my system use them for other stuff. But that strategy is never going to be perfect, so I don't feel like messing that much more with it. If you feel like adding it to the script, go for it. Actually its a dead simple scan, so implementing in scala seems super easy, but then people would need to download jvm, and might not want it. 

 

It might be helpful, if others can chime in helping out, if I already helped them through some issues. Read further up to see diagnostics check, something about running cache_dirs -L on the new version attached, if my memory serves me. I think it was Fireball3 I helped.

[Frank1940]

44 minutes ago, Alex R. Berg said:

Read further up to see diagnostics check, something about running cache_dirs -L on the new version attached,

I don't see any attachment...  

[Fireball3]

4 hours ago, Alex R. Berg said:

It might be helpful, if others can chime in helping out, if I already helped them through some issues. Read further up to see diagnostics check, something about running cache_dirs -L on the new version attached, if my memory serves me. I think it was Fireball3 I helped.

Yes, it was me. Thanks for your help - the issue is solved in my opinion.

 

Here is the post containing the files.

Copy txz to \\tower\flash\config\plugins\dynamix.cache.dirs 

and replace the plugins\dynamix.cache.dirs.plg

Rename your existing files for backup purposes.

 

There is much confusion about the dynamix plugin frontend, and the underlying shell-script.

I wish the dynamix plugin would be updated asap. Apparently more and more people face this issue.

Edited October 29, 2018 by Fireball3

[highdefinitely]

I am experiencing the issue as well, especially since updating to V6.6.3. 

 

Please, for the love of everything that is holy, can somebody be so kind and fix this plugin? Because, and I'm not exaggerating when I'm saying this, unRAID is close to being unusable without it working properly, especially if you have two dozen disks in your system(s).... 

[Frank1940]

1 hour ago, Fireball3 said:

 

Here is the post containing the files.

Copy txz to \\tower\flash\config\plugins\dynamix.cache.dirs 

and replace the plugins\dynamix.cache.dirs.plg

Rename your existing files for backup purposes.

 

 

6 hours ago, Alex R. Berg said:

I have created (temporary) release of the cache-dirs plugin in my fork of dynamix plugin, the PLG is attached and will download the archive itself. Place the plg in /boot/config/plugins (\\flash\config\plugins) and reboot. It is still the cache_dirs 2.2.1 version, but does add a logrotate script so cache_dirs logging won't waste your mem-mounted root-diskspace too much.

@Alex R. Berg, @Fireball3, I have cache_dirs version: 2.2.0d shown as running.  There appears to be some discrepancy between the version that I have and what @Alex R. Berg indicated I should have.   I followed the instructions in @Fireball3's post.  Is this the current version?  

[wgstarks]

42 minutes ago, Frank1940 said:

 

@Alex R. Berg, @Fireball3, I have cache_dirs version: 2.2.0d shown as running.  There appears to be some discrepancy between the version that I have and what @Alex R. Berg indicated I should have.   I followed the instructions in @Fireball3's post.  Is this the current version?  

I asked about that a few posts back. It’s my understanding that (at least in my case) I have the official version which was installed through CA and the link that @Alex R. Berg has posted is for a branch.

[Frank1940]

7 minutes ago, wgstarks said:

I asked about that a few posts back. It’s my understanding that (at least in my case) I have the official version which was installed through CA and the link that @Alex R. Berg has posted is for a branch.

So what is the version number that you are running?  @Alex R. Berg says that he has a version 2.2.1 available (I assume that is the version number of the script that does that actual work!)  and I followed @Fireball3's directions to install that version in my post and I currently have 2.2.0d installed.  (Applicable quotes in my post that you quoted me from for both of these posts.)   

[garycase]

FWIW one of my systems is running a version identified as version 2018.10.14 on the Plugin page but as 2.2.0j on the Settings page.

 

My other systems are using the version shown as 2016.08.26 and as 2.1.1 on the Settings page.

 

I've held off on updating the older versions since the newest one seems to have some issues.   I assume that at some point in the not-too-distant future the plugin will update to 2.2.1 (or later) with the built-in "Check for Updates" feature on the Plugin page, and that will resolve the current issues.

 

[craigr]

6 hours ago, highdefinitely said:

I am experiencing the issue as well, especially since updating to V6.6.3. 

 

Please, for the love of everything that is holy, can somebody be so kind and fix this plugin? Because, and I'm not exaggerating when I'm saying this, unRAID is close to being unusable without it working properly, especially if you have two dozen disks in your system(s).... 

+100

 

Thanks to all who are helping and to all who have helped in the past.  For now I have removed the script because it's working so poorly for me.  But I agree that it is essential for unRAID.

 

Kind regards,

 

craigr

Edited October 30, 2018 by craigr

[garycase]

6 hours ago, highdefinitely said:

… unRAID is close to being unusable without it working properly, especially if you have two dozen disks in your system(s).... 

It can certainly slow things down if you're constantly waiting for spin-ups to access directories.    One way to mitigate this is to set your disk spin-down timer to a value that will eliminate most spin-downs while you're actively using the system (e.g. 3 or 4 hours); and then click on Spin-Up when you're going to be using the system a lot.   If you do this, you might also want to set the md_write_method to "reconstruct write", which will makes writes appreciably faster (but this setting requires all disks be spinning for a write).

 

[craigr]

7 hours ago, Fireball3 said:

Yes, it was me. Thanks for your help - the issue is solved in my opinion.

 

Here is the post containing the files.

Copy txz to \\tower\flash\config\plugins\dynamix.cache.dirs 

and replace the plugins\dynamix.cache.dirs.plg

Rename your existing files for backup purposes.

 

There is much confusion about the dynamix plugin frontend, and the underlying shell-script.

I wish the dynamix plugin would be updated asap. Apparently more and more people face this issue.

So did putting -L in the "User defined options:" along with using the txz and plg solve your issues?

 

Does -U or -u still work for caching user shares?

 

I'm running preclear on two 8TB drives so it will be a day and a half before I can try it  because I think I need to reboot?  I did copy the txz and plg along with adding -L to "User defined options:"  but cache_dirs says it is not running.  It shows version: 2.2.0j in settings and version 

2018.09.30 in Plugins.  Plugins shows a newer release but I'm assuming this is not pertinent after overwriting the txz and plg?

 

Thanks

craigr

Edited October 30, 2018 by craigr

[alturismo]

Hi, seems here came an update for cache dirs (temp fix), may a question.

 

does it need -L as user option in fromtend ? and is a reboot required ?

 

or may any way to check if its really running ?

 

i enabled now the include user shares, when i started now the plugin all disks spinned up wich is probably correct for scanning ?

 

and some cpu load for a short time ...

 

thanks ahead for an advice

[Fireball3]

Guys, why don't you give it a try? After replacing the files you need to reboot - that's right.

The script version is 2.2.0d. I'm not sure what's displayed in the GUI - need to check when I'm home.

2018.09.30 should be the right version of the dynamix plugin shown in the GUI. If the plugins page shows a newer version, ignore it!

It might bring some GUI changes, but the script file 2.2.0d is not included as of today.

Once you're on the arberg fork of the plugin, you can update from GUI as usual.

There is a check box to enable scanning of user shares. No need to use additional user options.

#	- Added disabling of multithreaded scan of disks flag -T
#	- Added diagnostics generate zip, flag -L (usage: cache_dirs -L)
#	- Added -P print file count flag (usage: cache_dirs -P -i your_share_name)

 

14 hours ago, Frank1940 said:

I followed @Fireball3's directions to install that version in my post and I currently have 2.2.0d installed. 

I'm not sure what's your situation. Given, you are using the posted files you might face some other issue.

Then please go back some posts and try the troubleshooting instructions that @Alex R. Berg posted for me. If still buggy post the diagnostics here for Alex so he can check into it.

Edited October 30, 2018 by Fireball3

[alturismo]

as of update today the script version is 2.2.1, i make a reboot now, thanks for the note.

 

[Frank1940]

3 hours ago, Fireball3 said:

Guys, why don't you give it a try? After replacing the files you need to reboot - that's right.

The script version is 2.2.0d. I'm not sure what's displayed in the GUI - need to check when I'm home.

2018.09.30 should be the right version of the dynamix plugin shown in the GUI. If the plugins page shows a newer version, ignore it!

It might bring some GUI changes, but the script file 2.2.0d is not included as of today.

There is a check box to enable scanning of user shares. No need to use additional user options.

#	- Added disabling of multithreaded scan of disks flag -T
#	- Added diagnostics generate zip, flag -L (usage: cache_dirs -L)
#	- Added -P print file count flag (usage: cache_dirs -P -i your_share_name)

 

I'm not sure what's your situation. Given, you are using the posted files you might face some other issue.

Then please go back some posts and try the troubleshooting instructions that @Alex R. Berg posted for me. If still buggy post the diagnostics here for Alex so he can check into it.

As I said the version of the script is 2.2.0d and I used the files that you pointed to.  I turned on the 'Include user share (/mnt/user) in scan:' and it seems to work as of last night and a quick test shows it is still working this morning.  I also added back the caching of only the user shares that I normally would access with the 'Included folders' option.

 

I also see the arberg update on "Plugin" tab of this server (my main Media server) that but not on the other server (my Testbed server) which is currently on the 'standard' release path.  Strange...   Not sure what is going on there.  With things working at this time, I am not going to disturb (by 'updating') my Media server (WAF type) at this time.  I will wait for the dust to settle a bit...

 

I usually do most of my testing on the Testbed server but the issues with cache directories were so bad on the Media server that I had to do something/anything to resolve them!

[Fireball3]

14 minutes ago, Frank1940 said:

I also see the arberg update on "Plugin" tab of this server (my main Media server) that but not on the other server (my Testbed server) which is currently on the 'standard' release path.  Strange...   Not sure what is going on there.

 

Quoting @Squid

 

Quote

 

That .plg isn't the "official" one from bonienl.  You won't see an update.  If you want to use it, then you would uninstall the existing one, save that .plg to the flash and then install plugin and browse to the location and select it.  But, if/when bonienl releases the update to the "official" version of the plugin, then you won't see that as that's not the ver you have installed.

 

Or over write the .plg on the flash and .txz with that ver, but then you're in the same boat again.  Once the "official ver" is released, it won't see the updates.

 

This means, the @Alex R. Berg - "fork" of the plugin is not checking the @bonienl repository for updates and vice versa.

If my understanding ist correct, then it should be safe to update the plugin if it checks out from the arberg repository.

 

 

[Alex R. Berg]

Ups, forgot attachment as Frank mentioned. Sorry. I havn't read other comments yet.

 

Download this file: https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg

 

Future updates will be posted to this github archive, till Bergware gets his up2date. I suspect that when I push updates to this location, you will automatically get update to your plugin just like dynamix 'official' plugin which I forked. It does also mean if you install my version of the above plugin, I think I can actually automatically update your servers and point you back to Bergwares plugin when his is up2date, if I feel like it. But maybe dis dynamix package is being deprecated because parts are incorporated directly into unRaid, i don't know. Anyway update to above, report back, and stay tuned for future updates.

 

Best Alex

[Alex R. Berg]

Thanks for helping out sorting all this Fireball.

 

alturismo: It looks good.

 

Regarding plugin: 2018.10.14: That's the broken one. 

 

Regarding script version shown on the plugin-page of 'Folder-caching'. This is the version of cache-dirs. I messed up the versioning. The old version in the 2018.10.14 and previous is script version 2.2.0j. I started labeling from 2.2.0a in the above thread in my debugging session with Fireball3 so 2.2.0a - 2.2.0d is NEWER than 2.2.0j. But ignore all that and just go with the new version I linked above on my Fork of dynamix, it should read script version 2.2.1 when installed, just like @alturismo showed.

[highdefinitely]

5 hours ago, Fireball3 said:

Guys, why don't you give it a try?

I followed your instructions and it seems to work now. Thank you and a big thanks to Alex R. Berg. 

 

I also disabled auto update for the Dynamix Cache Dirs plugin for the time being. While it's working now, it still would be great to have an "official" solution to these problems with cache dirs, as I think it's a very essential part of making the unRAID experience enjoyable. So essential that it probably makes sense to fully integrate it into the unRAID code.