cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up


Recommended Posts

5 minutes ago, highdefinitely said:

I also disabled auto update for the Dynamix Cache Dirs plugin for the time being. While it's working now, it still would be great to have an "official" solution to these problems with cache dirs, as I think it's a very essential part of making the unRAID experience enjoyable.

See both posts above yours.

No need to disable auto update. Consider the arberg repository as the "official" solution as long Alex is maintaining the plugin and script.

 

8 minutes ago, highdefinitely said:

So essential that it probably makes sense to fully integrate it into the unRAID code.

I'm not sure if @limetech considers this "hack" as a feature to be in the core unRAID. But I agree, without this, the spin-down benefits are killing usability.

Link to comment
17 minutes ago, Alex R. Berg said:

suspect that when I push updates to this location, you will automatically get update to your plugin just like dynamix 'official' plugin which I forked.

This is true. I installed the 2.2.0d version which you had linked a few pages back and saw this morning that an update was available in my plugins tab. After running the update I now have 2.2.1 installed. Looks like the cache_dirs plugin should automatically continue with your branch when you push updates.

Link to comment

I am happy to announce that CacheDirs 2.2.2 is out. I have added the minDepth parameter to the adaptive scanning, so we can limit it its adaptiveness between two set levels min and max. It seems to work, but I may have missed something, so report back if its log says it still goes to level 1 sometimes.

 

Its so cool, I have published on my forked github, and just tested my tower, and it updates the plugin nicely. So if you have installed by 2.2.1 version above you can update via the unRaid Web-GUI > Plugins. Or you can install this new version at the same link: Replace the existing plg in /boot/config/plugins with (just like I mentioned before for 2.2.1)

 

https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg

 

This also means, if we, that mean sme and dynamix/Bergware/bonienl, decides to publish an updated version of the plugin, I can if I so choose push that update to you all via an update on my github fork. Though it seems easier for the time being if we stay at my version for a little bit, to make sure no other things pop up.

 

Best Alex

 

PS: @highdefinitelyI'm really glad it worked for you, and that you appreciate it. You can reenable auto-update I think, but I do think it is wise to do it manually for the next month or so with all the issues there's been, so you know when it updates.

Edited by Alex R. Berg
Link to comment
4 minutes ago, Alex R. Berg said:

I am happy to announce that CacheDirs 2.2.2 is out. I have added the minDepth parameter to the adaptive scanning, so we can limit it its adaptiveness between two set levels min and max. It seems to work, but I may have missed something, so report back if its log says it still goes to level 1 sometimes.

 

Its so cool, I have published on my forked github, and just tested my tower, and it updates the plugin nicely. So if you have installed by 2.2.1 version above you can update via the unRaid Web-GUI > Plugins. Or you can install this new version at the same link: Replace the existing plg in /boot/config/plugins with (just like I mentioned before for 2.2.1)

 

https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg

 

This also means, if we, that mean sme and dynamix/Bergware/bonienl, decides to publish an updated version of the plugin, I can if I so choose push that update to you all via an update on my github fork. Though it seems easier for the time being if we stay at my version for a little bit, to make sure no other things pop up.

 

Best Alex

 

PS: @highdefinitelyI'm really glad it worked for you, and that you appreciate it. You can reenable auto-update I think, but I do think it is wise to do it manually for the next month or so with all the issues there's been, so you know when it updates.

 

Hi Alex, tried to install:

 

plugin: installing: https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg
plugin: downloading https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg
plugin: downloading: https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg ... done
plugin: file doesn't exist or xml parse error

Edited by markswift
Link to comment
2 minutes ago, markswift said:

Oct 30 21:05:18 AlsServer emhttpd: req (8): cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=update&arg2=dynamix.cache.dirs.plg&csrf_token=****************
Oct 30 21:05:18 AlsServer emhttpd: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin update dynamix.cache.dirs.plg
Oct 30 21:05:18 AlsServer root: plugin: running: anonymous
Oct 30 21:05:18 AlsServer cache_dirs: Stopping cache_dirs process 8271
Oct 30 21:05:19 AlsServer cache_dirs: cache_dirs service rc.cachedirs: Stopped
Oct 30 21:05:19 AlsServer root: plugin: creating: /boot/config/plugins/dynamix.cache.dirs/dynamix.cache.dirs.txz - downloading from URL https://github.com/arberg/dynamix/releases/download/cache_dirs_2.2.2/dynamix.cache.dirs.txz
Oct 30 21:05:20 AlsServer root: plugin: checking: /boot/config/plugins/dynamix.cache.dirs/dynamix.cache.dirs.txz - MD5
Oct 30 21:05:20 AlsServer root: plugin: running: /boot/config/plugins/dynamix.cache.dirs/dynamix.cache.dirs.txz
Oct 30 21:05:20 AlsServer root: plugin: creating: /tmp/start_service - from INLINE content
Oct 30 21:05:20 AlsServer root: plugin: setting: /tmp/start_service - mode to 0770
Oct 30 21:05:20 AlsServer root: plugin: running: anonymous
Oct 30 21:05:20 AlsServer cache_dirs: cache_dirs service rc.cachedirs: Started: '/usr/local/emhttp/plugins/dynamix.cache.dirs/scripts/cache_dirs -i "Daten" -i "Media" -i "Temp" -u -l off 2>/dev/null'

Link to comment

 

6 hours ago, wgstarks said:

This is true. I installed the 2.2.0d version which you had linked a few pages back and saw this morning that an update was available in my plugins tab. After running the update I now have 2.2.1 installed. Looks like the cache_dirs plugin should automatically continue with your branch when you push updates.

Cool, that's what I thought would happen, when I linked the plugin to my github repo. I saw the same thing happen with my 2.2.2 version.

 

@markswift Ah thanks for reporting, I must have messed up something. Let me check again.

Link to comment

Ah I think maybe markswift was unlocky to be super fast to update if it works for alturismo. I pushed the plg-to the github repo, thinking that I had plenty time announcing it. But the new plg in the github repo, contain a link to the package which was not yet released. A chicken and egg-problem of the way I did the release.

Link to comment

Cache pressure 0 is dangerous. I've used it for some months with my 1.584.929 files as reported in the cache-dirs log and 16 GB memory. I now go for cache-pressure of 1. Works very well, and no crash due to out-of-memory. With zero I think it will never release cached-memory. With cache-pressure of 1 the dirs are almost never lost now it seems, but sometimes cache_dirs decreases depth because the system is under load, but when the load vanishes it scans the increased levels so quickly that I think they were already under caching. It which cache the adaptiveness doesn't matter when it works that well, but now that I have the adaptive feature I kind of prefer it.

 

Link to comment
2 minutes ago, Alex R. Berg said:

Cache pressure 0 is dangerous. I've used it for some months with my 1.584.929 files as reported in the cache-dirs log and 16 GB memory. I now go for cache-pressure of 1. Works very well, and no crash due to out-of-memory. With zero I think it will never release cached-memory. With cache-pressure of 1 the dirs are almost never lost now it seems, but sometimes cache_dirs decreases depth because the system is under load, but when the load vanishes it scans the increased levels so quickly that I think they were already under caching. It which cache the adaptiveness doesn't matter when it works that well, but now that I have the adaptive feature I kind of prefer it.

 

thanks for the info, i start with 2 then now, i guess a update will always require a reboot.

Link to comment

Good to know you've never installed it before. 

 

Are you on unRaid 6.4.0? The plugin says that's required, though I don't really know why, maybe something about the plugin and mdstat command.

 

Are you installing via WebGui > Plugins > Install Plugin and then entered the url https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg

 

I just tried that after removing the plugin first, and that went into an infinite loop, something about 

 

plugin: installing: https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg
plugin: downloading https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg
plugin: downloading: https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg ... done

Warning: simplexml_load_file(): /tmp/plugins/dynamix.cache.dirs.plg:44: parser error : Specification mandates value for attribute data-pjax-transient in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 216

Warning: simplexml_load_file(): name="request-id" content="C47A:2F90:F2F10:1CC859:5BD8BE3A" data-pjax-transient in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 216

Warning: simplexml_load_file(): ^ in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 216

Warning: simplexml_load_file(): /tmp/plugins/dynamix.cache.dirs.plg:49: parser error : Specification mandates value for attribute data-pjax-transient in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 216
 

Possibly that problem came for me because there were leftovers from the previous install. Try downloading the plg to /boot/config/plugins named 

dynamix.cache.dirs.plg

and reboot

Link to comment
1 minute ago, Alex R. Berg said:

Good to know you've never installed it before. 

 

Are you on unRaid 6.4.0? The plugin says that's required, though I don't really know why, maybe something about the plugin and mdstat command.

 

Are you installing via WebGui > Plugins > Install Plugin and then entered the url https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg

 

I just tried that after removing the plugin first, and that went into an infinite loop, something about 

 

plugin: installing: https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg
plugin: downloading https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg
plugin: downloading: https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg ... done

Warning: simplexml_load_file(): /tmp/plugins/dynamix.cache.dirs.plg:44: parser error : Specification mandates value for attribute data-pjax-transient in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 216

Warning: simplexml_load_file(): name="request-id" content="C47A:2F90:F2F10:1CC859:5BD8BE3A" data-pjax-transient in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 216

Warning: simplexml_load_file(): ^ in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 216

Warning: simplexml_load_file(): /tmp/plugins/dynamix.cache.dirs.plg:49: parser error : Specification mandates value for attribute data-pjax-transient in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 216
 

Possibly that problem came for me because there were leftovers from the previous install. Try downloading the plg to /boot/config/plugins named 

dynamix.cache.dirs.plg

and reboot

I'm running 6.6.3 - tried to install manually but it also give an error, I'll also do some digging and see what I can find...

Link to comment

Many thanks Alex for your efforts!

 

Should the folder caching status indicator accurate in the settings page? It shows me stopped, while in the log I have:

Oct 30 21:50:03 Tower cache_dirs: cache_dirs service rc.cachedirs: Started: '/usr/local/emhttp/plugins/dynamix.cache.dirs/scripts/cache_dirs -i "Movies" -c 2 -p 8 -l on 2>/dev/null'

Link to comment
9 minutes ago, olympia said:

Many thanks Alex for your efforts!

 

Should the folder caching status indicator accurate in the settings page? It shows me stopped, while in the log I have:

Oct 30 21:50:03 Tower cache_dirs: cache_dirs service rc.cachedirs: Started: '/usr/local/emhttp/plugins/dynamix.cache.dirs/scripts/cache_dirs -i "Movies" -c 2 -p 8 -l on 2>/dev/null'

+1 here, i know i had the status "running" before ... and i enabled logs, but there where no logs at /var/log/

 

i now turned logs off and rebooted server again.

 

image.thumb.png.befb9d6ed5d5c96aaf2ecc4ad93497e8.png 

Link to comment

Regarding status stopped after reboot: Send me a unraid diagnostics, and also do a cache_dirs -L, and send me the logs it collects. Before that run it with logs -enabled. Its likely there will be enough info for me in the syslog, so if you are in a hurry just send that. I'm off to bed. I'll look at it in the next couple of days. I did include a change from the broken 2018.10.14 version in the recent two plugins, maybe that is causing the restart issues, but I also added logging so the syslog should inform me.

 

The indicator worked for Fireball3 when he had issues, so I suspect it does. The cache_dirs -L gives me all that info as to whether its actually running now, and more detailed info.

Link to comment
9 minutes ago, Alex R. Berg said:

Regarding status stopped after reboot: Send me a unraid diagnostics, and also do a cache_dirs -L, and send me the logs it collects. Before that run it with logs -enabled. Its likely there will be enough info for me in the syslog, so if you are in a hurry just send that. I'm off to bed. I'll look at it in the next couple of days. I did include a change from the broken 2018.10.14 version in the recent two plugins, maybe that is causing the restart issues, but I also added logging so the syslog should inform me.

 

The indicator worked for Fireball3 when he had issues, so I suspect it does. The cache_dirs -L gives me all that info as to whether its actually running now, and more detailed info.

image.png.1c34ce2717867afabea605e2dcc5da6b.png

 

chmod to 777 changed the status to running ;)

 

root@AlsServer:~# cache_dirs -L
################## PS #######################
 PPID   PID  PGID %CPU %MEM TT          VSZ   RSS     ELAPSED     TIME RGROUP    NI COMMAND  COMMAND
    1 15550  7898  0.2  0.0 ?         10312  5476       00:34 00:00:00 root       0 cache_di /bin/bash /usr/local/emhttp/plugins/dynamix.cache.dirs/scripts/cache_dirs -i Daten -i Media -i Temp -p 1 -l on
14094 19639 19639  0.0  0.0 pts/1      8964  3348       00:00 00:00:00 root       0 cache_di /bin/bash /usr/local/bin/cache_dirs -L
19639 19641 19639  0.0  0.0 pts/1      8964  2588       00:00 00:00:00 root       0 cache_di /bin/bash /usr/local/bin/cache_dirs -L
19639 19642 19639  0.0  0.0 pts/1      2944  1784       00:00 00:00:00 root       0 tee      tee cache_dirs_diagnostics_processes.log
19641 19646 19639  0.0  0.0 pts/1      5712  2028       00:00 00:00:00 root       0 grep     grep cache_dirs\|find\|wc
################## PSTREE #######################
        |-avahi-daemon(7870)---avahi-daemon(7871)
        |-avahi-dnsconfd(7880)
        |-cache_dirs(15550)---sleep(19586)
        |-crond(1868)
        |-dbus-daemon(1798)
--
        |            `-smbd-notifyd(1885)
        |-sshd(7819)
        |-ttyd(7920)-+-bash(14094)---cache_dirs(19639)-+-cache_dirs(19641)-+-grep(19648)
        |            |                                 |                   `-pstree(19647)
        |            |                                 `-tee(19642)
################## TOP #######################
top - 22:31:36 up 35 min,  1 user,  load average: 0.68, 0.89, 0.97
Tasks: 641 total,   1 running, 638 sleeping,   0 stopped,   2 zombie
%Cpu(s): 23.7 us,  8.7 sy,  0.0 ni, 66.7 id,  0.0 wa,  0.0 hi,  1.0 si,  0.0 st
MiB Mem :  32112.6 total,   4553.1 free,  12987.4 used,  14572.1 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  17627.6 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 9658 root      20   0 9326140   8.1g  20528 S 337.5  25.7  22:52.54 qemu-system-x86
    1 root      20   0    2444   1616   1512 S   0.0   0.0   0:08.35 init
    2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd
    3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
    4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp
    6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-kblockd
    7 root      20   0       0      0      0 I   0.0   0.0   0:00.67 kworker/u24:0-btrfs-endio-write
    8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq
    9 root      20   0       0      0      0 S   0.0   0.0   0:00.14 ksoftirqd/0
   10 root      20   0       0      0      0 I   0.0   0.0   0:00.80 rcu_sched
   11 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_bh
   12 root      rt   0       0      0      0 S   0.0   0.0   0:00.03 migration/0
   13 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0
  adding: cache_dirs_diagnostics_processes.log (deflated 65%)
  adding: cache_dirs.log (deflated 91%)
  adding: syslog (deflated 86%)

Generated cache_dirs_diagnostics.zip
root@AlsServer:~#

Edited by alturismo
Link to comment

You need to specify RAW in the URL when linking to the PLG.

 

Try using: https://raw.githubusercontent.com/arberg/dynamix/master/unRAIDv6/dynamix.cache.dirs.plg

 

When I used the URL without the RAW, what I got was an HTML file, with the first bits looking like this:


<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
  <link rel="dns-prefetch" href="https://assets-cdn.github.com">
  <link rel="dns-prefetch" href="https://avatars0.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars1.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars2.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars3.githubusercontent.com">
  <link rel="dns-prefetch" href="https://github-cloud.s3.amazonaws.com">
  <link rel="dns-prefetch" href="https://user-images.githubusercontent.com/">

  <link crossorigin="anonymous" media="all" integrity="sha512-DMUv06uA9jcLpuzWEnwuWTuYn/1rt3UmOKgIrS5ESQLJtqWQzexcD85XhWhAk0wcGQmlhspauIq+6Hjmue1A5g==" rel="stylesheet" href="https://assets-cdn.github.com/assets/frameworks-1291ffafdf7ffb4d9be67d690fd09212.css" />
  <link crossorigin="anonymous" media="all" integrity="sha512-T7Yq7rbiCzjLhP8CjEZRSSk/MBGeGhrW/1qjEVxq3LRxojFj+lZBb+irUNAsx12/UMTqKfO4++C5RoVNxAoJtQ==" rel="stylesheet" href="https://assets-cdn.github.com/assets/github-040328cdcb5f37143588d5b5e7893ba5.css" />

Link to comment
5 hours ago, alturismo said:

chmod to 777 changed the status to running

That did the trick.  For anyone else who wants to do this the cache_dirs file is actually in this directory:

 

/usr/local/emhttp/plugins/dynamix.cache.dirs/script

The commands using the built-in Terminal popup to make the permission change are :

cd /usr/local/emhttp/plugins/dynamix.cache.dirs/scripts
chmod 777 cache_dirs

EDIT:  IF you reboot your computer you will have to make these changes again as you are making the change to the file in on the RAM disk that Unraid uses to install the OS on.  

Edited by Frank1940
  • Upvote 1
Link to comment

ok, 1 st real test today morning after some hours idle, i still get spinning up disks when accessing my shares.

 

i guess my settings aint properly or does adaptive mean theres a learning curve ?

 

i changed now from adaptive to fixed

 

here from my log file now

 

cache_dirs version 2.2.2
Setting Memory ulimit to 50000
Setting cache_pressure=1
Arguments=-i Daten -i Media -i Temp -p 1 -l on -D 9999
Max Scan Secs=10, Min Scan Secs=1
Scan Type=fixed
Max Scan Depth=none
Use Command='find -noleaf'
---------- Caching Directories ---------------
Daten Media Temp
----------------------------------------------
Setting Included dirs: Daten,Media,Temp
Setting Excluded dirs:
min_disk_idle_before_restarting_scan_sec=60
scan_timeout_sec_idle=150
scan_timeout_sec_busy=30
scan_timeout_sec_stable=30
frequency_of_full_depth_scan_sec=604800
cache_dirs started

 

then there are alot of thos entries ...

 

2018.10.31 06:55:59 Executed find in (0s) 00.09s, wavg=00.09s   depth 9999 slept 0s Disks idle before/after 785s/785s suc/fail cnt=0/0/0 mode=2 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=1 CPU= 3%, filecount[9999]=18753
2018.10.31 06:56:00 Executed find in (0s) 00.04s, wavg=00.06s   depth 9999 slept 1s Disks idle before/after 786s/786s suc/fail cnt=0/0/0 mode=2 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=1 CPU=17%, filecount[9999]=18753
2018.10.31 06:56:01 Execut.....

 

should this be correct ?

 

i only have 3 shares using the physical disks ... all with the setting to prefer cache (using scheduled mover).

 

all other shares here are cache only anyway.

Link to comment

@alturismo

 

Looks good. 9999 is my internal way of representing infinite depth in the cach-dirs bash script. 
 

 Disks idle before/after 786s/786s

This means its not reading on your disks and nothing else is, because they are idle before and after a scan.

 

filecount[9999]=18753

With only 18753 files, fixed depth is the way to go. Also your cache-pressure of 1 would be perfect for you I suspect.

 

By the way, you didn't include user-share. I don't either, but if your disks spin up, include it and see if it helps.

Edited by Alex R. Berg
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.