Alex R. Berg

yep, exactly, try that and we'll see the result. I'm very interested in seeing a snapshot af the 100% cpu spike, and the cachelog at the same time. But the diagnostics should produce that, but your job is to fire it when its at 100% cpu. ./cache_dirs -L Also I'm very interested in seeing it at a time where there are defunct processes again, and also whether the 100% cpu problem always coincide with the defunct processes. I found a better way to do the ps-command, I've updated cache_dirs attached, so its log-diagnostics is better. If you run the attached find_defunct.sh then it'll wait and do the scan when it finds defunct processes (if it works) cache_dirs find_defunct.sh

You could also try adding to the cache_dirs command -U 100000 to set memory limit higher. Maybe the system is killing the process, or suspending it because of lack of memory. Just a guess. ./cache_dirs -i Video -e Audio -e Backup -e Fotos -e Spiele -l on -U 100000

Ah, please use this cache_dirs instead for your tests. I noticed the defunct process happened at 3:42 which means right after moving started. But I still cannot reproduce the issue. cache_dirs

I've made some changes to cache_dirs, and attached the file. Download it to somewhere, and then do sudo chmod 755 ./cache_dirs; ./cache_dirs -q; ./cache_dirs -i Video -e Audio -e Backup -e Fotos -e Spiele -l on -T The last -T paramater disabled multithreaded disk-scan, and makes it more like the last 2.1.0 cache-dirs. See if the problem appears with and without -T to get diagnostics you can now run (on the new attached cache_dirs) ./cache_dirs -L When you reboot the machine, or start cache_dirs from the plugin, the installed cache_dirs plugin will be running, not the attached script. cache_dirs

Interesting with the 'defunct' processes. I've seen that myself once a long time ago. 'defunct' processes are apparently dead processes, kept around because of some crash or something because their parent exits, ie. cache_dirs. However those processes should not be running, so cannot generate CPU-load. I need to see what is causing the cpu-load. Can you include the following, and keep doing it until you get 100%: top -b -n1 |head -20 > top-output.txt; cat top-output.txt I have attached two files containing all the diagnostics I need, and also zips the zip file. Run cache_dirs_diagnostics_generate_zip.sh and keep running it until you see the 100% cpu process in the list of top-processes. 2.2.0j added process management where each disk is scanned in its own process. Apparently there's a problem with that. Did you stop the array at any time before the defunct processes appear? (probably not) How many disks do you have? (not that it should matter) Best Alex cache_dirs_diagnostics.sh cache_dirs_diagnostics_generate_zip.sh

Here's some relevant info for you to learn to debug the problem pstree |-cache_dirs(10139)---sleep(30575) This says which subprocesses cache_dirs currently has. Right now its sleeping. Cache-dirs does all its work in the 'find' and the 'wc' file-counting sub-process, though the wc is only executid when the script is first started to count files at each depth. The ps-command gives info into how subcommands are executed, but there were no sub-commands. Why do you think its running 100% cpu? crontab I didn't know about the existence of the file : /etc/cron.d, good for me to know. The last line is incorrect, hence the complaints in your syslogs. Something on your system must be messing it up. You could debug what it is if you want, by removing other plugins and reenabling one by one or doing binary search enabling half at a time. This is how my file looks: # Generated cron settings for plugin autoupdates 0 0 * * * /usr/local/emhttp/plugins/ca.update.applications/scripts/updateApplications.php >/dev/null 2>&1 # Generated ssd trim schedule: 0 0 * * * /sbin/fstrim -a -v | logger &> /dev/null # Generated docker monitoring schedule: 10 0 1 * * /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/dockerupdate.php check &> /dev/null # Generated mover schedule: 40 3 * * * /usr/local/sbin/mover &> /dev/null # Generated plugins version check schedule: 10 0 1 * * /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugincheck &> /dev/null # Generated array status check schedule: 20 0 * * 1 /usr/local/emhttp/plugins/dynamix/scripts/statuscheck &> /dev/null The status could be lying. Running the ps or pstree command tells you the truth. Its a good indication though that nothing is cached, and drives are spun down, and spinning up when you access drives. Linux caches lots of stuff in memory, it might be different stuff is cached. Best indication is disks spinning up when you access dirs that should be cached. Note only dirs are cached, not file content, but I'm sure you know that. The cache-dirs log gives you info about what its doing, and how many files you have cached at each level. 2018.09.27 10:26:06 Executed find in (0s) 00.10s, wavg=00.10s Idle____________ depth 9999 slept 10s Disks idle before/after 1538036766s/1538036766s suc/fail cnt=69/70/0 mode=4 scan_tmo=150s maxCur=9999 maxWeek=9999 isMaxDepthComputed=1 CPU= 4%, filecount[9999]=16460 This line from a previous diagnostics cache-dirs log indicated filecount[1]=0 filecount[2]=1213 filecount[3]=7832 filecount[4]=11794 filecount[5]=14207 filecount[6]=15873 filecount[7]=16460 filecount[9999]=16460 This means 1213 files at depth 2 (/mnt/user/share/X) and 15873 files at depth 6 (/mnt/user/share/X/Y/Z/A/B). I'm probably off by one or two on the counts, but this gives you an idea. If the next scan says 'Disks idle before/after 9999s/9999s' then disks are idle or spun down, and scanning didn't bring up the disks. You could reduce Cache Pressure to 1 while debugging, but you might run out of memory crashing the system, especially if you reduce to 0. In the last log at 20:14 cache-dirs was also started at startup. Disks where spun down later. But there's no cache_log. Maybe you forgot it? I just played with the diagnostics myself, I can see its not automatically included. It would be cool if the plugin could tell unRaid to include the log-files in the diagnostics, but I don't know if that is possible. Best Alex

@Fireball3 Things look great. the cache_dirs log in 20180927-1021\logs\cache_dirs.log indicate its running in each scan in 0 seconds, and waiting 10 secs in between running. Did you try the ps + pstree ? Regarding your syslog crontab problem I saw in your syslog: your crontab looks fine. Maybe you have a bad script in one of the dirs /etc/cron.* Regarding the startup problem, it does seem like the cache dirs starts and system boot at is should. It starts within seconods after mounting the disks. Why do you think its not running on system boot? Sep 27 10:01:53 Tuerke kernel: REISERFS (device sdi1): Using r5 hash to sort names Sep 27 10:01:53 Tuerke emhttp: shcmd (80): set -o pipefail ; mount -t reiserfs -o remount,user_xattr,acl,noatime,nodiratime /dev/sdi1 /mnt/cache |& logger Sep 27 10:01:53 Tuerke emhttp: shcmd (81): sync Sep 27 10:01:56 Tuerke emhttp: shcmd (82): mkdir /mnt/user0 Sep 27 10:01:56 Tuerke emhttp: shcmd (83): /usr/local/sbin/shfs /mnt/user0 -disks 4094 -o noatime,big_writes,allow_other |& logger Sep 27 10:01:56 Tuerke emhttp: shcmd (84): mkdir /mnt/user Sep 27 10:01:56 Tuerke emhttp: shcmd (85): /usr/local/sbin/shfs /mnt/user -disks 4095 1024000000 -o noatime,big_writes,allow_other -o remember=0 |& logger Sep 27 10:01:56 Tuerke emhttp: shcmd (86): cat - > /boot/config/plugins/dynamix/mover.cron <<< "# Generated mover schedule:#012 /usr/local/sbin/mover |& logger#012" Sep 27 10:01:56 Tuerke emhttp: shcmd (87): /usr/local/sbin/update_cron &> /dev/null Sep 27 10:01:56 Tuerke s3_sleep: ---------------------------------------------- Sep 27 10:01:56 Tuerke s3_sleep: command-args=-C 2 -h 00 -h 17 -h 18 -h 19 -h 20 -h 21 -h 22 -h 23 -a -m 15 -n -e eth0 -N 12500 -i 192.168.178.150 -i 192.168.178.153 -l -L -D 0 Sep 27 10:01:56 Tuerke s3_sleep: action mode=shutdown Sep 27 10:01:56 Tuerke s3_sleep: check disks status=yes Sep 27 10:01:56 Tuerke s3_sleep: check network activity=yes Sep 27 10:01:56 Tuerke s3_sleep: check active devices=192.168.178.150 192.168.178.153 Sep 27 10:01:56 Tuerke s3_sleep: check local login=yes Sep 27 10:01:56 Tuerke s3_sleep: check remote login=yes Sep 27 10:01:56 Tuerke s3_sleep: version=3.0.6 Sep 27 10:01:56 Tuerke s3_sleep: ---------------------------------------------- Sep 27 10:01:56 Tuerke s3_sleep: included disks=sdb sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn Sep 27 10:01:56 Tuerke s3_sleep: excluded disks=sda Sep 27 10:01:56 Tuerke s3_sleep: ---------------------------------------------- Sep 27 10:01:56 Tuerke s3_sleep: s3_sleep process ID 4470 started, To terminate it, type: s3_sleep -q Sep 27 10:01:56 Tuerke emhttp: Starting services... Sep 27 10:01:56 Tuerke emhttp: nothing to sync Sep 27 10:01:56 Tuerke sudo: root : TTY=unknown ; PWD=/ ; USER=nobody ; COMMAND=/bin/bash -c /usr/local/emhttp/plugins/unbalance/unbalance -port 6237 Best Alex

@Fireball3 yeah, ps gives you processes, and the grep filters the ones i'm interested in. pstree gives processes and subprocesses but does not give much info into what is going on. You'll have to wait till the 100% cpu problem happens. Its not unnormal for high CPU, when all files are cached and during scan. But it shouldn't scan that often, and I doubt it should hit 100% cpu. Best Alex

@Fireball3 by the way, you have a cron-problem. from your syslog: Sep 14 23:05:11 Tuerke crond[1500]: failed parsing crontab for user root: /usr/local/sbin/mover |& logger maybe its related to the dynamix.mover plugin Sep 13 19:17:17 Tuerke emhttp: shcmd (86): cat - > /boot/config/plugins/dynamix/mover.cron <<< "# Generated mover schedule:#012 /usr/local/sbin/mover |& logger#012" you can do 'sudo cronjob -l' to investigate best Alex

Ah, good to know the 2.1.1 doesn't give you trouble. I'll need the cache-dirs log in /var/log/cache_dirs.log. You need to enable logging first in plugin (or touch the file) to get it written. Also it will be help-ful to see what is spinning, if you want to dig into the issue: ps -e x -o ppid -o pid -o pgid -o tty -o vsz -o rss -o etime -o cputime -o rgroup -o ni -o fname -o args | grep "cache_dirs\|find\|wc" pstree -p | grep -2 cache_dirs

I cannot see the cache-dirs script executing in your syslog, because its not logging that start/stop behaviour. I've added that to my system, so I can see what is being done. The cache-dirs starts on the array-start event if enabled. I've added some files that change the \usr\local\sbin\emhttp_event so that it logs what it does to syslog. I recommend not keeping it in the custom folder when your done testing, in case limetech changes the file at some time in the future. Be sure not to overwrite any existing files if you don't want to lose them. Though the only risk is your /boot/config/go script and it seemed that was the default. Maybe there are better ways to get unRaid to log what happens in the events, but I'm not aware of them. Unzip the file to /boot (or rather unzip to /tmp and investigate, then unzip or copy to /boot). Best Alex flash-log-events.zip

Oh right, two threads... I'm not sure anybody is maintaining the cache-dirs script, but I am the last one who got their hands dirty in the script. Honestly I'm really sick and tired of the bash script, bash is a terrible language for a script this long, and I made the script a lot longer by adding a lot of adaptive logic to it. But anyway it would be nice if the script works, but it does seem to work quite well for me. The dynamix plugin contains the standalone cache-dirs script in an updated version, and the plugin itself is just a wrapper to run it and configure via GUI. The script in this thread has not been updated for years. I think it would be best to have the cache-dirs conversation in a thread separate from all the other dynamix plugins, or the conversation will be muddled. I think this place is a good place, since Joe seems to not maintain the original script. Now I'm not promising that I will maintain it, but I might listen along. If the problem with the plugin is that it does not start, then the dynamix thread is probably the best place to address the problem. The cache-dirs script (also in the plugin version) can be run manually. The plugin installs the script to /usr/local/bin/cache_dirs and service files to /etc/rc.d/rc.cachedirs Fireball3: It sounds like the issue that it doesn't start could be a plugin issue, I'll add a comment in that thread Best Alex

I included the lines in case you knew your way around the bash shell, and didn't mind getting your hands dirty. Just ignore those lines. It looks like it works fine anyway in your last log, and those lines are only for reporting anyway. Best Alex

These are the lines used to extract the duration since last disk-access. Seems there's a problem with them on your server. You could try fidling with those, if you want. It might give some insight into what the actual idle time is, and you could update the script if you want to help. I'm not an awk expert, and kind of don't feel like messing much around this crazy long bash script at the moment. But I'll probably help out getting a correction into the code-base https://github.com/bergware/dynamix mdcmd_cmd=/usr/local/sbin/mdcmd # rdevLastIO will be non-zero if a disk is spinning, it will be the timestamp of last IO (in seconds since epoch) last=$($mdcmd_cmd status | grep -a rdevLastIO | grep -v '=0') echo "$(echo $last | awk '{t=systime(); gsub("rdevLastIO..=",""); for(i = 1; i <= NF; i++) a[++y]=$i}END{c=asort(a); if (NF > 0) print t-a[NF]; else print 9999; }')" Best Alex

Hi Eisi, This may sound stupid, but are you sure its the cache-dirs doing it? You don't have many files under watch, only 57171 files and you've only watched till depth 5. Setting maxdepth=5 The log reports when disks were last accessed, but there seems to be a bug, since it reports a crazy duration, 1537449213s/1537449213s , the first being the idle time before scanning the dirs, the latter the idle time after scanning the dirs. Here below something else touched the disks it seems, because it slept 10 secs and in between disks idle time became sensible: 2018.09.20 15:14:24 Executed find in (0s) 00.15s, wavg=01.33s Idle____________ depth 5 slept 10s Disks idle before/after 9999s/9999s suc/fail cnt=18/18/0 mode=4 scan_tmo=150s maxCur=5 maxWeek=5 isMaxDepthComputed=1 CPU= 3%, filecount[5]=57171 2018.09.20 15:14:35 Executed find in (0s) 00.16s, wavg=00.98s Idle____________ depth 5 slept 10s Disks idle before/after 8s/8s suc/fail cnt=19/19/0 mode=3 scan_tmo=30s maxCur=5 maxWeek=5 isMaxDepthComputed=1 CPU=22%, filecount[5]=57171 Personally i've reduced cache-pressure to 0 or 1, but then I have plenty of ram. Cache-pressure of 0 makes it very risky of running out of ram, so now I use 1. But I also have a crazy amount of files cached, 1.6 million files. Best Alex

It will only cache directories on the array. The plex docker app of cause is using media-files on the array via a volume mapping, and those directories on the array will be cached. The internal 'operating system' files of the docker app will not be cached.

Cache_dirs update 2.2.0 I think I'm done with the 2.2.0 update. I've attached a version of the plugin that points to my forked github so you can test it until it goes live. Note that because of how the plugin system works I've fixed the issues mentioned in my previous comment, about the plugin not starting on boot, it was my bad. Users notes The new cache_dirs scans your disks when theive been idle for 60 seconds. This can now be adjusted in the options. If the disks are idle once in a while, it seems to work far better than before. Cache-pressure 0 works impressively for my 1.5 million files and 16 GB of memory, but it may crash the system, because linux docs says it will not release cache-memory even if the system is running out of memory. Programmers notes I've moved the cache_dirs to the dynamix plugin github. Te plugin and the cache-dirs version are tied together, the current cache-dirs page does not work properly with older cache_dirs versions. The plugin contained the MD5 of the cache_dirs.zip, and if the cache_dirs.zip was updated but the plugin was not updated, it would cause 'unexpected' behaviour, such as redownload on each reboot. Since I've moved cache_dirs to dynamix github I've added a note that its not under copyright of dynamix. You can move it back out if you want. The newest script is also committed in dlandons github (in my fork). Future ideas The cache_dirs log is very useful to tweak your personal system. If it was possible to copy the Syslog.htm page and point it to /var/log/cache_dirs.log and put a link to it in the Folder Caching page it would be very useful for users. It would be nice with restart buttons like on the PhAzE's Resilio sync plugin. Release notes # Version 2.2.0 - When cache lost, don't wait for disks idle, instead decrease depth # - Now scans each disks and cache in separate processes. Thus removed option -u for scan user # - Fixed loss of adaptive after 1 week # - Weekly rescan only when disks been idle long # - Apaptive scan will not retry increasing level to previous seen stable level until disks have been idle for a long time (20s) # - -q now also kills subproccesses immediately (so cache_dirs does not prevent stopping the array) # - Added file-count and depth adjustment by filecount # arberg Best Alex dynamix.cache.dirs.zip

Its not yet promoted to the automatic update stream. That will happen when its tested a bit more and approved. Till then you can try the changes by replacing the cache-dirs plugin with the version i linked. You can just copy the file over the existing in /boot/config/plugins/dynamix.cache.dirs.plg and reboot (or run something like installplg or remove and install from gui). Best Alex

Cache_dirs update I've update the cache_dirs plugin. Made many many small changes that I consider improvements. I've update the dynamix plugin folder, fixed a bug in the start script, and added options to the settings page and added help info. I've forked dlandon and dynamix githubs, and pushed my changes. I have updated the dynamix-txz and cache_dirs.zip in my forks, so if you use this temporary plg attached here which points to my forks, you can test it. The new plugin requires the new cache_dirs script because I've added new options to the script. The current attached cache_dirs plugin does not start when the server reboots on my server where it auto-start the array. I hope someone can fix that for me. If you send me logs from /var/log/cache_dirs.log and /var/log/cache_dirs_lost_cache.csv I can inspect them, though I'll probably quickly tire of looking at logs, but it would be helpful to see how cache_dir is behaving on other systems. # Version 2.2.0g- When cache lost, don't wait for disks idle, instead decrease depth # - Now scans each disks and cache in separate processes. Thus removed option -u for scan user # - Fixed loss of adaptive after 1 week # - Weekly rescan only when disks been idle long # - Apaptive scan will not retry increasing level to previous seen stable level until disks have been idle for a long time (20s) # - -q now also kills subproccesses immediately (so cache_dirs does not prevent stopping the array) # - Added file-count and depth adjustment by filecount # arberg If your disks are never idle for 20 seconds the adaptive scanning will no longer work for you. If the disks are idle once in a while, I believe it will work far better than before. Best Alex dynamix.cache.dirs.zip

I've considered updating the help for the cache-dir plugin. I've found it valuable to set max-scan depth even though I've now made the adaptive scanning. Its valuable to make sure not to have more files than needed under cache-dirs scanning eye, to reduce disk-spam, and I might write that in the documentation. Do we have any kind of guideline anywhere for how to write help? If I update FolderCachingSettings.page and make a pull-request, will BergWare do his magic? https://github.com/bergware/dynamix/blob/master/source/cache-dirs/FolderCachingSettings.page Best Alex

Good. I've committed my changes for now to my own repository here: https://github.com/arberg/cache_dirs When I think its worthy of your time, i'll post a pull-request to your repository. Or wait a week and have a look, if I should suddenly get busy elsewhere and not return. I have not read up on all the issues posted here on cache-dirs, but as far as I have picked up (and from my own experience) cache-dirs still in my adaptive version spams to much on the disks. I found some definite design-issues with my adaptive scan, where it would keep trying to hard to get to it previous best depth. I have committed my changes, and will keep looking at it.

Thanks, dlandon. I want to have a github version-tracking of non-zipped, so I can track changes properly. It would we valuable for the community if someone hosted such a one on Github. It would probably be better if someone else hosts the cache_dirs, maybe someone who responds as quickly as you do dlandon I've changed my unRAID repository to contain the raw file, and added links to dlandon and dynamix githubs. https://github.com/arberg/unRAID I've updated the cache_dirs a bit, but its still work in progress. I hope to make the script prevent spamming disks, which was my idea for the adaptiveness in the first place. So I might make the script back off a lot sooner and for a longer period when it detects it is spamming disk. Alex PS on your GitHub comment >> Thank you Alex R. Berg for your updates to cache_dirs. You are welcome

I figured I might do a little cleanup of logging and maybe fix a bug or two in the cache_dirs plugin. I can see bergware did some nice updates on the plugin, and dlandon cleaned up the cache_dirs script itself. dlandon made the last changes I can see in the current dynamix plugin. Do we have a github page containing the latest plugin, or rather the latest cache_dirs script (in non-zipped format?) On a side-note, I tried creating new topic for this in 'plugin support', but couldn't. I'm not sure it belongs in new topic anyway though. Best Alex

Awesome script Skidelo, I've been looking for something like that. I've updated the script. I found some problems, and have fixet some of them. Maybe the xml format or parse has changed. Changes: Fixed parsing of muliple volumes and ports (didn't Added quotes around volumes Parse multiple files like with *.xml Added dryrun and verbose flags Current Problem: The xmlint xml-decoder does not parse xml-encoded strings like &#xF8;, so beware. For me the docker command failed, and I could manually add that docker. Best Alex RunDockerTemplate.zip

Hi Joseph, It may be your array just registered the disk as not in the array, possibly the disks are still ok. If possible capture the /var/log/syslog* before rebooting, it is deleted on boot, and may contain important information. Have you seen this: If the disks are mounted I think its the mdcmd comm which does the array dismount, but it has behaved weirdly for me lately, though I havn't had parity errors afterwards. You can find open files using (useful if umount fails) sudo lsof | grep mnt If you have unmounted and executed the mdcmd I would use powerdown -r I have added an updated version of my arraystop script with some more details and some private stop commands like stop docker. Good luck arraystop