Alex R. Berg

Yes certainly that is easy to do with 'docker -v'. I just don't need the svn client so I havn't done it. If I had lots of working dirs scattered in lots of places, it would be a hassle. I also frequently get confused and mess up with the virtual volumes in docker, I mean there's an indirection so its one more possible error, not to mention user id trouble. If I needed an SVN client on unRaid, I would prefer it to be on the system natively I think, but I must admit, I absolutely love the docker services that just work, no matter what version packages unRaid runs. PS: I need the svn server with Cyrus Sasl authentication.

I have now tried updating my old subversion plugin to make it be based on current slackware 14.2. I kept running into missing packages, and then I have up on that approach. I then tried SCM-Manager docker. It worked great, super easy to setup and get working, but I didn't succeed with setting it up securely for remote access. Its probably easy, but I left it. I then tried building my own docker-file, because it sucks being dependent on managing our own dependencies from slackware and it sucks that it breaks when unRaid is updated. I got it working. It was a bit of a pain, but it turned out my Cyrus-Sasl password with a '#'-char didn't work in the new Cyrus-Sasl. But anyway it works now. The dockerhub images name is 'arberg/subversion'. The github docker-file repository is arberg/docker-subversion. dmacias: If you want to get subversion to working again, I captured the list of packages which ubuntu installs, when it installs subversion. All of the dependencies that I noticed was missing are in the list, such as serf. Here's the list of packages I tried installing from slackware-current, it wasn't enough: <!ENTITY icu4c "icu4c-63.1-x86_64-1.txz"> <!ENTITY subversion "subversion-1.11.0-x86_64-1.txz"> <!ENTITY neon "neon-0.30.2-x86_64-1.txz"> <!ENTITY apr "apr-1.6.5-x86_64-1.txz"> <!ENTITY aprutil "apr-util-1.6.1-x86_64-7.txz"> <!ENTITY serf "serf-1.3.9-x86_64-3.txz"> Ahh - here's the alpine OS package list, its a lot shorter and looks like slackware packages: (1/15) Installing db (5.3.28-r0) (2/15) Installing krb5-conf (1.0-r1) (3/15) Installing libcom_err (1.44.2-r0) (4/15) Installing sqlite-libs (3.24.0-r0) (5/15) Installing heimdal-libs (7.5.0-r1) (6/15) Installing libsasl (2.1.26-r13) (7/15) Installing cyrus-sasl (2.1.26-r13) (8/15) Installing libuuid (2.32-r0) (9/15) Installing apr (1.6.3-r1) (10/15) Installing expat (2.2.5-r0) (11/15) Installing apr-util (1.6.1-r2) (12/15) Installing lz4-libs (1.8.2-r0) (13/15) Installing serf (1.3.9-r4) (14/15) Installing subversion-libs (1.10.0-r0) (15/15) Installing subversion (1.10.0-r0) My Docker doesn't give me an svn command on the unRaid system which kind of sucks. It should be easy enough to add a svn script like "docker exec subversion svn $*" but that won't have access to local files, so not really an option.

I suspect there is currently a mismatch of some versions for running subversion. It seems my unRaid 6.6.5 now updates packages from slackware-current, or something else on my system does (maybe NerdPack itself). svn says: `/usr/bin/svnserve: error while loading shared libraries: libicui18n.so.56: cannot open shared object file: No such file or directory` If I install http://slackware.cs.utah.edu/pub/slackware/slackware64-14.2/slackware64/l/icu4c-56.1-x86_64-2.txz with 'installpkg' or `/sbin/upgradepkg - -install-new` then it works. Except it doesn't fully work because my Cyrus-Sasl authentication is invalid. I see Cyrus SASL 2.1.27 is installed, which is the version from slackware-current. If I then roll back to 2.1.26, my authentication works. If I then upgrade to the new icu5c package on slackware-current I get the same svn error as above. So my guess is there is version mismatches, and probably icu4c was already installed. Maybe I'll be happy switching to docker https://hub.docker.com/r/sdorra/scm-manager/ It looks great, but requires some work making it secure for remote access I think.

Interesting. It should definitely be built in something other than bash, but actually its a very simple fundamental idea, so could relatively easily be built in another language. If I was to do it, I would do it in scala. We would need a database to put the scan-durations in, which in my eyes complicates the setup quite a bit. I wonder if that's overengineering it a bit. I doubt many would access it, unless there's also a web-page to display the timings in some aggregate format. I doubt I'll play more with it. I don't think its worth it, because in my experience tinkering with cache_dirs, it can never be really good. Linux will discard the cache when it wants, and we don't know whether or not it has discarded cache or system is otherwise busy. I think there are many other projects much more valuable to put my efforts into. Actually I write to a csv-file which can be opened in excel. I've just added a 2.2.6 version that puts most or all the data in a csv-file. I think that might be useful to some. Best Alex

I've publish new version (sorry to bonienl who just merged, but I respond to no blame for bugs in my code-base ) I noticed from @wgstarts logs that it didn't sleep as I expected. When using fixed depth, it now sleeps up to 10 secs (-M param) depending on disks-idle and current scan duration compared to avg. If its well received I'll send people to the main dynamix branch in around a weeks time.

Yeah it sucks with the fork, I guess its what happens when unpaid voluntary hands put their efforts together. I know I'm looknig at cache_dirs only when I feel like it, I'm not kind of committed to helping in the same way as my paying customers (ie. my employer), where I have a tighter relationship. I'm guessing boniel has it the same way. But maybe we can manage to do a better team-work effort in the future, that would be nice for everybody I suspect But actually its not that bad if looked at in another way. If we think of my branch as the beta-branch, then it makes sense. I can send everybody back to main when I'm done, so everybody gets an automatic update in the future from the main dynamix branch. Good to hear its working again NAS.

I can send you all back there, by updating my plugin. And actually another time we could switch everybody the other way, it it looks like I might be doing several updates in a row to fix something. bonienl is there anyway I can contact you that would speed up that process, besides posting here?

@wgstarkswell under 60 sec to load 100 files, as in >= 30? Are you in 1980 with pin-hole paper? Sounds weird to me. I can see from the logs that its scanning every second, and I would say it should have backed off to sleep 10s by that time. But its only CPU and files are fully cached since disks spin down, so you have a different problem if that little CPU load can hurt your pc.

@bonienlIt seems the cache-dirs plugin is pretty stable now with unRaid 6.6.5. If you want to pull into your dynamix main, please feel free to do so. I think I have a pull-request that has all the changes, or you can just pull from my master-upstream directly. I'm publishing releases via releases-tab so I don't get git history poluted with old archives. But you can revert that in the plugin, and just use your strategy. Or we could have a conversation about where to host the plugin going forward. Ahh, I just noticed a 13 days old mail in my mailbox I havn't checked for some time. Sorry boniel...

@pluginCop Certainly. Dynamix can just accept my pull request, that would be great for all those still on the old plugin. Its also the old plugin url that's in the 'appstore'. We can try pinging dynamix again, and see if he responds, I think its boniel. The downside is that at the moment I'm more active than he is. I don't know how to issue PR's. I you most welcome to ping him, if you know the way. It seems the plugin is stable now with unRaid 6.6.5, so would be good to pull to 'original' github repository. I've justed pinged boniel (if using @ syntax pings him) here: Ahh, I just noticed a 13 days old mail in my mailbox I havn't checked for some time. Sorry boniel...

Its all in the help of folder caching plugin

@Fireball3 Neat @jowe Awesome, that's good to hear. I've released a new version with minor changes to plugin page. Fixed help-text and moved user-share info up, so its more visible. I was wondering whether the unRaid version could also have something to do with some users (like you Fireball) having to include user-share in the scan to avoid disks spin-ups. Its probably not that though.

ROFL Man just a week ago my wife was deeply frustrated because there was no sound on the HTPC, and it was important to be fixed right then and I was busy for 15 min in an online game. Too bad for her. A reboot fixed it for her when I came to her later, and then she frustrated said 'Oh I always forget that rebooting is an option'. There's Value Learning Experience. I think she just rose a level in the hacker-class I just updated to 6.6.3 from 6.4.0 a week ago. I did a custom schedule of the mover, and found later that i didn't set it for every second month but every single day every second month. I doubt that's the issue others are seing, but that certainly caused my disks to spin, and cache-dirs will frequently lose its cache, when the system is under pressure I have found, though particularly when my md5-scan updates its data. @Jowe how much memory do you have in your machine? I'm just curious, I probably cannot help with the problem. @MaticksUpdate as @Fireball3 pointed out. I notice some disk-idle timestamp durations in your log, that should be fixed in new version, though I'm not sure it'll matter. I'm not certain cache-pressure 0 will make machine crash if the memory is exhausted, I don't think I have ever experienced that, though it seems likely. If using cache-pressure 0, and if its still implemented in kernel to never release dirs, I would think it would be possible to spin down disks manually, and they should stay down unless other things are reading from the disk. 'sudo lsof | grep /mnt/' might help to figure out what is accessing disks, though it just gives a snapshot. So if cache_dir logs says scan duration 1s,1s,1s,1s,1s,64s,1s and the disks do not spin up after manually spinning them down, obviously the 64s was not due to lost cache, but must have been because CPU was under heavy load. However I find such debugging to be tiresome because its so difficult to narrow down what is accessing the disks. But still there might be valuable info from that test. My guess though is something is actually reading or writing the disks. @wgstarks I'm not sure I'll investigate much further in it, but maybe I'll throw a quick glance at your logs. What is UD mount? cache-dirs reports which dirs it scans. Is that list correct?

I've released new plugin, no function change. I've highlighted cache-pressure in plugin page by moving it to the top. I couldn't figure out how to make it bold... Jowe: I was suspecious if I was reading it correctly. I cannot see your settings becauese that was cut from the log file, but it seems your not caching user-share. Try including that. Not that I see why it should matter, but then I still don't get why some have to scan user share to avoid disks spinning up. I've attached cache-dirs archive. My 2.0 versions are missing, I don't know there they are, probably somewhere in this long thread. You have probably been using 2.1.1 or 1.6.9 earlier. You can try one of those if you wish, just stop cache_dirs and run the script manually, like in the good old days before the dynamix plugin. If you have less free memory now than back then, that could also be a reason for the change. I can see from your logs, that it seems like your system flushes the dirs from the memory cache, since once a minute a scan takes very long. Its also scanning with only 1s sleep in between each. I doubt Joe's original version of the script or my previous versions would change that. I also doubt the user-share scanning would make a difference but you can give it a try. I've attached a test_free_memory.sh script which writes 2*4 GB of data on your /tmp drive which is mounted to ram on unRaid. It may crash your system, if you have to little memory, though it doesn't for me. But I think I have mounted filesystem with only 50% memory like this in my go-script: # Mount tmpfs at 50% capacity of memory. Will only use memory if filesystem is written to, and hopefully my logrorate will move it out (though it wont if its overfilled) echo "Mount tmpfs" mount -t tmpfs -o remount,size=2% tmpfs /var/log Syntax test for 1x4 GB: test_free_memory.sh 1G Manually delete /tmp/testing* afterwards, if you kill script, and my trap fails. cache_dirs-releases.zip test_free_memory.sh

Good to hear it's working for you jowe. My experience is also that cache pressure is the most important parameter. Nnate, cache_dirs does not access files, only dirs. I'm not sure what activity monitor you are seeing, but in my experience guess work regarding what arbitrary services on my machine accessing files is fruitless. Lsof might be your friend but I think you will be frustrated tracking that down, and might spend a lot of time gaining near nothing. But if it's fun go for it ;)

@jowe Try cache_pressure 1. You might also debug with less files, like depth 2 or something ridiculously low. Maybe you have to little memory. Also next time please include diagnostics run 'cache_dirs -L'. Possibly remove syslog if you don't want to risk giving away sensitive data, if any. It might be a bit clearer for me. > Disks idle before/after 1s/0s This is a good indication that cache_dirs possibly accessed your disks, of cause sometimes its just other apps. I trust you verified disks go to sleep, if cache_dirs isn't running. > "After some time it starts to increase the "Disks idle before/after" That means at that point in time at least you had enough memory and everything was cached. With cache_pressure 10 I often get dirs get Best Alex

I can update the logrotate on next release adding the missingok. I have attached a suggested logrotate-file for cachedirs So first scan went smooth, everythings cached, we do lightning fast scans. Then suddenly the scan takes longer than expected and the scan is killed. Default timeout for stable scan is the 30s you have in the log file. 'depth 9999(timeout 30s:Error=1)' means infinite depth, but scan timed out after 30s (ie. we killed the process). The third scan was reduced to depth 4. That supprises me that it went that far down, but I have made it adaptive based on number of files, 117348 @ depth 4 vs 214446 at depth infinite. You can try disabling adaptive scan in settings, if you so please. I experience this behaviour when other stuff puts a load on the system (CPU), or when cache is lost. cache_dirs

Great. I just pushed yet another plugin version 2018.11.01-1, which adds chmod 775 to more stuff in the scripts folder pr @dmacias suggestion, there are some other stuff there like logrotate and bin.

@alturismo I answered the idle just one above, maybe I posted as you were writing. suc/fail: just ignore it. Its for adaptive. Its some counters saying how many of the last scans than completed completely or took to long so the script killed the scans (timeout params). Your logs look good.

I have now pushed a new plugin version 2018.11.01 which fixes the permission problem on cache_dirs script, so cache_dirs -L works. Its linked here so it should be directly executable from any bash shell prompt by default, without specifying path: /usr/local/bin/cache_dirs

@alturismo Looks good. 9999 is my internal way of representing infinite depth in the cach-dirs bash script. Disks idle before/after 786s/786s This means its not reading on your disks and nothing else is, because they are idle before and after a scan. filecount[9999]=18753 With only 18753 files, fixed depth is the way to go. Also your cache-pressure of 1 would be perfect for you I suspect. By the way, you didn't include user-share. I don't either, but if your disks spin up, include it and see if it helps.

Oh yeah, of cause. That's also how I link the plugin to its update github repository. I just forgot that. @markswiftDoes this also solve your installation problem?

Regarding status stopped after reboot: Send me a unraid diagnostics, and also do a cache_dirs -L, and send me the logs it collects. Before that run it with logs -enabled. Its likely there will be enough info for me in the syslog, so if you are in a hurry just send that. I'm off to bed. I'll look at it in the next couple of days. I did include a change from the broken 2018.10.14 version in the recent two plugins, maybe that is causing the restart issues, but I also added logging so the syslog should inform me. The indicator worked for Fireball3 when he had issues, so I suspect it does. The cache_dirs -L gives me all that info as to whether its actually running now, and more detailed info.

Good to know you've never installed it before. Are you on unRaid 6.4.0? The plugin says that's required, though I don't really know why, maybe something about the plugin and mdstat command. Are you installing via WebGui > Plugins > Install Plugin and then entered the url https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg I just tried that after removing the plugin first, and that went into an infinite loop, something about plugin: installing: https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg plugin: downloading https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg plugin: downloading: https://github.com/arberg/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg ... done Warning: simplexml_load_file(): /tmp/plugins/dynamix.cache.dirs.plg:44: parser error : Specification mandates value for attribute data-pjax-transient in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 216 Warning: simplexml_load_file(): name="request-id" content="C47A:2F90:F2F10:1CC859:5BD8BE3A" data-pjax-transient in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 216 Warning: simplexml_load_file(): ^ in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 216 Warning: simplexml_load_file(): /tmp/plugins/dynamix.cache.dirs.plg:49: parser error : Specification mandates value for attribute data-pjax-transient in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 216 Possibly that problem came for me because there were leftovers from the previous install. Try downloading the plg to /boot/config/plugins named dynamix.cache.dirs.plg and reboot

@markswiftPlease test again and let me know if it still fails.