Alex R. Berg

How do I make a safe array stop, reboot, shutdown in unRaid 6 from terminal? The stop-array here does not work for me anymore (maybe due to my btrf dual drive cache?): https://wiki.lime-technology.com/How-To's When I inspect WebGui's reboot /usr/local/emhttp/plugins/dynamix/include/Boot.php it seems it runs either (but I'm not certain I read the code correctly) /sbin/reboot /sbin/poweroff Is it safe to run these to get proper shutdown of array? In this old post 2014 by dlandon it says DON'T use reboot, but I'm guessing that is deprecated: How can I stop array like in 'good' old days when this don't work: https://wiki.lime-technology.com/Console#To_cleanly_Stop_the_array_from_the_command_line /usr/local/sbin/emhttp_event stopping_scvs /usr/local/sbin/emhttp_event unmounting_disks /usr/local/sbin/samba stop for i in /dev/md* do echo umount ${i} /sbin/umount ${i} done /usr/local/sbin/mdcmd stop In the above 'mdcmd stop' gives error. I have attached my arraystop script which does the above and a bit more. arraystop

Hi Zin105, You can limit the cache_dirs to only scan some shares, and you can also limit it to only scan to a certain depth. I have attached a program you can use to count where the bulk of your files are, it runs the following command at various depth find $f -type f -maxdepth ${DEPTH} | wc -l You can also you my modified cache-dirs which I link at this post: I modified it to stop scanning disks forever, if it cannot have files in cache, it now adaptively adjust its scan-depth according to how much it can have in memory, which is judged by the scan duration. When my server is pressed with other stuff, cache_dirs would lose its cache, and start scanning, causing my server to be even more pressed, which seemed silly. My adjustments makes cache_dirs scan only to a low depth in such a pressed situation and then stop after a couple of minutes attempt. You are welcome to try it out, but don't expect support, because I'm to busy elsewhere... But good luck with whatever you choose. cache_dirs_file_count

Ha! I think you need glasses if that's all the change you see But I totally get the need for a better description than the change-log. I don't have the time right now to go into details and I don't remember everything I did but this is what I wrote earlier: I have added adaptive depth level, to prevent cache_dirs from thrashing disks when they are otherwise occupied and cache is evicted. I found the cache was often evicted with the number of files I had when system become occupied with other things. I added the ability to adjust depth automatically based on whether scans are judged to cause disk access or not. It judges that a disk has been accessed during scan if scan takes a long time or if any recent disk access was made (and no recent disk access was made before scanning). The purpose being to avoid the situations where cache_dirs will continuously search through my files keeping disks busy all the time. Before it was also rather difficult to tell if cache_dirs was 'trashing' my disks, now its quite clear from the log if logging is enabled (though the log is rather large at the moment). If disks are kept spinning for some consecutive scans, the depth is decreased, and future rescan is scheduled at higher depth. If the file '/var/log/cache_dirs_lost_cache.log' exists then it will write a log that is easily imported into spreadsheet (excel) so its easier to check whether it trashes disks with current settings. I also added the kill I mentioned and some other quite minor bug-fixes. If you need more let me know, and I might supply more detail over christmas. If you think it looks good and useful I might do a clean up run on the script. I havn't felt like spending more time on the script if nobody but me used it. Best Alex

The updated version of the script which I have made fixes this problem. The script is available here: https://github.com/arberg/unRAID By now I'm guessing Joe won't incorporate the changes I have made into the main line. I've made lots of changes, so if all you are interested in is the child-killing ability you may have to extract that change-set yourself. Best Alex

I have updated my extended version of the cache dirs. To spike your memory: In my version I added adaptive depth level, to prevent cache_dirs from thrashing disks when they are otherwise occupied and cache is evicted. In the new version I bugfixed and improved it and tested on x64. I think I should publish it under another name such as cache_dirs_adaptive if Peter does not have time or the desire to check/approve/edit the changes. Do you agree? Best Alex

I think its probably more like 2-4Gb (maybe 4GB if the whole 32bit can be used and are not used elsewhere). I once tested memory usage of cachedyrs on x86 unRaid by filling ram by placing huge file on / and if my memory serves me it was back then more than 1GB cache_dirs consumed, though I'm not sure. But anyway, exact numbers are irrelevant I guess.

I also have too many files for unRaid 5 (32bit) to cache all files, hence the motivation for my changes. unRaid 6 with x64 and thus capability of using more memory it can now cache all files. I have (and had on unRaid 5) 16GB ram. I have 921091 files. Best Alex

I don't know the unit, but 5000 is atmost 2mb as that was the max on 32 bit os, and I'm pretty sure its not too much. Though you can try cache_dirs -w -U 5000 -d 5 for limited to max depth 5 (you can try different depths). Do you have filled up your root filesystem, unRaids filesystem runs in memory, so you can exhaust your memory as far as I know. 'df' will not tell you root storage it seems. You can try du -hs /* (if you unmount the array it wont include you files). It might also be a plugin which installs some bad libraries, but that sounds unlikely. Otherwise I'm lost and have no further ideas for you. Best Alex

I'm guessing its caused by out of memory. If you use x64 OS (unRaid 6) then cache_dirs is able to consume more memory, and defaults at a relatively large buffer. Possibly try reducing buffer by using param -U, see help from cachedirs ie this line: echo " -U NN = set ulimit to NN to limit memory used (default=5000 on 32 bit, 50000 on 64 bit OS, '-U 0' sets no ulimit at all)" best Alex

I had 8Gb ram. when I was on unRaid 5 with only 32bits I could use depth 8 or 9, and any deeper and cache_dirs just kept my disks spinning and continously scanned drives, because cache was to small to keep all dirs in memory. on unRaid 6 I can go to depth 14. Its solely dependent on how many files you've got in that depth. If you have x64 you can probably just keep it on default. Best Alex

I have made some modifications to the cache_dirs script. I added the ability to adjust depth automatically based on whether scans are judged to cause disk access or not. It judges that a disk has been accessed during scan if scan takes a long time or if any recent disk access was made (and no recent disk access was made before scanning). The purpose being to avoid the situations where cache_dirs will continuously search through my files keeping disks busy all the time. Before it was also rather difficult to tell if cache_dirs was 'trashing' my disks, now its quite clear from the log if logging is enabled (though the log is rather large at the moment). If disks are kept spinning for some consequtive scans, the depth is decreased, and future rescan is scheduled at higher depth. I havn't published it anywhere yet (except for a new public github account which I havn't shared yet), I wanted to hear you you Joe on how to proceed. You have been the creator and maestro of cache_dirs till now it appears with no messing around in your codebase, so I'm hesistant to share it before I get your approval. It would also be nice to just continue to have one version if you agree/accept on the changes. Let me know if I should PM you a link, or just upload it here or what you think. Best Alex PS: I believe the changes are much less useful for x64 users with huge amounts of ram, but I'm still on x86 hence the sudden desire to implement my ideas.