dlandon Posted December 12, 2015 Share Posted December 12, 2015 The updated version of the script which I have made fixes this problem. The script is available here: https://github.com/arberg/unRAID By now I'm guessing Joe won't incorporate the changes I have made into the main line. I've made lots of changes, so if all you are interested in is the child-killing ability you may have to extract that change-set yourself. Best Alex Yes, I think Joe L. has moved on. What changes have you made to the cache_dirs script and are there any incompatibilities to the original script? EDIT: So far all I see is that you changed the 'B' to 'b' for disks busy. Quote Link to comment
Alex R. Berg Posted December 13, 2015 Share Posted December 13, 2015 Yes, I think Joe L. has moved on. What changes have you made to the cache_dirs script and are there any incompatibilities to the original script? EDIT: So far all I see is that you changed the 'B' to 'b' for disks busy. Ha! I think you need glasses if that's all the change you see But I totally get the need for a better description than the change-log. I don't have the time right now to go into details and I don't remember everything I did but this is what I wrote earlier: I have added adaptive depth level, to prevent cache_dirs from thrashing disks when they are otherwise occupied and cache is evicted. I found the cache was often evicted with the number of files I had when system become occupied with other things. I added the ability to adjust depth automatically based on whether scans are judged to cause disk access or not. It judges that a disk has been accessed during scan if scan takes a long time or if any recent disk access was made (and no recent disk access was made before scanning). The purpose being to avoid the situations where cache_dirs will continuously search through my files keeping disks busy all the time. Before it was also rather difficult to tell if cache_dirs was 'trashing' my disks, now its quite clear from the log if logging is enabled (though the log is rather large at the moment). If disks are kept spinning for some consecutive scans, the depth is decreased, and future rescan is scheduled at higher depth. If the file '/var/log/cache_dirs_lost_cache.log' exists then it will write a log that is easily imported into spreadsheet (excel) so its easier to check whether it trashes disks with current settings. I also added the kill I mentioned and some other quite minor bug-fixes. If you need more let me know, and I might supply more detail over christmas. If you think it looks good and useful I might do a clean up run on the script. I havn't felt like spending more time on the script if nobody but me used it. Best Alex Quote Link to comment
dlandon Posted December 13, 2015 Share Posted December 13, 2015 Yes, I think Joe L. has moved on. What changes have you made to the cache_dirs script and are there any incompatibilities to the original script? EDIT: So far all I see is that you changed the 'B' to 'b' for disks busy. Ha! I think you need glasses if that's all the change you see But I totally get the need for a better description than the change-log. I don't have the time right now to go into details and I don't remember everything I did but this is what I wrote earlier: I have added adaptive depth level, to prevent cache_dirs from thrashing disks when they are otherwise occupied and cache is evicted. I found the cache was often evicted with the number of files I had when system become occupied with other things. I added the ability to adjust depth automatically based on whether scans are judged to cause disk access or not. It judges that a disk has been accessed during scan if scan takes a long time or if any recent disk access was made (and no recent disk access was made before scanning). The purpose being to avoid the situations where cache_dirs will continuously search through my files keeping disks busy all the time. Before it was also rather difficult to tell if cache_dirs was 'trashing' my disks, now its quite clear from the log if logging is enabled (though the log is rather large at the moment). If disks are kept spinning for some consecutive scans, the depth is decreased, and future rescan is scheduled at higher depth. If the file '/var/log/cache_dirs_lost_cache.log' exists then it will write a log that is easily imported into spreadsheet (excel) so its easier to check whether it trashes disks with current settings. I also added the kill I mentioned and some other quite minor bug-fixes. If you need more let me know, and I might supply more detail over christmas. If you think it looks good and useful I might do a clean up run on the script. I havn't felt like spending more time on the script if nobody but me used it. Best Alex I did end up taking the time to look through the change log that you added. I saw the changes you made noted in the change log. I should have done that right away. I have put your revised cache_dirs script on a github repository and bonienl has updated the dynamix cache_dirs plugin to install your modified script. It is working awesome on my server! Seems to have lightened the load on the server and it is killed immediately as it should. I'm not sure I need more detail. I would be happy to add more detail to the repository. I think I'll add the notes you just gave me to the README.md on the github for documentation. Quote Link to comment
Joe L. Posted December 13, 2015 Author Share Posted December 13, 2015 Yes, I think Joe L. has moved on. What changes have you made to the cache_dirs script and are there any incompatibilities to the original script? EDIT: So far all I see is that you changed the 'B' to 'b' for disks busy. Ha! I think you need glasses if that's all the change you see But I totally get the need for a better description than the change-log. I don't have the time right now to go into details and I don't remember everything I did but this is what I wrote earlier: I have added adaptive depth level, to prevent cache_dirs from thrashing disks when they are otherwise occupied and cache is evicted. I found the cache was often evicted with the number of files I had when system become occupied with other things. I added the ability to adjust depth automatically based on whether scans are judged to cause disk access or not. It judges that a disk has been accessed during scan if scan takes a long time or if any recent disk access was made (and no recent disk access was made before scanning). The purpose being to avoid the situations where cache_dirs will continuously search through my files keeping disks busy all the time. Before it was also rather difficult to tell if cache_dirs was 'trashing' my disks, now its quite clear from the log if logging is enabled (though the log is rather large at the moment). If disks are kept spinning for some consecutive scans, the depth is decreased, and future rescan is scheduled at higher depth. If the file '/var/log/cache_dirs_lost_cache.log' exists then it will write a log that is easily imported into spreadsheet (excel) so its easier to check whether it trashes disks with current settings. I also added the kill I mentioned and some other quite minor bug-fixes. If you need more let me know, and I might supply more detail over christmas. If you think it looks good and useful I might do a clean up run on the script. I havn't felt like spending more time on the script if nobody but me used it. Best Alex no, not moved on... Just have precious free time to be as heavily involved as I was a few years ago. (when I was not working.) My servers are both built with out-dated hardware. I cannot contribute in the same way I did in the past. (One is an original server sold by Limetech, with IDE based drives, the second newer, but incapable to handle virtualization) I do follow the threads... and respond occasionally... Joe L. Quote Link to comment
dlandon Posted December 14, 2015 Share Posted December 14, 2015 Yes, I think Joe L. has moved on. What changes have you made to the cache_dirs script and are there any incompatibilities to the original script? EDIT: So far all I see is that you changed the 'B' to 'b' for disks busy. If you need more let me know, and I might supply more detail over christmas. If you think it looks good and useful I might do a clean up run on the script. I havn't felt like spending more time on the script if nobody but me used it. Best Alex If you have some time, could you review the script and let me know if it needs any changes, or let me know any areas that need reviewing. Your modified script is now being used in the Dynamix Cache Dirs plugin, so it is being used by more people than just you. I will also do a full code review when I get some time. Thanks for your good work. Your updated cache_dirs script is working great for me! Quote Link to comment
stealth82 Posted December 30, 2015 Share Posted December 30, 2015 I've not read the whole thread, sorry, but the use of this plugin seems pretty straightforward to me. Either I'm misinterpreting its use or something is wrong. This is the configuration: Dec 28 21:20:05 Tower cache_dirs: ============================================== Dec 28 21:20:05 Tower cache_dirs: Starting cache_dirs: Dec 28 21:20:05 Tower cache_dirs: Arguments=-d 10 -e docker -e music -e torrents -e vm Dec 28 21:20:05 Tower cache_dirs: Cache Pressure=10 Dec 28 21:20:05 Tower cache_dirs: Max Scan Secs=10, Min Scan Secs=1 Dec 28 21:20:05 Tower cache_dirs: Max Scan Depth=10 Dec 28 21:20:05 Tower cache_dirs: Use Command='find -noleaf' Dec 28 21:20:05 Tower cache_dirs: Version=2.1.0 Dec 28 21:20:05 Tower cache_dirs: ---------- Caching Directories --------------- Dec 28 21:20:05 Tower cache_dirs: downloads Dec 28 21:20:05 Tower cache_dirs: home Dec 28 21:20:05 Tower cache_dirs: movies Dec 28 21:20:05 Tower cache_dirs: photos Dec 28 21:20:05 Tower cache_dirs: software Dec 28 21:20:05 Tower cache_dirs: tvshows Dec 28 21:20:05 Tower cache_dirs: videos Dec 28 21:20:05 Tower cache_dirs: ---------------------------------------------- Dec 28 21:20:05 Tower cache_dirs: cache_dirs process ID 12016 started and the process root 12016 1 0 Dec28 ? 00:06:39 /bin/bash /usr/local/emhttp/plugins/dynamix.cache.dirs/scripts/cache_dirs -d 10 -e docker -e music -e torrents -e vm This should cover a simple share browsing down to the following depth without spinning up a disk, right? /mnt/user/movies/1991/Fried Green Tomatoes (1991) But the disk having that folder is spinning up anyway. Where's my mistake? Quote Link to comment
bonienl Posted December 30, 2015 Share Posted December 30, 2015 Programs which read more than just directory information, i.e. file content, will cause the drive to spin up. How are you browsing your folders? Quote Link to comment
bonienl Posted December 30, 2015 Share Posted December 30, 2015 ssh what command do you use? btw your set up is correct, so caching of those folders should take place (unless there isn't enough RAM memory). Quote Link to comment
stealth82 Posted December 30, 2015 Share Posted December 30, 2015 I'm just using cd and I guess it's the same thing through programs like WinSCP. I have 8GB of RAM and unRAID tells me only 20% is used. What I can do is to spin down the disk after it was spun up. I browse the folder again, launch a ls command and this time the disk doesn't spin up. Quote Link to comment
methanoid Posted January 7, 2016 Share Posted January 7, 2016 I cannot understand why this feature is not in UnRAID as standard by now.... Quote Link to comment
garycase Posted January 7, 2016 Share Posted January 7, 2016 I cannot understand why this feature is not in UnRAID as standard by now.... There's a plugin that makes it very simple to use the feature. Go to the Plugins page; click on Install plugin; and copy the following line into the URL blank ... then click Install. https://raw.github.com/bergware/dynamix/master/unRAIDv6/dynamix.cache.dirs.plg Quote Link to comment
segator Posted March 28, 2016 Share Posted March 28, 2016 Hi guys, I have this plugin installed but it seems that not work so good for me. I hear the drive every few seconds working and if I have look and the drive is spin down every time that I acced to a smb directory the disks spin up this is the log Mar 28 11:19:12 segator cache_dirs: Starting cache_dirs: Mar 28 11:19:12 segator cache_dirs: Arguments=-u -m 2 -M 5 -d 999 -i aina -i aportes -i encodings -i home\ video -i movies -i music -i photos -i segator -i software -i stcugat -i steamLibrary -i temporal -$ Mar 28 11:19:12 segator cache_dirs: Cache Pressure=10 Mar 28 11:19:12 segator cache_dirs: Max Scan Secs=5, Min Scan Secs=2 Mar 28 11:19:12 segator cache_dirs: Scan Type=adaptive Mar 28 11:19:12 segator cache_dirs: Max Scan Depth=999 Mar 28 11:19:12 segator cache_dirs: Use Command='find -noleaf' Mar 28 11:19:12 segator cache_dirs: Version=2.1.1 Mar 28 11:19:12 segator cache_dirs: ---------- Caching Directories --------------- Mar 28 11:19:12 segator cache_dirs: aina Mar 28 11:19:12 segator cache_dirs: aportes Mar 28 11:19:12 segator cache_dirs: encodings Mar 28 11:19:12 segator cache_dirs: home video Mar 28 11:19:12 segator cache_dirs: movies Mar 28 11:19:12 segator cache_dirs: music Mar 28 11:19:12 segator cache_dirs: photos Mar 28 11:19:12 segator cache_dirs: segator Mar 28 11:19:12 segator cache_dirs: software Mar 28 11:19:12 segator cache_dirs: stcugat Mar 28 11:19:12 segator cache_dirs: steamLibrary Mar 28 11:19:12 segator cache_dirs: temporal Mar 28 11:19:12 segator cache_dirs: tvshow Quote Link to comment
itimpi Posted March 28, 2016 Share Posted March 28, 2016 I wonder if the issue is that you simply do not have enough RAM to keep all your directory information in RAM? Quote Link to comment
segator Posted March 28, 2016 Share Posted March 28, 2016 I wonder if the issue is that you simply do not have enough RAM to keep all your directory information in RAM? Well I checked while cache dirs is running and I have 3,5GB free acording to free -m command information Quote Link to comment
BRiT Posted March 28, 2016 Share Posted March 28, 2016 I wonder if the issue is that you simply do not have enough RAM to keep all your directory information in RAM? Well I checked while cache dirs is running and I have 3,5GB free acording to free -m command information If you're using windows explorer to check, be aware that it will inspect the files which is not part of what is cached, just the directory entries. You need to use a file client that wont inspect the files themselves. The best way to check if a disk directory is part of the cache is using a linux command shell and typing "ls /mnt/user/share/dir/subdir" or "ls /mnt/disk#/share/dir/subdir". If the disk is spundown and cached by cachedirs it should not spin up using that command. Quote Link to comment
segator Posted March 30, 2016 Share Posted March 30, 2016 I wonder if the issue is that you simply do not have enough RAM to keep all your directory information in RAM? Well I checked while cache dirs is running and I have 3,5GB free acording to free -m command information If you're using windows explorer to check, be aware that it will inspect the files which is not part of what is cached, just the directory entries. You need to use a file client that wont inspect the files themselves. The best way to check if a disk directory is part of the cache is using a linux command shell and typing "ls /mnt/user/share/dir/subdir" or "ls /mnt/disk#/share/dir/subdir". If the disk is spundown and cached by cachedirs it should not spin up using that command. You're right, I have a binded smb directory to my windows computer, it's seems it's the problem. When i Start my computer the NAS drives automatically spin up. Using NFS it work fine, it's possible to use my cache drive as a read cache? something like NFS cachefsd that works very well I have read cached 1TB of my most viewed media in one server with plex installed. Quote Link to comment
cookiemole Posted June 10, 2016 Share Posted June 10, 2016 I am also having problems with Plex not automatically detecting directory changes when a new movie is added. (I'm running Plex through a docker). However, it does allow me to use Plex's periodic file scan without spinning up the disks. This happens much faster now because it just searches through the list of files in RAM. Could this plugin be intercepting and clearing the "something has changed" flag before Plex has a chance to see it? Quote Link to comment
jeffreywhunter Posted June 22, 2016 Share Posted June 22, 2016 I've installed this app and it says its running, if drives are all spun down and I do an SMB directory list/search, all the drives spin up...and it takes a long time. Unraid 6.1.9 System details and Free -m http://my.jetscreenshot.com/12412/20160622-qbsb-125kb.jpg[/img] I've not been able to find a console other than the config: http://my.jetscreenshot.com/12412/20160622-baq8-40kb.jpg[/img] Pretty much default install except to scan user shares. Is there a console or some status we can get from the system other than "Running" on the config screen? Quote Link to comment
RobJ Posted June 22, 2016 Share Posted June 22, 2016 It's not usually necessary to cache the User Shares, since they are already in memory. And I strongly recommend setting the Included folders, include only those shares and folders you actually want cached. I have additional comments about the CacheDirs plugin in the Upgrading to v6, Dynamix Plugins section. Quote Link to comment
kocane Posted July 22, 2016 Share Posted July 22, 2016 Any trouble in using this script on a regular Debian installation? I wanna keep Sonarr from spinning up my HDDs. I could imagine the script would need quite some modification ? Quote Link to comment
Ropo Posted November 13, 2016 Share Posted November 13, 2016 Hi, I use plugin "Dynamix Cache Directories" (dynamix.cache.dirs from 2016.08.26) in upgraded my Unraid server to version 6.2.4. I install new installation of Unraid on new USB flash disk. I copied older config files from my old version 5.0. And install some plugins. But if I enable folder caching in plugin section then folder caching not working. I found http://lime-technology.com/forum/index.php?topic=22518.msg199655#msg199655 that in *.cfg is poorly written this configuration: In /boot/config ANYSHARE.cfg: shareUseCache="no" But I think that this config must modify the plugin configuration of "Dynamix Cache Directories". But the plugin not do it. Where is a problem that if I change configuration of plugin "Dynamix Cache Directories" and caching feature not working? I must manually modify ANYSHARE.cfg. Many thanks for answer Quote Link to comment
RobJ Posted November 14, 2016 Share Posted November 14, 2016 I use plugin "Dynamix Cache Directories" (dynamix.cache.dirs from 2016.08.26) in upgraded my Unraid server to version 6.2.4. I install new installation of Unraid on new USB flash disk. I copied older config files from my old version 5.0. And install some plugins. Hopefully, you only installed v6 plugins, none of the v5 plugins will work. But if I enable folder caching in plugin section then folder caching not working. I found http://lime-technology.com/forum/index.php?topic=22518.msg199655#msg199655 that in *.cfg is poorly written this configuration: In /boot/config ANYSHARE.cfg: shareUseCache="no" But I think that this config must modify the plugin configuration of "Dynamix Cache Directories". But the plugin not do it. Where is a problem that if I change configuration of plugin "Dynamix Cache Directories" and caching feature not working? I must manually modify ANYSHARE.cfg. Respectfully, you have 2 things mixed up, the Cache disk and its usage in User Shares, and caching directory entries using the CacheDirs plugin. The Dynamix Cache Directories plugin has absolutely nothing to do with the Cache disk or shares. Its purpose is to keep directory entries in RAM so drives don't have to keep spinning up. The Cache disk is used with shares to speed up writes to those shares (plus other uses). To change share settings, you go to the Global Share Settings, and the Shares Settings for the individual shares. There, you can change the Cache disk usage for each share. Quote Link to comment
RobJ Posted November 14, 2016 Share Posted November 14, 2016 Any trouble in using this script on a regular Debian installation? I wanna keep Sonarr from spinning up my HDDs. I could imagine the script would need quite some modification ? Belatedly responding, in case you ever come back ... You would need to modify it, as it's designed specifically for unRAID array data disks. Might not be hard, but I haven't looked at it in awhile. You would need to tear out the unRAID drive and share paths, and insert your own way of finding your drives. Quote Link to comment
Ropo Posted November 14, 2016 Share Posted November 14, 2016 Hopefully, you only installed v6 plugins, none of the v5 plugins will work. I don't used plugins of the version v5. I use only the v6 plugins. Its purpose is to keep directory entries in RAM so drives don't have to keep spinning up. Aha, I understand! Many thanks for this sentance. "Dynamix Cache Directories" only keeps directory structure of users shares in directory entries on cache drive. Copy structure of users share. The Cache disk is used with shares to speed up writes to those shares (plus other uses). To change share settings, you go to the Global Share Settings, and the Shares Settings for the individual shares. There, you can change the Cache disk usage for each share. OK. I check it out. I want speed up writes to shares. How is the name of "secret" directory on cache disk which don't move out from cache disk by the mover? I want directory entry which will stay on cache disk and not will moved to shares. I found settings for users share -> "use cache disk". Can I set this opt to "Prefer" or "Only" to prevention move the share-directory to users share? RobJ many thanks. Wish a nice day. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.