Jump to content
Joe L.

cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up

1106 posts in this topic Last Reply

Recommended Posts

The updated version of the script which I have made fixes this problem. The script is available here: https://github.com/arberg/unRAID

 

By now I'm guessing Joe won't incorporate the changes I have made into the main line. I've made lots of changes, so if all you are interested in is the child-killing ability you may have to extract that change-set yourself.

 

Best Alex

 

Yes, I think Joe L. has moved on.  What changes have you made to the cache_dirs script and are there any incompatibilities to the original script?

 

EDIT: So far all I see is that you changed the 'B' to 'b' for disks busy.

Share this post


Link to post

Yes, I think Joe L. has moved on.  What changes have you made to the cache_dirs script and are there any incompatibilities to the original script?

 

EDIT: So far all I see is that you changed the 'B' to 'b' for disks busy.

 

Ha! I think you need glasses if that's all the change you see :) But I totally get the need for a better description than the change-log.

 

I don't have the time right now to go into details and I don't remember everything I did but this is what I wrote earlier:

 

I have added adaptive depth level, to prevent cache_dirs from thrashing disks when they are otherwise occupied and cache is evicted. I found the cache was often evicted with the number of files I had when system become occupied with other things.

I added the ability to adjust depth automatically based on whether scans are judged to cause disk access or not. It judges that a disk has been accessed during scan if scan takes a long time or if any recent disk access was made (and no recent disk access was made before scanning). The purpose being to avoid the situations where cache_dirs will continuously search through my files keeping disks busy all the time. Before it was also rather difficult to tell if cache_dirs was 'trashing' my disks, now its quite clear from the log if logging is enabled (though the log is rather large at the moment). If disks are kept spinning for some consecutive scans, the depth is decreased, and future rescan is scheduled at higher depth.

 

If the file '/var/log/cache_dirs_lost_cache.log' exists then it will write a log that is easily imported into spreadsheet (excel) so its easier to check whether it trashes disks with current settings. I also added the kill I mentioned and some other quite minor bug-fixes.

 

If you need more let me know, and I might supply more detail over christmas. If you think it looks good and useful I might do a clean up run on the script. I havn't felt like spending more time on the script if nobody but me used it.

 

Best Alex

Share this post


Link to post

Yes, I think Joe L. has moved on.  What changes have you made to the cache_dirs script and are there any incompatibilities to the original script?

 

EDIT: So far all I see is that you changed the 'B' to 'b' for disks busy.

 

Ha! I think you need glasses if that's all the change you see :) But I totally get the need for a better description than the change-log.

 

I don't have the time right now to go into details and I don't remember everything I did but this is what I wrote earlier:

 

I have added adaptive depth level, to prevent cache_dirs from thrashing disks when they are otherwise occupied and cache is evicted. I found the cache was often evicted with the number of files I had when system become occupied with other things.

I added the ability to adjust depth automatically based on whether scans are judged to cause disk access or not. It judges that a disk has been accessed during scan if scan takes a long time or if any recent disk access was made (and no recent disk access was made before scanning). The purpose being to avoid the situations where cache_dirs will continuously search through my files keeping disks busy all the time. Before it was also rather difficult to tell if cache_dirs was 'trashing' my disks, now its quite clear from the log if logging is enabled (though the log is rather large at the moment). If disks are kept spinning for some consecutive scans, the depth is decreased, and future rescan is scheduled at higher depth.

 

If the file '/var/log/cache_dirs_lost_cache.log' exists then it will write a log that is easily imported into spreadsheet (excel) so its easier to check whether it trashes disks with current settings. I also added the kill I mentioned and some other quite minor bug-fixes.

 

If you need more let me know, and I might supply more detail over christmas. If you think it looks good and useful I might do a clean up run on the script. I havn't felt like spending more time on the script if nobody but me used it.

 

Best Alex

 

I did end up taking the time to look through the change log that you added.  I saw the changes you made noted in the change log.  I should have done that right away.

 

I have put your revised cache_dirs script on a github repository and bonienl has updated the dynamix cache_dirs plugin to install your modified script.  It is working awesome on my server!  Seems to have lightened the load on the server and it is killed immediately as it should.

 

I'm not sure I need more detail.  I would be happy to add more detail to the repository.  I think I'll add the notes you just gave me to the README.md on the github for documentation.

Share this post


Link to post

Yes, I think Joe L. has moved on.  What changes have you made to the cache_dirs script and are there any incompatibilities to the original script?

 

EDIT: So far all I see is that you changed the 'B' to 'b' for disks busy.

 

Ha! I think you need glasses if that's all the change you see :) But I totally get the need for a better description than the change-log.

 

I don't have the time right now to go into details and I don't remember everything I did but this is what I wrote earlier:

 

I have added adaptive depth level, to prevent cache_dirs from thrashing disks when they are otherwise occupied and cache is evicted. I found the cache was often evicted with the number of files I had when system become occupied with other things.

I added the ability to adjust depth automatically based on whether scans are judged to cause disk access or not. It judges that a disk has been accessed during scan if scan takes a long time or if any recent disk access was made (and no recent disk access was made before scanning). The purpose being to avoid the situations where cache_dirs will continuously search through my files keeping disks busy all the time. Before it was also rather difficult to tell if cache_dirs was 'trashing' my disks, now its quite clear from the log if logging is enabled (though the log is rather large at the moment). If disks are kept spinning for some consecutive scans, the depth is decreased, and future rescan is scheduled at higher depth.

 

If the file '/var/log/cache_dirs_lost_cache.log' exists then it will write a log that is easily imported into spreadsheet (excel) so its easier to check whether it trashes disks with current settings. I also added the kill I mentioned and some other quite minor bug-fixes.

 

If you need more let me know, and I might supply more detail over christmas. If you think it looks good and useful I might do a clean up run on the script. I havn't felt like spending more time on the script if nobody but me used it.

 

Best Alex

no, not moved on... Just have precious free time to be as heavily involved as I was a few years ago.  (when I was not working.) 

My servers are both built with out-dated hardware.  I cannot contribute in the same way I did in the past.  (One is an original server sold by Limetech, with IDE based drives, the second newer, but incapable to handle virtualization)

 

I do follow the threads... and respond occasionally...

 

Joe L.

Share this post


Link to post

Yes, I think Joe L. has moved on.  What changes have you made to the cache_dirs script and are there any incompatibilities to the original script?

 

EDIT: So far all I see is that you changed the 'B' to 'b' for disks busy.

If you need more let me know, and I might supply more detail over christmas. If you think it looks good and useful I might do a clean up run on the script. I havn't felt like spending more time on the script if nobody but me used it.

 

Best Alex

 

If you have some time, could you review the script and let me know if it needs any changes, or let me know any areas that need reviewing.  Your modified script is now being used in the Dynamix Cache Dirs plugin, so it is being used by more people than just you.

 

I will also do a full code review when I get some time.

 

Thanks for your good work.  Your updated cache_dirs script is working great for me!

Share this post


Link to post

I've not read the whole thread, sorry, but the use of this plugin seems pretty straightforward to me.

Either I'm misinterpreting its use or something is wrong.

 

This is the configuration:

Dec 28 21:20:05 Tower cache_dirs: ==============================================
Dec 28 21:20:05 Tower cache_dirs: Starting cache_dirs:
Dec 28 21:20:05 Tower cache_dirs: Arguments=-d 10 -e docker -e music -e torrents -e vm
Dec 28 21:20:05 Tower cache_dirs: Cache Pressure=10
Dec 28 21:20:05 Tower cache_dirs: Max Scan Secs=10, Min Scan Secs=1
Dec 28 21:20:05 Tower cache_dirs: Max Scan Depth=10
Dec 28 21:20:05 Tower cache_dirs: Use Command='find -noleaf'
Dec 28 21:20:05 Tower cache_dirs: Version=2.1.0
Dec 28 21:20:05 Tower cache_dirs: ---------- Caching Directories ---------------
Dec 28 21:20:05 Tower cache_dirs: downloads
Dec 28 21:20:05 Tower cache_dirs: home
Dec 28 21:20:05 Tower cache_dirs: movies
Dec 28 21:20:05 Tower cache_dirs: photos
Dec 28 21:20:05 Tower cache_dirs: software
Dec 28 21:20:05 Tower cache_dirs: tvshows
Dec 28 21:20:05 Tower cache_dirs: videos
Dec 28 21:20:05 Tower cache_dirs: ----------------------------------------------
Dec 28 21:20:05 Tower cache_dirs: cache_dirs process ID 12016 started

 

and the process

 

root     12016     1  0 Dec28 ?        00:06:39 /bin/bash /usr/local/emhttp/plugins/dynamix.cache.dirs/scripts/cache_dirs -d 10 -e docker -e music -e torrents -e vm

 

This should cover a simple share browsing down to the following depth without spinning up a disk, right?

 

/mnt/user/movies/1991/Fried Green Tomatoes (1991)

 

But the disk having that folder is spinning up anyway.

 

Where's my mistake?

Share this post


Link to post

Programs which read more than just directory information, i.e. file content, will cause the drive to spin up.

 

How are you browsing your folders?

 

Share this post


Link to post

ssh

 

what command do you use?

 

btw your set up is correct, so caching of those folders should take place (unless there isn't enough RAM memory).

Share this post


Link to post

I'm just using cd and I guess it's the same thing through programs like WinSCP.

 

I have 8GB of RAM and unRAID tells me only 20% is used.

 

What I can do is to spin down the disk after it was spun up.

I browse the folder again, launch a ls command and this time the disk doesn't spin up.

Share this post


Link to post

Hi guys, I have this plugin installed but it seems that not work so good for me.

I hear the drive every few seconds working  and if I have look and the drive is spin down every time that I acced to a smb directory the disks spin up

 

this is the log

 

Mar 28 11:19:12 segator cache_dirs: Starting cache_dirs:

Mar 28 11:19:12 segator cache_dirs: Arguments=-u -m 2 -M 5 -d 999 -i aina -i aportes -i encodings -i home\ video -i movies -i music -i photos -i segator -i software -i stcugat -i steamLibrary -i temporal -$

Mar 28 11:19:12 segator cache_dirs: Cache Pressure=10

Mar 28 11:19:12 segator cache_dirs: Max Scan Secs=5, Min Scan Secs=2

Mar 28 11:19:12 segator cache_dirs: Scan Type=adaptive

Mar 28 11:19:12 segator cache_dirs: Max Scan Depth=999

Mar 28 11:19:12 segator cache_dirs: Use Command='find -noleaf'

Mar 28 11:19:12 segator cache_dirs: Version=2.1.1

Mar 28 11:19:12 segator cache_dirs: ---------- Caching Directories ---------------

Mar 28 11:19:12 segator cache_dirs: aina

Mar 28 11:19:12 segator cache_dirs: aportes

Mar 28 11:19:12 segator cache_dirs: encodings

Mar 28 11:19:12 segator cache_dirs: home video

Mar 28 11:19:12 segator cache_dirs: movies

Mar 28 11:19:12 segator cache_dirs: music

Mar 28 11:19:12 segator cache_dirs: photos

Mar 28 11:19:12 segator cache_dirs: segator

Mar 28 11:19:12 segator cache_dirs: software

Mar 28 11:19:12 segator cache_dirs: stcugat

Mar 28 11:19:12 segator cache_dirs: steamLibrary

Mar 28 11:19:12 segator cache_dirs: temporal

Mar 28 11:19:12 segator cache_dirs: tvshow

 

Share this post


Link to post

I wonder if the issue is that you simply do not have enough RAM to keep all your directory information in RAM? 

Share this post


Link to post

I wonder if the issue is that you simply do not have enough RAM to keep all your directory information in RAM?

Well I checked while cache dirs is running and I have 3,5GB free acording to free -m command information

Share this post


Link to post

I wonder if the issue is that you simply do not have enough RAM to keep all your directory information in RAM?

Well I checked while cache dirs is running and I have 3,5GB free acording to free -m command information

If you're using windows explorer to check, be aware that it will inspect the files which is not part of what is cached, just the directory entries. You need to use a file client that wont inspect the files themselves.

 

The best way to check if a disk directory is part of the cache is using a linux command shell and typing "ls /mnt/user/share/dir/subdir" or "ls /mnt/disk#/share/dir/subdir". If the disk is spundown and cached by cachedirs it should not spin up using that command.

Share this post


Link to post

I wonder if the issue is that you simply do not have enough RAM to keep all your directory information in RAM?

Well I checked while cache dirs is running and I have 3,5GB free acording to free -m command information

If you're using windows explorer to check, be aware that it will inspect the files which is not part of what is cached, just the directory entries. You need to use a file client that wont inspect the files themselves.

 

The best way to check if a disk directory is part of the cache is using a linux command shell and typing "ls /mnt/user/share/dir/subdir" or "ls /mnt/disk#/share/dir/subdir". If the disk is spundown and cached by cachedirs it should not spin up using that command.

 

You're right, I have a binded smb directory to my windows computer, it's seems it's the problem.

When i Start my computer the NAS drives automatically spin up.

 

Using NFS it work fine, it's possible to use my cache drive as a read cache?

 

something like NFS cachefsd that works very well I have read cached 1TB of my most viewed media in one server with plex installed.

 

Share this post


Link to post

I am also having problems with Plex not automatically detecting directory changes when a new movie is added. (I'm running Plex through a docker).

 

However, it does allow me to use Plex's periodic file scan without spinning up the disks. This happens much faster now because it just searches through the list of files in RAM.

 

Could this plugin be intercepting and clearing the "something has changed" flag before Plex has a chance to see it?

Share this post


Link to post

I've installed this app and it says its running, if drives are all spun down and I do an SMB directory list/search, all the drives spin up...and it takes a long time.

 

Unraid 6.1.9

System details and Free -m

width=300http://my.jetscreenshot.com/12412/20160622-qbsb-125kb.jpg[/img]

 

I've not been able to find a console other than the config:

width=300http://my.jetscreenshot.com/12412/20160622-baq8-40kb.jpg[/img]

 

Pretty much default install except to scan user shares.

 

Is there a console or some status we can get from the system other than "Running" on the config screen?

Share this post


Link to post

It's not usually necessary to cache the User Shares, since they are already in memory.  And I strongly recommend setting the Included folders, include only those shares and folders you actually want cached.  I have additional comments about the CacheDirs plugin in the Upgrading to v6, Dynamix Plugins section.

Share this post


Link to post

Any trouble in using this script on a regular Debian installation? I wanna keep Sonarr from spinning up my HDDs.

 

I could imagine the script would need quite some modification :/?

Share this post


Link to post

Hi,

 

I use plugin "Dynamix Cache Directories" (dynamix.cache.dirs from 2016.08.26) in upgraded my Unraid server to version 6.2.4.

I install new installation of Unraid on new USB flash disk. I copied older config files from my old version 5.0. And install some plugins.

 

But if I enable folder caching in plugin section then folder caching not working.

 

I found http://lime-technology.com/forum/index.php?topic=22518.msg199655#msg199655 that in *.cfg is poorly written this configuration:

 

In /boot/config ANYSHARE.cfg:

shareUseCache="no"

 

But I think that this config must modify the plugin configuration of "Dynamix Cache Directories". But the plugin not do it.

 

Where is a problem that if I change configuration of plugin "Dynamix Cache Directories" and caching feature not working?

I must manually modify ANYSHARE.cfg.

 

Many thanks for answer

Share this post


Link to post

I use plugin "Dynamix Cache Directories" (dynamix.cache.dirs from 2016.08.26) in upgraded my Unraid server to version 6.2.4.

I install new installation of Unraid on new USB flash disk. I copied older config files from my old version 5.0. And install some plugins.

Hopefully, you only installed v6 plugins, none of the v5 plugins will work.

 

But if I enable folder caching in plugin section then folder caching not working.

 

I found http://lime-technology.com/forum/index.php?topic=22518.msg199655#msg199655 that in *.cfg is poorly written this configuration:

 

In /boot/config ANYSHARE.cfg:

shareUseCache="no"

 

But I think that this config must modify the plugin configuration of "Dynamix Cache Directories". But the plugin not do it.

 

Where is a problem that if I change configuration of plugin "Dynamix Cache Directories" and caching feature not working?

I must manually modify ANYSHARE.cfg.

 

Respectfully, you have 2 things mixed up, the Cache disk and its usage in User Shares, and caching directory entries using the CacheDirs plugin.  The Dynamix Cache Directories plugin has absolutely nothing to do with the Cache disk or shares.  Its purpose is to keep directory entries in RAM so drives don't have to keep spinning up.

 

The Cache disk is used with shares to speed up writes to those shares (plus other uses).  To change share settings, you go to the Global Share Settings, and the Shares Settings for the individual shares.  There, you can change the Cache disk usage for each share.

Share this post


Link to post

Any trouble in using this script on a regular Debian installation? I wanna keep Sonarr from spinning up my HDDs.

 

I could imagine the script would need quite some modification :/?

 

Belatedly responding, in case you ever come back ...  ;)

 

You would need to modify it, as it's designed specifically for unRAID array data disks.  Might not be hard, but I haven't looked at it in awhile.  You would need to tear out the unRAID drive and share paths, and insert your own way of finding your drives.

Share this post


Link to post

Hopefully, you only installed v6 plugins, none of the v5 plugins will work.

I don't used plugins of the version v5. I use only the v6 plugins.

 

Its purpose is to keep directory entries in RAM so drives don't have to keep spinning up.

Aha, I understand! Many thanks for this sentance.

"Dynamix Cache Directories" only keeps directory structure of users shares in directory entries on cache drive. Copy structure of users share.

 

The Cache disk is used with shares to speed up writes to those shares (plus other uses).  To change share settings, you go to the Global Share Settings, and the Shares Settings for the individual shares.  There, you can change the Cache disk usage for each share.

 

OK. I check it out. I want speed up writes to shares.

 

How is the name of "secret" directory on cache disk which don't move out from cache disk by the mover? I want directory entry which will stay on cache disk and not will moved to shares.

I found settings for users share -> "use cache disk". Can I set this opt to "Prefer" or "Only" to prevention move the share-directory to users share?

 

RobJ many thanks. Wish a nice day.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.