Cache_dirs scan level depth "auto" not enough?


Recommended Posts

i have also been experiencing spin-ups (sometimes my entire 19-drive array, one at a time) with no apparent reason other than creating a new folder from a Windows client.  It seems to me to have possibly coincided with my version 6 upgrade, but I can't be sure.  This absolutely never happened in the past.  My array runs no dockers or any application.  I have 8G memory installed, and run very few plug-ins.  I also use a cache drive.  Cache_dirs used to work flawlessly for me from a Windows client perspective.  I would never wait for a spin-up when creating a new file or folder until several months ago.  Since then I have wracked my brains to get this resolved.  I even ground-zeroed and did a clean re-install of both my Windows client and the unRaid server.  Oddly, I noticed that two of my drives almost never have to spin up when this occurs (disk5 and disk14).  My shares are available on all of the disks, and there is nothing unique on any one drive.  The only thing that makes any sense to me is that the size of my directory entries has finally exceeded the available buffer for cache_dirs.  I have roughly 3,000 media folders with a handful of files in each.  Is it possible that a very vanilla unRaid (75TB media) server with 8GB memory actually needs more memory installed to cache the directories?  Will the system allow file copy buffering in memory to squeeze out this directory caching operation?

 

At least I'm not the only one seeing this issue.  Thanks for any help.

Link to comment
  • Replies 60
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Posted Images

I have a suggestion, but I really don't know if it will help.  Use the Tips and Tweaks plugin to decrease the vm.dirty* settings, from the defaults of 10 and 20, to 5 and 10, or even 2 and 3 (turn on the Help).  This decreases the disk buffering the kernel uses, might decrease the memory pressure, and enable more of the directory entries to stay cached.

 

What could cause a disk to spin up:

- any write to the disk

- any read of any part of the disk that has not been cached

- any read of once cached items that have been flushed from the cache

 

What CacheDirs tries to do is influence the kernel to keep all or most of the directory entries cached.  It does not read the files, only the folder contents, the directory entries.  If a tool that periodically scans a folder for changes ONLY looks at directory entries, then CacheDirs works well.  If that tool however also looks inside any of the files (e.g. for media metadata such as minutes, resolution, icons, etc), then CacheDirs won't help.  I noticed that you mentioned Sonarr not spinning up drives, but Radarr did.  That would seem to imply that Sonarr did not open files, but Radarr did.  Perhaps in the past, Radarr didn't, but a newer version does now?

 

As Dan said above, very large file transfers seem to flush the cache, and once the directory entries have been flushed out of the cache, then they have to be read back in by spinning up the drive and re-reading them again.  I'm hopeful (but no guarantees!) that smaller buffering may drain the buffers quicker, and not need as much data buffered during the transfer, thereby not needing to flush the directory cache.

Link to comment

I am about to post a new plugin that shows write/modify file activity on all disk shares.  This might help you understand what is causing the disk spin ups.  It does not show read file activity that can also cause spin ups, but it should help.  I will post in the plugins forum soon and also add an app xml for CA so it can be installed from CA and recognized as a known plugin.  Until then paste this URL in the install plugin line to install the plugin:

 

https://github.com/dlandon/file.activity/raw/master/file.activity.plg

 

CA will issue a warning that it is not compatible which you can ignore for the moment.

 

If you decide to try the plugin. please let me know if it helps in troubleshooting your spin ups.

 

NOTE: If you do not have cache_dirs installed, this plugin could cause spin ups when it looks for modified files.

Link to comment

I loaded dlandon's plug-in yesterday and watched several times as Windows File Explorer activity (just listing the share's root folders)  caused sporadic spin ups.  I have two folders in the root of my media share.  Check out this crazy symptom.  I've been experiencing similar results for many months:

 

Click on a root folder - no drives spin up

Click on the other folder - no drives spin up

Click again on the first folder - a couple of drives spin up

I repeat the clicking back and forth among these two folders and watch most of my drives eventually spin up

 

Nothing was logged in the plug-in, and the drives which spun up showed no read or write activity.  Also, I noticed from the system log this morning that the Mover activity which had moved one new media folder spun up 16 of my 18 data drives.  I routinely see this happen.  I opened the GUI and with all the drives spun down I checked the file activity log.  The GUI hung for about 45 seconds and I noticed three disks had spun up.  The file activity log showed nothing but cache drive activity (Plex docker running). 

 

BTW - the Plex docker was only added this past week, and is definitely not a part of my issue.

 

Although I really don't think the plug-in is the problem, I'm ready to disable cache_dirs and create a spin-up group just to avoid the PITA of these spin-ups.  Or, I may revert back to version 5 just as a sanity check.  I would also buy additional memory if someone had any conviction it would make a difference.

Link to comment

...watched several times as Windows File Explorer activity ...

Did you read what RobJ said earlier in this thread?

... something beside the simple directory entry being requested by the browsing tool.  That may apply to your Windows browsing.  If the Windows tool (Explorer or anything else) wants to check internal metadata (like resolution, minutes, etc) or thumbnails or a file icons, then those aren't cached by CacheDirs, and the drive will have to spin up to access that data.  If the file itself has to be opened for other info or data, then CacheDirs can't help.  If that's the problem, then you should be able to stop that by configuring the browsing tool not to access such extra info (icons, thumbnails, and other metadata).

Link to comment

Of course I read that post.  Tell me how this applies to a file browser action that causes sporadic spin ups when displaying the same directory information in a matter or seconds?  I could see this happening if my Windows client action was requesting metadata or file information that required opening or querying the file data.  In the case I mentioned I am simply clicking on a root folder to show hundreds of subfolders - no files at all.  I'm at a loss to explain how this would trigger a drive spin up on the third or fourth click for the same directory information.  And how about the other issue with spinning up three disks this morning just from looking at the GUI?  This is not a Windows client problem.

Link to comment

Yes, and I do appreciate the help.  No sarcasm intended.  I just wanted to clearly point out that these issues are also happening outside of Windows File Explorer.  This morning I never touched File Explorer, nor did the Mover that ran overnight.  The GUI (dlandon's plugin log view) somehow triggered a few drives to spin up this morning. 

 

 

Link to comment

I also agree with the relevancy of the Windows File Explorer mode question.  But if this were a cause of needing to spin up a drive to gather file information why would it only occur on subsequent clicks and not the first?  I'm not sure if a directory query from a disk would show up as a read operation, but the drive stats on newly spun up disks do not show any reads or writes.

Link to comment

Glad (well not actually glad) I'm not the only one experiencing these issues, of which, Explorer is definitely one of them that causes this from time to time.

The other day, after upgrading to 6.3.1, no drives spun up when using radarr, sonarr or anything else, as I created a new tv show, or movie.  Nothing spun, all was cached properly.  Then of course the next day it began again.

 

I'm installing the file.activity plugin as we speak, hopefully that helps pinpoint some of this to a better degree, and will look into the tips and tricks plugin as well.

 

Update:

After installing the file.activity plugin, and tweaking down the cache settings in tips & tweaks to 5 and 10, after one episode downloaded in sonarr, file.activity outputs this:

 

2017-02-13 12:30 => /mnt/cache/Television

2017-02-13 12:30 => /mnt/cache/Television/Maine.Cabin.Masters

2017-02-13 12:30 => /mnt/cache/Television/Maine.Cabin.Masters/Season.01

2017-02-13 12:30 => /mnt/cache/Television/Maine.Cabin.Masters/Season.01/S01E05.-.Rebuilding.the.Old.Schoolhouse.SDTV.x264.AAC.nfo

 

Drive 3 spun, the location where I'm sure the episode would be going to.

Couldn't help but notice that the writing of the actual file didn't display in the activity list.. not sure why, as the file was also written to the same location.

Nothing else was browsed, no explorer action on my part.  File gets downloaded, sonarr picks up, notifies plex and kodi.  This does work without any spinups, and normally should work without any spinups, and has worked, for a very long time without any.

Link to comment

Update:

After installing the file.activity plugin, and tweaking down the cache settings in tips & tweaks to 5 and 10, after one episode downloaded in sonarr, file.activity outputs this:

 

2017-02-13 12:30 => /mnt/cache/Television

2017-02-13 12:30 => /mnt/cache/Television/Maine.Cabin.Masters

2017-02-13 12:30 => /mnt/cache/Television/Maine.Cabin.Masters/Season.01

2017-02-13 12:30 => /mnt/cache/Television/Maine.Cabin.Masters/Season.01/S01E05.-.Rebuilding.the.Old.Schoolhouse.SDTV.x264.AAC.nfo

 

It looks like the /mnt/cache/Television directory was modified and that's why it shows and not the file.  The file was written to the cache.

 

The files activity plugin will show any file or directory modifications and not just file writes.

Link to comment

This shows there is no write activity to cause the disk 3 spin-up that magiin83 experienced, right?  The File Activity plugin showed the same for me (cache disk activity).  I'm mostly curious about the drive spin ups that happened to me in the GUI this morning.  Opening the File Activity log caused 3 of my disks to spin up.  I assume the plug-in is logging these activities and opening the log would not have to read from the disks.  Is this correct?

 

FWIW - he and I are both at 8GB memory, with a cache drive installed.  I also upgraded to the latest unRAID version last week with no change in this behavior.

Link to comment

The files activity plugin searches for the files and directories that have been modified by using the 'find' command.  It cannot log the activity.  It has to query the disks for the time of the last file or directory modified time.  If the directories are not cached by cache_dirs, the the disks will have to spin up to provide the information.

 

The example shown by magiin83 does show writes to disk3.  The writes were to the file directory structure even though the file was written to the cache drive.

Link to comment

This shows there is no write activity to cause the disk 3 spin-up that magiin83 experienced, right?  The File Activity plugin showed the same for me (cache disk activity).  I'm mostly curious about the drive spin ups that happened to me in the GUI this morning.  Opening the File Activity log caused 3 of my disks to spin up.  I assume the plug-in is logging these activities and opening the log would not have to read from the disks.  Is this correct?

 

FWIW - he and I are both at 8GB memory, with a cache drive installed.  I also upgraded to the latest unRAID version last week with no change in this behavior.

 

You also mentioned you had significantly more folders contained with your media. 

As for the change in behavior, the day I upgraded to 6.3.1 from 6.3.0, is the one day I saw no spin ups occur that weren't unnecessary ones, so for a moment there, it was all working as it properly should. 

I'm not saying cache_dirs isn't functioning, because it definitely does as I can essentially browse all of my media in various ways (MC, ftp, etc) without any spin ups (Explorer excluded), in and out of dirs.  But when it comes to these slight changes that are apparently occurring and kicking a folder out of cached memory, it doesn't make much sense to what or why one simple change within /Television or /Movies is causing it to do that, when everything else was cached in the first place.

 

So, what exactly needs to happen for cache_dirs to basically "update" the folder structure to not have a drive spin up?  And why all of a sudden has this started happening when it functioned for months at a time without a problem?

Link to comment

cache_dirs does not 'uncache' directory entries.  It only encourages Linux to keep the directory information in memory.  If Linux needs the memory, it will through out the entries, not cache_dirs.  If you feel the information is not staying in memory, you may need more memory.

 

cache_dirs can do nothing to keep a disk from spinning up when files and/or directories are modified.  The example shown here shows that the directory structure on disk 3 has been modified and the disk spun up for that reason.  The file was written to the cache, but the directory entries on disk 3 were updated.  This forces a write to the disk.  Linux will commit all disk write information to the disk and not keep it in memory.

 

So, what exactly needs to happen for cache_dirs to basically "update" the folder structure to not have a drive spin up?

 

There is nothing cache_dirs can do.

Link to comment

cache_dirs does not 'uncache' directory entries.  It only encourages Linux to keep the directory information in memory.  If Linux needs the memory, it will through out the entries, not cache_dirs.  If you feel the information is not staying in memory, you may need more memory.

 

cache_dirs can do nothing to keep a disk from spinning up when files and/or directories are modified.  The example shown here shows that the directory structure on disk 3 has been modified and the disk spun up for that reason.  The file was written to the cache, but the directory entries on disk 3 were updated.  This forces a write to the disk.  Linux will commit all disk write information to the disk and not keep it in memory.

 

So, what exactly needs to happen for cache_dirs to basically "update" the folder structure to not have a drive spin up?

 

There is nothing cache_dirs can do.

 

Understood, but if it were a memory issue, which I would gladly expand if it was really necessary, wouldn't I be seeing a much higher percentage of ram used up if that were the case?

And again, why is this an all of a sudden case, or the last few weeks for me.  Just seemed like it came completely out of no where... and if I'm running into this issue at 8gb of ram, and a total of 1600 folders that are being cached, what kinda of case would people need with twice as many folders, or an even larger amount of included folders in cache_dirs?

 

JQHT9B2.jpg

Link to comment

The file was written to the cache, but the directory entries on disk 3 were updated.  This forces a write to the disk.

 

 

All I can say is this is not how my system behaved before I started seeing these issues months ago.  If I wrote something to unRAID using a cache drive no other drive spun up - ever.  Now apparently something is pushing directory information out of the cache.

 

I am ordering additional memory.  I'll report any difference it makes.

 

 

Link to comment

The file was written to the cache, but the directory entries on disk 3 were updated.  This forces a write to the disk.

 

 

All I can say is this is not how my system behaved before I started seeing these issues months ago.  If I wrote something to unRAID using a cache drive no other drive spun up - ever.  Now apparently something is pushing directory information out of the cache.

 

I am ordering additional memory.  I'll report any difference it makes.

 

I'm also ordering ram, but as a side note, I did decide to turn off the plex docker, and sure enough, nothing spun, for an episode downloading and importing as well as! a movie downloading and importing to radarr.... honestly never would have thought this was a ram issue, nothing about this would have alluded to it (at least in my eyes)

Link to comment

I have one other (possibly related) thing to ask about.  I have never used spinup groups in my system.  They are not, and have never been enabled.  Yet, I noticed that unRAID wants to assign group identifications to my disks.  A friend of mine does not have the spinup issue, and all of his disks are listed as SpinupGroup="".

 

My disk.cfg shows the following values for my disks:

diskSpinupGroup.0="host2"

diskSpinupGroup.1="host1"

diskSpinupGroup.2="host1"

diskSpinupGroup.3="host1"

diskSpinupGroup.4="host1"

diskSpinupGroup.5="host1"

diskSpinupGroup.6="host1"

diskSpinupGroup.7="host1"

diskSpinupGroup.8="host1"

diskSpinupGroup.9="host3"

diskSpinupGroup.10="host4"

diskSpinupGroup.11="host5"

diskSpinupGroup.12="host6"

diskSpinupGroup.13="host7"

diskSpinupGroup.14="host8"

diskSpinupGroup.15="host2"

diskSpinupGroup.16="host2"

diskSpinupGroup.17="host2"

diskSpinupGroup.18="host2"

cacheSpinupGroup="host2"

 

 

I have gone into the GUI and changed these all to null, but upon restart they all reset to these values.  I also noticed this was similar in my old version 5 configuration, but with fewer disks.  magiin83 - can you check your values and see if you have the same type of spinup group ID's?

 

 

Link to comment

I have one other (possibly related) thing to ask about.  I have never used spinup groups in my system.  They are not, and have never been enabled.  Yet, I noticed that unRAID wants to assign group identifications to my disks.  A friend of mine does not have the spinup issue, and all of his disks are listed as SpinupGroup="".

 

My disk.cfg shows the following values for my disks:

diskSpinupGroup.0="host2"

diskSpinupGroup.1="host1"

diskSpinupGroup.2="host1"

diskSpinupGroup.3="host1"

diskSpinupGroup.4="host1"

diskSpinupGroup.5="host1"

diskSpinupGroup.6="host1"

diskSpinupGroup.7="host1"

diskSpinupGroup.8="host1"

diskSpinupGroup.9="host3"

diskSpinupGroup.10="host4"

diskSpinupGroup.11="host5"

diskSpinupGroup.12="host6"

diskSpinupGroup.13="host7"

diskSpinupGroup.14="host8"

diskSpinupGroup.15="host2"

diskSpinupGroup.16="host2"

diskSpinupGroup.17="host2"

diskSpinupGroup.18="host2"

cacheSpinupGroup="host2"

 

 

I have gone into the GUI and changed these all to null, but upon restart they all reset to these values.  I also noticed this was similar in my old version 5 configuration, but with fewer disks.  magiin83 - can you check your values and see if you have the same type of spinup group ID's?

 

I also have hosts next to the disks that exist:

 

diskSpinupGroup.1="host1"

diskSpinupGroup.2="host5"

diskSpinupGroup.3="host2"

Link to comment

UPDATE OF MY SITUATION:

 

For several months now (seemingly the same period of these spin-up issues) I have noticed a delay in my Kodi client when trying to add movies to its library.  I generally update new media metadata using Ember from my Windows PC.  It allows me to easily choose the poster and fanart, plus I can do custom genre tagging like 3D or Holiday.  Anyway, doing this requires me to refresh each new movie individually in Kodi using the NFO file only.  Just opening the Movies folder and each movie subfolder had been taking 15-20 seconds each.  For the first time in a long time this process last night snapped right through.  It was painless like it used to be.  So something has definitely changed.

 

What I changed last night:

Turned on spinup groups and cleared the group name from each disk

Updated the Recycle Bin plugin

Changed the Movies share to remove all explicitly listed disks to return the default to “All” (I had changed this in my earlier troubleshooting)

Rebooted the server

 

Fooling around with Windows File Explorer afterwards I still found a couple of disks spinning up.  But I thought this might just be a result of the cache building up.  Although this morning I noticed the Mover had spun up all of the drives in the array again.

 

But I am seriously wondering if the spinup group labeling is somehow in the mix of my issues.  Spinup groups are now enabled with all disks showing no group association.  This matches my friend's (who doesn't have this issue) configuration.  Might be worth a test on your system magiin83.

Link to comment

UPDATE OF MY SITUATION:

 

For several months now (seemingly the same period of these spin-up issues) I have noticed a delay in my Kodi client when trying to add movies to its library.  I generally update new media metadata using Ember from my Windows PC.  It allows me to easily choose the poster and fanart, plus I can do custom genre tagging like 3D or Holiday.  Anyway, doing this requires me to refresh each new movie individually in Kodi using the NFO file only.  Just opening the Movies folder and each movie subfolder had been taking 15-20 seconds each.  For the first time in a long time this process last night snapped right through.  It was painless like it used to be.  So something has definitely changed.

 

What I changed last night:

Turned on spinup groups and cleared the group name from each disk

Updated the Recycle Bin plugin

Changed the Movies share to remove all explicitly listed disks to return the default to “All” (I had changed this in my earlier troubleshooting)

Rebooted the server

 

Fooling around with Windows File Explorer afterwards I still found a couple of disks spinning up.  But I thought this might just be a result of the cache building up.  Although this morning I noticed the Mover had spun up all of the drives in the array again.

 

But I am seriously wondering if the spinup group labeling is somehow in the mix of my issues.  Spinup groups are now enabled with all disks showing no group association.  This matches my friend's (who doesn't have this issue) configuration.  Might be worth a test on your system magiin83.

 

Interesting turn of events.

Ok so if I'm to understand, you now have groups enabled? Or you enabled them, set shares to groups (also cleared out the group names from each disk (how did you do this? editing disk.cfg?) and then disabled groups to make sure they flushed?

 

I didnt even know there was a recycle bin plugin, I'll have to look into that.

As I mentioned in the previous posts, I do have the issue with kodi essentially spinning up drives to capture new movies (not tv, but that more than likely has to do with sonarr's specific folder update, rather than radarr which only has the full library scan functioning so far)

 

Also, I do still have the ram being delivered tomorrow as well, which couldn't hurt in the long run regardless of this working without the need for the upgrade.

But I agree, this issue of spinning up did occur some months ago, after it working and nothing ever spinning for months at a time.  Like a flip of a switch

Link to comment

Spinup groups are enabled in Disk Settings.  You can assign or change group names to each disk by clicking on the individual disks.  I assume that you could also edit the disk.cfg file and restart the array as well, but I'm not sure.  I do know if spinup groups are not enabled and you edit the group names in the file it gets changed back upon array startup.  Change them through the GUI.

 

I expected spinup groups to not be functional without group names assigned to disks.  But now I'm trying to determine if I have done little more than enable a default spinup group (hence all drives spinning up for the Mover this morning).  But something definitely changed for the better from my Kodi client perspective, and maybe this is all related.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.