cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up

[NAS]

Can you describe a scenario where that would be beneficial?

[WeeboTech]

Can you describe a scenario where that would be beneficial?

 

As Tom mentioned to me.  with NFS and the SHFS/USER share, FUSE let's go of internal file handle structures after a certain amount of time when unused.  This is the reason for the NFS stale file handle issue that crops up on the user shares.

[Joe L.]

Can you describe a scenario where that would be beneficial?

no, that is why it (-u) is not enabled by default.

[Joe L.]

Can you describe a scenario where that would be beneficial?

 

As Tom mentioned to me.  with NFS and the SHFS/USER share, FUSE let's go of internal file handle structures after a certain amount of time when unused.  This is the reason for the NFS stale file handle issue that crops up on the user shares.

then, it (-u) might be something to try if you use NFS.

[Ulvan]

Quick query on depth. Does that option count the root as depth zero, or one? ie:

 

root/folder 1/2/3

 

or

 

root/2/3/4

 

In first case you'd have to set cache_depth to 3, in latter to 4 to cache all dirs.

 

I'm trying to figure out how to get caching to work while allowing the disks to spin down. I'm lowering the value to see if it could be an issue with running out of memory.

 

Can you describe a scenario where that would be beneficial?

 

As Tom mentioned to me.  with NFS and the SHFS/USER share, FUSE let's go of internal file handle structures after a certain amount of time when unused.  This is the reason for the NFS stale file handle issue that crops up on the user shares.

then, it (-u) might be something to try if you use NFS.

 

This is largely gibberish to me. To a layman who uses SMB instead of NFS (hope I got that right), is there a situation when you'd use -u?

[NAS]

As confirmed byu Joe the -u flag is is an edge case almost no should use.

[acurcione]

I set this up on my new system and have a question on one option:

 

wait for array to come online before start of cache scan of directories

 

I've searched for a better explanation of it, but couldn't really find one.  Does setting this to "no" mean that it will scan the drives all the time and not allow them to spin down?  And "yes" means it doesn't do a new scan of the directories in cache until the array comes online at some point?

 

Thanks in advance.

[DaleWilliams]

See here:

http://lime-technology.com/forum/index.php?topic=4500.0

for a lengthy discussion of Cache Directories.

 

'wait' is discussed very early on...If I understand correctly, 'Cache_directories' is a script that runs, and is started when you boot up unRAID. BUT, at the time the script is started, the array is not yet 'online'.  {So my mental image of what's happening is that the cache_directories script runs, sees there are no disks, and will exit/stop running.  } The -w flag says, 'wait' until the array is actually started, the drives are mounted, and the disks become available, and THEN continue running.

[acurcione]

 

See here:

http://lime-technology.com/forum/index.php?topic=4500.0

for a lengthy discussion of Cache Directories.

 

'wait' is discussed very early on...If I understand correctly, 'Cache_directories' is a script that runs, and is started when you boot up unRAID. BUT, at the time the script is started, the array is not yet 'online'.  {So my mental image of what's happening is that the cache_directories script runs, sees there are no disks, and will exit/stop running.  } The -w flag says, 'wait' until the array is actually started, the drives are mounted, and the disks become available, and THEN continue running.

 

Wow. I looked at that thread several times and I totally missed that. You description certainly sounds correct and that's how I'm running it so it should be fine. I am still noticing a long delay the first time accessing he shares though so I wasn't sure it was actually cached yet.

 

 

Sent from my iPad using Tapatalk

[DaleWilliams]

I am still noticing a long delay the first time accessing he shares though so I wasn't sure it was actually cached yet.

Yes. I find that when I first START my array, unRAID is substantially 'unresponsive' for several minutes.

Based upon that little flickering 'hard drive activity LED' on the front of the box, it appears that cache_dirs has a pretty high priority as it first populates memory with directories.  I just go get more coffee and wait it out.

[eshelon]

Any chance this will get fully supported for unraid 6.0-beta4?

[Falc410]

Hi,

 

I'm new to unRaid and using Dynamix with cache_dirs Plugin, but I have no idea what settings I should use. Is there some recommendation I can read up on? I'm using the default settings from Dynamix but for some reason I have disks spinning up or keeping busy for no apparent reason - someone suggested it might be due to this plugin. So I'm wondering if I can restrict it somehow. Yes I read the first page, but I'm still not sure what to make of it.

[acurcione]

Hi,

 

I'm new to unRaid and using Dynamix with cache_dirs Plugin, but I have no idea what settings I should use. Is there some recommendation I can read up on? I'm using the default settings from Dynamix but for some reason I have disks spinning up or keeping busy for no apparent reason - someone suggested it might be due to this plugin. So I'm wondering if I can restrict it somehow. Yes I read the first page, but I'm still not sure what to make of it.

 

Not sure I'm the best to answer, but since I only recently installed this I'll give it a shot.  I'm pretty much using the default settings, although I'm expressly only including one user share where all my media is.  Here are my settings:

 

Folder caching function: enabled

Wait until array is online: yes

Force disks busy: no

Suspend during Mover process: yes

Scan user shares: yes

Minimum interval between folder scans (sec): blank

Maximum interval between folder scans (sec): blank

Maximum scan level depth: 4

Excluded folders (separated by comma): blank

Included folders (separated by comma): Media

User defined options: blank

 

Hope this helps.  My array is behaving normally and isn't spinning up when it shouldn't.

[RobJ]

Scan user shares: yes

 

This was an option left in by Joe to satisfy users who still felt it accomplished something for them, but it is difficult to see the rationale for that.  User shares are a file system maintained entirely in memory, so caching them is just making an extra copy of the file system elements in the same RAM.  That seems like a waste of RAM, with no benefit.  The idea behind Cache_dirs is to keep in memory those file system elements that are constantly being requested from disk, so that the disk does not have to keep spinning up.  I would set this to NO and free up some memory.

[WeeboTech]

Scan user shares: yes

 

This was an option left in by Joe to satisfy users who still felt it accomplished something for them, but it is difficult to see the rationale for that.  User shares are a file system maintained entirely in memory, so caching them is just making an extra copy of the file system elements in the same RAM.  That seems like a waste of RAM, with no benefit.  The idea behind Cache_dirs is to keep in memory those file system elements that are constantly being requested from disk, so that the disk does not have to keep spinning up.  I would set this to NO and free up some memory.

 

 

Normally, it should be set to NO.

However, if you are using NFS, it could make a difference to help alleviate the Stale NFS handle issue.

As Tom explained to me, it keeps fuse blocks in memory as well as the dentry information.

So there is a benefit in small use cases where NFS is used to access the user share.

[acurcione]

 

Scan user shares: yes

 

This was an option left in by Joe to satisfy users who still felt it accomplished something for them, but it is difficult to see the rationale for that.  User shares are a file system maintained entirely in memory, so caching them is just making an extra copy of the file system elements in the same RAM.  That seems like a waste of RAM, with no benefit.  The idea behind Cache_dirs is to keep in memory those file system elements that are constantly being requested from disk, so that the disk does not have to keep spinning up.  I would set this to NO and free up some memory.

 

Well I guess I completely misunderstood the purpose of this. The only thing anyone should be accessing are user shares. If the entire user share(s) in memory, then why does it take so long to pull up when it's accessed (at least the first time)? Isn't that the point of this plugin?

 

I mean if your right then I totally did this wrong and should remove the "Media" inclusion AND change that setting to no. But again, this is still confusing based on your comment.

 

 

Sent from my iPad using Tapatalk

[WeeboTech]

How much ram is in your system and how many files on that 'Media' share?

[acurcione]

 

How much ram is in your system and how many files on that 'Media' share?

 

16GB, 806 movies, 1104 tv shows

 

And yes, my memory is pretty well maxed out, but I figured that was normal. I'm guessing not?

 

 

Sent from my iPad using Tapatalk

[WeeboTech]

Is that whole 16GB allocated to unRAID or is it divided up with XEN/Vmware?

 

From my experience, you should be working fine.

[acurcione]

For now it's dedicated 100% to unRAID. I changed the settings anyway and things are actually OK memory wise. Also, I wasn't actually having any issues when I posted. I was just replying to the OP. But I learned something in the process so it isn't all bad.

 

Something was going on though that I can't explain. I had rebooted this afternoon so all the drives were spun up, but when I just hopped on the server they were all still spun up, including the parity drive, but nothing was accessing the box at that point. Also checked the logs and it showed the spin down event several hours ago, but no spin up event even after I accessed the cache drive to spin it up - even though it's. SSD so in theory it should never spin down even if it shows it.

 

I dunno what that's all about, but I haven't had any issues with drives spinning down.

 

 

Sent from my iPad using Tapatalk

[RobJ]

Well I guess I completely misunderstood the purpose of this. The only thing anyone should be accessing are user shares. If the entire user share(s) in memory, then why does it take so long to pull up when it's accessed (at least the first time)? Isn't that the point of this plugin?

 

I mean if your right then I totally did this wrong and should remove the "Media" inclusion AND change that setting to no. But again, this is still confusing based on your comment.

 

I suspect that there are many users who have installed Cache_Dirs, but don't actually need it.  If access to your files is solely through the User Share system, and you don't fall under the NFS exception mentioned by WeeboTech (learned something new today!), then you don't need Cache_Dirs.  The User Share system acts as a Cache of Dirs, because it is the merger of the individual disk directories, kept in memory.  And if you do access your files at times directly from the disks, but don't have any programs that poll those disk directories and you only rarely 'explore' them (with a file manager, such as Windows Explorer), then you don't strictly need Cache_Dirs either.  Cache_Dirs may add some convenience though, because if you do feel like browsing through the directories of any disk, Cache_Dirs will have them in memory and the drive won't have to spin up.

 

I recommend reading the original document on Cache_Dirs, Keep directory entries cached.  It's a little dated, but still helpful.

 

If the entire user share(s) in memory, then why does it take so long to pull up when it's accessed (at least the first time)? Isn't that the point of this plugin?

 

If you need to access a file, for example to play it, then you are asking for the content of the file, and that isn't cached.  So the disk it is on will have to spin up, causing the delay on first access you see.

[acurcione]

 

Well I guess I completely misunderstood the purpose of this. The only thing anyone should be accessing are user shares. If the entire user share(s) in memory, then why does it take so long to pull up when it's accessed (at least the first time)? Isn't that the point of this plugin?

 

I mean if your right then I totally did this wrong and should remove the "Media" inclusion AND change that setting to no. But again, this is still confusing based on your comment.

 

I suspect that there are many users who have installed Cache_Dirs, but don't actually need it.  If access to your files is solely through the User Share system, and you don't fall under the NFS exception mentioned by WeeboTech (learned something new today!), then you don't need Cache_Dirs.  The User Share system acts as a Cache of Dirs, because it is the merger of the individual disk directories, kept in memory.  And if you do access your files at times directly from the disks, but don't have any programs that poll those disk directories and you only rarely 'explore' them (with a file manager, such as Windows Explorer), then you don't strictly need Cache_Dirs either.  Cache_Dirs may add some convenience though, because if you do feel like browsing through the directories of any disk, Cache_Dirs will have them in memory and the drive won't have to spin up.

 

I recommend reading the original document on Cache_Dirs, Keep directory entries cached.  It's a little dated, but still helpful.

 

If the entire user share(s) in memory, then why does it take so long to pull up when it's accessed (at least the first time)? Isn't that the point of this plugin?

 

If you need to access a file, for example to play it, then you are asking for the content of the file, and that isn't cached.  So the disk it is on will have to spin up, causing the delay on first access you see.

 

Thanks Rob. Yeah, I pretty much think I don't need it based on that article. The only time I access the disks directly are when I'm being anal and double checking things. And I probably shouldn't even be doing that.

 

Having said that, I'm a little disheartened that documentation isn't better OR at the very least setup for a newbie to get a system setup with more ease. I thought I had found that, but it seems even that walk through is outdated. It's unrelated to this though, since this isn't something covered there. After reading about it, it sounded like something everyone should have installed.

 

Anyway, I appreciate the information and I will be uninstalling this since it seems completely unnecessary for how I'm using it.

 

 

Sent from my iPad using Tapatalk

[WeeboTech]

With the file count as previously defined, cache_dirs will help prevent unnecessary spin ups. The fuse usershare does not cache everything. It caches some data, not all. There is a finite table just like the dentry table.

 

So where cache_dirs helps is in keeping all of the directory inodes in ram even if a large file is read.

This helps when the user share has to wade through higher level directory. user share will search all drives for the named directory at that level. Therefore, having those directory inodes available in ram allows the user share to walk through directories that much faster.

 

Where this becomes ineffective is when there are so many files and a limited amount of low memory thus causing memory pressure in other ways.  In my system I have so many music files that cache_dirs is counter productive. With a small count of files, cache_dirs is a good aide in alleviating unnecessary spin ups.

[PeterB]

I suspect that there are many users who have installed Cache_Dirs, but don't actually need it.  If access to your files is solely through the User Share system, and you don't fall under the NFS exception mentioned by WeeboTech (learned something new today!), then you don't need Cache_Dirs.

 

Actually, I believe that Cache_Dirs still has a benefit for these users in that it pre-caches rather than waiting for your first access.  My Movies collection is still small compared to many here.  However, split across three drives, there is a delay of about 20 seconds on first access.  For those who can keep their system up for months on end, this is a non-issue.  However, if you experience multiple power outages a day, it is a significant aggravation.

[acurcione]

I suspect that there are many users who have installed Cache_Dirs, but don't actually need it.  If access to your files is solely through the User Share system, and you don't fall under the NFS exception mentioned by WeeboTech (learned something new today!), then you don't need Cache_Dirs.

 

Actually, I believe that Cache_Dirs still has a benefit for these users in that it pre-caches rather than waiting for your first access.  My Movies collection is still small compared to many here.  However, split across three drives, there is a delay of about 20 seconds on first access.  For those who can keep their system up for months on end, this is a non-issue.  However, if you experience multiple power outages a day, it is a significant aggravation.

 

So are you using it with "scan user shares" set to yes or no then?