once i wake up in the morning this will be the first thing i try.

 

Kudos Joe

Joe so far everything looks great at my end.

 

Question: how would i tell it to cache everything except /flash?

Joe so far everything looks great at my end.

 

Question: how would i tell it to cache everything except /flash?

Everything under /mnt/disk*/ that has a top level folder named like a user-share is cached.  If you have a top-level folder on the hard-disks that is named "flash" (perhaps holding a copy of your flash drive contents) then you can exclude it by using

cache_dirs -e flash

 

The "-e" option excludes the share named as its argument.

 

If you were talking about your USB flash drive itself, it is not involved, nor cached.

 

Joe L.

 

 

 

 

I completely misunderstood the meaning of the -e switch i.e. i thought it was INCLUDE. Dont ask me why brain farts happen.

 

Swapping over my baby cache script to your grand pappy now

 

Hopefully I will get a chance to test this new script out this weekend.  I have some stuff I want to do to the server so this will be another good excuse to shut it down and do my stuff.

This can be run on 4.3 as well right???

This can be run on 4.3 as well right???

 

I dont see any reason it wouldnt run on any version at all

 

Go on you know you want to... you wont believe how much better shares feel with it

Yeah I have been trying to get the ls -r trick to work for a while now. It works great for browsing and not spinning the drives up but if I run a movie directly by typing in the full path it spins up all disks within that share. I am playing around with the "find" command now and think I am making some progress. I think my problem may have been trying to cache too much at one time...also the first cache seems to take alot longer than 10sec so I am not sure how that affects it. I am trying the command manually till I get it to work the way I want.

You will likely drive yourself mad trying to find a logic to this based on FS based observations. The whole ls thing is a hack dealing with a symptom and not a soltion of the root problem. If the kernel did what the docs say it should we wouldn't really need it.

 

All im saying is, run this tool, let it keep running and thats likely all you can do.

 

Joes version is extremely slick and probably close to the pinnacle of where this trick can take us.

 

Looks like Joes script also deals with cache pressure for us as well so that a line that can come out of the go script

I hear ya

 

It's so weird!!!!

 

I have 6 disks with a Movies share. I find a movie on Disk 4 as a test (Gremlins). I restart the server to start fresh. After the reboot while all the disks are spinning I issue at the telnet prompt: find /mnt/user/MOVIES -maxdepth 1

After about 10-15 sec I get a list of about 740 entries (Movies). I spin all of the disks down and at the command prompt I type: \\tower\MOVIES\Gremlins.mkv

I see Disk4 light up and then one by one every other disk lights up and perform a bunch of "reads" until my movie FINALLY starts up and runs from Disk4. Now if I stop that movie and spin all the disks down again and try a different movie, only the disk it's on spins up 

 

So yes I am driving myself crazy :D

Yeah I have been trying to get the ls -r trick to work for a while now. It works great for browsing and not spinning the drives up but if I run a movie directly by typing in the full path it spins up all disks within that share. I am playing around with the "find" command now and think I am making some progress. I think my problem may have been trying to cache too much at one time...also the first cache seems to take alot longer than 10sec so I am not sure how that affects it. I am trying the command manually till I get it to work the way I want.

If disks are initially spun down, and if the directory information is not already cached, then each disk involved will spin up in turn.  As you said, this will take much more than 10 seconds.

 

 

The duration of "disk access" has nothing to do with the duration of the inter-scan delay.  If the first scan took 30 seconds, the first loop delay would be half way between the min and max values.  If you did nothing to set them from their defaults, it would sleep 5 seconds.

 

The second directory scan would take a fraction of a second.  This would then be faster than the first, so the "avg" time would decrease, and the loop delay increased to 6 seconds.  Eventually, unless something odd occurs, the sleep duration between scans would end up at 9 or 10 seconds.  Once activity on a disk starts, and the scan takes longer than the average time, the loop delay between scans of the directories would decrease.

 

The script has a fair number of comments in it, so you can look at it for more detail of what it is doing, but basically it:

 

1. The script looks for a lock-file to prevent the simultaneous running of two copies of the script.

2. If /mnt/user does not exist (and the array is off-line) the script sleeps 10 seconds and checks again, and again.

3. Makes a list of all the top level directories on /mnt/user.  These represent the user-shares

4. For each of your physical disks, it looks for similarly named folders and checks if they are in an exclusion list

5. If not in the exclusion list, the script will perform the "command" on the physical disk/folder combinations.

6. The script times the duration of a set of "commands" on the disk/folder combinations.  It keeps a crude running average.

7. The script then sleeps a bit and starts back at step 4 (sleep duration adjusted based on last duration vs. average duration)

 

Joe L.

If disks are initially spun down, and if the directory information is not already cached, then each disk involved will spin up in turn.  As you said, this will take much more than 10 seconds.

 

 

 

If this runs from the GO script then all of the disks should already be spun up...correct? I am running this while all disks are spun up then spinning them back down and trying to launch a movie and have the results in my last post.

 

Does this not work the way I am trying to get it to work or is this just to help to keep the disks spun down while browsing??

 

Thanks,

Scott

I hear ya

 

It's so weird!!!!

 

I have 6 disks with a Movies share. I find a movie on Disk 4 as a test (Gremlins). I restart the server to start fresh. After the reboot while all the disks are spinning I issue at the telnet prompt: find /mnt/user/MOVIES -maxdepth 1

After about 10-15 sec I get a list of about 740 entries (Movies). I spin all of the disks down and at the command prompt I type: \\tower\MOVIES\Gremlins.mkv

I see Disk4 light up and then one by one every other disk lights up and perform a bunch of "reads" until my movie FINALLY starts up and runs from Disk4. Now if I stop that movie and spin all the disks down again and try a different movie, only the disk it's on spins up 

 

So yes I am driving myself crazy :D

Must be some kind of "Gremlin"   ;)

 

Or it could be windows explorer, scanning the directory structure, even though you are only playing the one movie.  When you use a path of "\\server\folder\file" in windows, who knows what it actually does.  

 

I'm not saying this script is perfect, it's far from it... but it will help most people, and perhaps give others ideas to make new suggestions.

 

Joe L.

If disks are initially spun down, and if the directory information is not already cached, then each disk involved will spin up in turn.  As you said, this will take much more than 10 seconds.

 

 

 

If this runs from the GO script then all of the disks should already be spun up...correct? I am running this while all disks are spun up then spinning them back down and trying to launch a movie and have the results in my last post.

Sounds like it would work.

Does this not work the way I am trying to get it to work or is this just to help to keep the disks spun down while browsing??

 

Thanks,

Scott

No, the disks should eventually all spin down.  The movies listings will all end up cached.

When you play a movie, and the cached directory entries are displaced, then each disk in turn will spin up as needed to get the data about its files once more. 

 

The big issue we all have is that movies are multiple gigabytes in length, and our memory used for cache is much smaller by comparison. (I've only got 512 Meg of RAM. Most of my movies are 5-6Gig in size)  At some point, the directory entry might be purged as the movie block is thought to be more recently accessed, and it will stay in the cache when your directory info is re-used for the next bit of your movie.

 

Or... Depending on your version of unRAID, it might not be aware the disks are spun down, so it will access them itself.  (something only fixed in recent versions) What version unRAID are you running?

 

What we are basically trying to do is make all the directory information about your media exist in the cache with a time-stamp that is MORE-RECENT than the oldest-accessed data block of the movie you are watching.  If that is the case, and the server goes to cache the newest block of data of the movie you are watching, it will use the least-used block of memory in cache, the older part of your movie, rather than any directory data blocks, accessed more recently. 

I am running 4.3...should I try 4.4 or 4.5???

 

My disks do eventually spin down after a movie starts, So an hour or so into the movie the only disk I have spun up is the one its playing from. My gripe is having all my disks spin up when I call a movie for the first time which in turn takes a little bit for the movie to actually start.

 

This is not devastating since I know disks spin up and down all the time, I was just trying to make things more efficient for myself and I thought this script would have helped with this

I hear ya

 

It's so weird!!!!

 

I have 6 disks with a Movies share. I find a movie on Disk 4 as a test (Gremlins). I restart the server to start fresh. After the reboot while all the disks are spinning I issue at the telnet prompt: find /mnt/user/MOVIES -maxdepth 1

As pointed out earlier in this thread, everything under /mnt/user/MOVIES is already in RAM.   That is why I am not scanning it... no need to use up additional memory.

 

The "find" needs to continuously occur... Once is not enough.  As soon as the scanned directory blocks are older than the movie blocks being accessed, they will be re-used as cache for the movie.

 

Joe L.

I am running 4.3...should I try 4.4 or 4.5???

 

My disks do eventually spin down after a movie starts, So an hour or so into the movie the only disk I have spun up is the one its playing from. My gripe is having all my disks spin up when I call a movie for the first time which in turn takes a little bit for the movie to actually start.

 

This is not devastating since I know disks spin up and down all the time, I was just trying to make things more efficient for myself and I thought this script would have helped with this

Start it running with the default parameters... let it stabilize a while (so it's average time is somewhat respective of your system) and then try your access of a movie.

 

I cannot stop "windows explorer" from exploring on its own.  It likes to "directory cache" on its own, unless you somehow disable that "feature"

 

Joe L.

Will do.

 

So I can just add: /boot/cache_dirs -F -d 3 -e "data" or a variant of it to the GO script?

 

Can I remove the cache pressure setting from the GO script since this script handles it...or does it??

Will do.

 

So I can just add: /boot/cache_dirs -F -d 3 -e "data" or a variant of it to the GO script?

 

Can I remove the cache pressure setting from the GO script since this script handles it...or does it??

No, in the "go" script you would NEVER use the "-F" option, otherwise, it would fill and use up all your memory with its statistics output.

Nor would you use "-e data" unless you had a user-share named "data" you did not care to cache.

 

Assuming you put the script in the top-most folder of the flash drive, and it therefore existed at /boot/cache_dirs, all you would need in the "go" script is:

/boot/cache_dirs -d 3

 

I have my script in /boot/custom/bin, therefore, I invoke it as

/boot/custom/bin/cache_dirs -d 3 -e "data"

 

Yes, you can remove the line from your "go" script that sets cache pressure to zero, this script handles it for you.

 

edit: if you also add the "-w" option, cache_dirs will wait until unRAID is on-line and has started sharing the disks on the lan.  The "-w" option should probably always be used when being started from the "go" script.

 

Joe L.

Yeah...sorry about that, I typed that post as I was on my way out. I wasn't going to use the -F and was going to change the -e for my needs.

 

Thanks for all the help and I will let you know how it turns out.

Joe, you probably thought you were finally finished with this, but good work just inspires more ideas!  I thought I'd feed them to you now, while it is all still fresh in your mind, but I understand if other things have become more important.

 

1.  I'd like to see the Quit method folded back inside, partly to hide its code, partly to keep its mechanisms and vars 'encapsulated' in case of future changes, and partly to simplify and unify for less technical users.  Perhaps a -q option that runs the lock remover statement (eg. cache_dirs -q)?

 

2.  I'd like to see more of our scripts and tweaks logged to the syslog.  I don't need to see a copy of the script or copyright and background info, but just what was started and any vars of particular interest to the user and troubleshooter.  For example,

* logger -t cache_dirs "cache_dirs v2 $*"    (I'll be happy with whatever you choose to show)

* logger -t cache_dirs 'vm.vfs_cache_pressure=0'  (might be nice to show the result of the 'set' instead, to verify it was set correctly)

* logger -t cache_dirs $initial_num_seconds $initial_depth $initial_maxdepth  $

* logger -t cache_dirs share and folder names being cached  (optional, but would be nice)

 

3.  This is great for those with User Shares, but does not help those of us who don't use them.  Why not mirror your excellent -e excludes with -i includes for user-specified folder names in a separate array.  The syntax I would suggest is:  disk number, forward slash, path_to_folder.  Examples would be  -i 3/Music -i '1/Photo Archive' -i '5/Backups/Acronis Images', etc.  Why?  Because the script can fully specify them by /mnt/disk$array_included_folders[j], plus they are easy to understand for others.  You don't need my implementation ideas, but here's my suggestion any way.  There are 3 cases for any folder, whether it is a User Share or included disk folder, (1) does not exist (User Shares not used, or folder name mis-typed), (2) exists but not mounted yet, (3) exists and ready.  Cases 1 and 2 can generally be treated as the same, if you have a forgiving main loop, that checks for the existence of /mnt/user and /mnt/disk# before processing each.  In other words, remove the early check for mounted state and its continue.

 

4.  Very very minor quibble:  the comment about "disk0 is the user share" should be that it is the parity drive.  Should "[ "$i" = "disk0" ]" be [ "$i" = "/mnt/disk0" ] or [ "$i" ~ "disk0" ] instead?  Canceled, author ignorance!

 

5.  What about the Cache drive?  Should that be checked and cached too, if they have User Shares on?

Joe, you probably thought you were finally finished with this, but good work just inspires more ideas!  I thought I'd feed them to you now, while it is all still fresh in your mind, but I understand if other things have become more important.

 

1.  I'd like to see the Quit method folded back inside, partly to hide its code, partly to keep its mechanisms and vars 'encapsulated' in case of future changes, and partly to simplify and unify for less technical users.  Perhaps a -q option that runs the lock remover statement (eg. cache_dirs -q)?

Excellent idea.  I'll add it.

2.  I'd like to see more of our scripts and tweaks logged to the syslog.  I don't need to see a copy of the script or copyright and background info, but just what was started and any vars of particular interest to the user and troubleshooter.  For example,

* logger -t cache_dirs "cache_dirs v2 $*"    (I'll be happy with whatever you choose to show)

* logger -t cache_dirs 'vm.vfs_cache_pressure=0'  (might be nice to show the result of the 'set' instead, to verify it was set correctly)

* logger -t cache_dirs $initial_num_seconds $initial_depth $initial_maxdepth  $

* logger -t cache_dirs share and folder names being cached  (optional, but would be nice)

Also a great idea.  We'll then be able to figure out how it was invoked by looking in the log

3.  This is great for those with User Shares, but does not help those of us who don't use them.  Why not mirror your excellent -e excludes with -i includes for user-specified folder names in a separate array.  The syntax I would suggest is:  disk number, forward slash, path_to_folder.  Examples would be  -i 3/Music -i '1/Photo Archive' -i '5/Backups/Acronis Images', etc.  Why?  Because the script can fully specify them by /mnt/disk$array_included_folders[j], plus they are easy to understand for others.  You don't need my implementation ideas, but here's my suggestion any way.  There are 3 cases for any folder, whether it is a User Share or included disk folder, (1) does not exist (User Shares not used, or folder name mis-typed), (2) exists but not mounted yet, (3) exists and ready.  Cases 1 and 2 can generally be treated as the same, if you have a forgiving main loop, that checks for the existence of /mnt/user and /mnt/disk# before processing each.  In other words, remove the early check for mounted state and its continue.

Let me think about that one.  I can see how in its current state, a person who does not use "User-Shares" would not be able to use the script.  I like the -i include_dir idea.  Perhaps a "-u" to have it use what it finds in user-shares, and if "-u" not given as an argument, then you must include at least one "-i directory_name"  Do you think that might work?

4.  Very very minor quibble:  the comment about "disk0 is the user share" should be that it is the parity drive.  Should "[ "$i" = "disk0" ]" be [ "$i" = "/mnt/disk0" ] or [ "$i" ~ "disk0" ] instead?

No, /mnt/disk0 is not the parity drive.  The parity drive is not "mounted" but I had blindly copied and implemented that from the pseudo-code RobJ had posted earlier in the thread.    There is not, to my knowledge, any /mnt/disk0, so the line can be removed, it will never be hit.  There is a /mnt/user0 if you have a cache drive.  Perhaps he was thinking of it. (can't have an ego when a constructive code review is under way, can I  )

5.  What about the Cache drive?  Should that be checked and cached too, if they have User Shares on?

Yes... I think you are right, it should be cached.  (and you can now tell I don't usually have a cache drive defined)

 

Joe L.

No, /mnt/disk0 is not the parity drive.  The parity drive is not "mounted" but I had blindly copied and implemented that from the pseudo-code RobJ had posted earlier in the thread.    There is not, to my knowledge, any /mnt/disk0, so the line can be removed, it will never be hit.  There is a /mnt/user0 if you have a cache drive.  Perhaps he was thinking of it. (can't have an ego when a constructive code review is under way, can I  )

Sounds like an ignorant user wrote that pseudo-code! 

 

3.  This is great for those with User Shares, but does not help those of us who don't use them.  Why not mirror your excellent -e excludes with -i includes for user-specified folder names in a separate array.  The syntax I would suggest is:  disk number, forward slash, path_to_folder.  Examples would be  -i 3/Music -i '1/Photo Archive' -i '5/Backups/Acronis Images', etc.  Why?  Because the script can fully specify them by /mnt/disk$array_included_folders[j], plus they are easy to understand for others.  You don't need my implementation ideas, but here's my suggestion any way.  There are 3 cases for any folder, whether it is a User Share or included disk folder, (1) does not exist (User Shares not used, or folder name mis-typed), (2) exists but not mounted yet, (3) exists and ready.  Cases 1 and 2 can generally be treated as the same, if you have a forgiving main loop, that checks for the existence of /mnt/user and /mnt/disk# before processing each.  In other words, remove the early check for mounted state and its continue.

Let me think about that one.  I can see how in its current state, a person who does not use "User-Shares" would not be able to use the script.  I like the -i include_dir idea.  Perhaps a "-u" to have it use what it finds in user-shares, and if "-u" not given as an argument, then you must include at least one "-i directory_name"  Do you think that might work?

 

I first thought of doing something like that, but realized many users probably have hybrid systems, since User Shares are not an 'either/or' thing.  They may have both User Shares for media storage, and single disk folders for backup storage.  So it would be good to handle both in the same loop.  Something simple like:  (ignore syntax issues, off-by-one errors, boundary issues)

 

while true
   if [ -d /mnt/user ] then;
      process them, only if they are turned on, and exist, and are mounted; skip on if not;

   if [ $included_folder_count .gt 0 ] then;
      for f = 1 to $included_folder_count
         if [ -d /mnt/disk$included_folders[f] ]
            process /mnt/disk$included_folders[f]

   sleep $num_seconds
   process timer stuff
endwhile

 

Lousy code, but it easily handles cases of User Shares only, disk shares only, and hybrids, whether or not they exist, are turned on, or mounted yet.  It basically ignores all errors.  The only error that might be useful to report, is if an included folder does not exist, when the disk for it does exist.  In this one case, the user clearly wanted to cache something, but specified it wrongly, and should probably be informed.

No, /mnt/disk0 is not the parity drive.  The parity drive is not "mounted" but I had blindly copied and implemented that from the pseudo-code RobJ had posted earlier in the thread.    There is not, to my knowledge, any /mnt/disk0, so the line can be removed, it will never be hit.  There is a /mnt/user0 if you have a cache drive.  Perhaps he was thinking of it. (can't have an ego when a constructive code review is under way, can I  )

Sounds like an ignorant user wrote that pseudo-code!   

And an foolish developer wrote the code according to the specs given to him.... even if it never could work... 

 

Yes, we should alert users of spelling errors... that would be a nice addition. 

Now, to make the code understandable, and maintainable...  That's the trick.

 

Joe (no ego) L.

I must be doing something wrong as everyone else is reporting this utility is working great, but I can't seem to get it to work.  I cannot use ls -R due to the sheer number of folders and files that are on the system, so I was ecstatic to see a find with a depth limiting feature version, but it doesn't seem to read any of the folders.  When I run the command, none of my spun-down disks spin back up to try and read the folders and I am sure there are dozens of folders that are not cached.  Below is a brief output of what I am seeing:

 

root@Tower:/boot/utils# ./cache_dirs -F -d 4

Executed find -maxdepth 4 in 0.013799 seconds, avg=0.013799 seconds, now sleeping 5 seconds

Executed find -maxdepth 4 in 0.013808 seconds, avg=0.013804 seconds, now sleeping 4 seconds

Executed find -maxdepth 4 in 0.013591 seconds, avg=0.013733 seconds, now sleeping 5 seconds

root@Tower:/boot/utils# ./cache_dirs -F -d 7

Executed find -maxdepth 7 in 0.014136 seconds, avg=0.014136 seconds, now sleeping 5 seconds

Executed find -maxdepth 7 in 0.013572 seconds, avg=0.013854 seconds, now sleeping 6 seconds

Executed find -maxdepth 7 in 0.013779 seconds, avg=0.013829 seconds, now sleeping 7 seconds

Executed find -maxdepth 7 in 0.013557 seconds, avg=0.013761 seconds, now sleeping 8 seconds

 

I have tried several different depths, but nothing seems to be adding to the cached dirs list.  Am I doing something wrong?  I want to cache the first few levels of all of my user shares.