cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up

[trurl]

...It would be great if the disks started up in anticipation of a file being opened.

How would it anticipate a file being opened?

[Helmonder]

This will not work for several reasons:

 

1) If cache_dirs would spin up a disk you were browing on it would spin up all disks if you access a share that is spread over several disks, something that you do not want (or at least, the majority of users will not)

 

2) cache_dirs, though great in what it does, is basically just using a standard linux function, directories are kept in memory to speed up browsing, this is however removed from memory if the memory is needed somewhere else. Cache_dirs continuasly access the directories making the system believe they are constantly used and therefor the info stays in memory.

 

However, this might help you: I to have a share that IF I browse will allmost allways also result in a file beiing accessed, for me this is my Music share (itunes). I have excluded this drive from cache_dirs, this means that the disk spins up as soon as the share is accessed. I have also made sure that this share is only on one disk so it does not spin up stuff unnecessary.

[Joe L.]

...It would be great if the disks started up in anticipation of a file being opened.

How would it anticipate a file being opened?

I use a different script that uses "lsof" to detect open files anticipating access.  It works pretty well for me.

spinup_when_accessed.sh

It tracks activity browsing directories and spins up the affiliated drives.

 

The script works by using lsof on /mnt/user to find the files and folders

being accessed and then gets a list of the disks with the same top level

folders (the user-shares) and then spins up the affiliated disks if they

are not currently spinning.

 

As soon as one of my media players performs a directory listing,

or accesses a file, the affiliated drives are spun up. When I press

"Play" the drives are ready and the media player does not time-out.

 

A copy of the script is attached.

type

spinup_when_accessed.sh -?

to see the usage help.

 

Edit: fixed reported bug creating file named "0" on flash drive.  New version attached.  New version number is 1.4

type

spinup_when_accessed.sh -v

to see version number.

 

Joe L.

spinup_when_accessed.zip

[dalben]

I use a different script that uses "lsof" to detect open files anticipating access.  It works pretty well for me.

spinup_when_accessed.sh

It tracks activity browsing directories and spins up the affiliated drives.

 

The script works by using lsof on /mnt/user to find the files and folders

being accessed and then gets a list of the disks with the same top level

folders (the user-shares) and then spins up the affiliated disks if they

are not currently spinning.

 

As soon as one of my media players performs a directory listing,

or accesses a file, the affiliated drives are spun up. When I press

"Play" the drives are ready and the media player does not time-out.

 

A copy of the script is attached.

type

spinup_when_accessed.sh -?

to see the usage help.

 

Joe L.

 

I'll have a look at this, thanks. It's doing exactly what I was after by the sound of it.  Have a share set to spin up on directory access.

 

I'm not sure if you're still developing cache_dirs but if this solution could be another option in that, it would be lovely.  Chose which shares to ignore, which to cache only, and which to cache and spin up.

 

I'm still trying to work out what xaminder found so funny though.

[Helmonder]

I would also use this on selected shares..

[sambo]

Hi,

 

I still have issue, sometimes disk spinup and i can see it on my ram graph.

I just set my swappiness from 60 to 95, lets see if its better

 

Someone already played with this value?

[beckp]

Joe,

 

There is an error in spinup_when_accessed.sh that could cause a premature flash drive failure. I noticed that the flash drive LED was constantly flashing, therefore the drive was constantly being accessed. I didn't  seen that before. Reviewing the flash drive I noticed a file named "0" was the newest file.

 

Line 316 is the offending line. It creates a file with the name "0".

 

if [ $exclude_share_count > 0 ]

 

it should be

 

if [ $exclude_share_count -gt 0 ]

 

PB

 

[Joe L.]

Joe,

 

There is an error in spinup_when_accessed.sh that could cause a premature flash drive failure. I noticed that the flash drive LED was constantly flashing, therefore the drive was constantly being accessed. I didn't  seen that before. Reviewing the flash drive I noticed a file named "0" was the newest file.

 

Line 316 is the offending line. It creates a file with the name "0".

 

if [ $exclude_share_count > 0 ]

 

it should be

 

if [ $exclude_share_count -gt 0 ]

 

PB

Thanks.

 

Corrected version of spinup_when_accessed.sh attached to prior post.

[Joe L.]

I just posted a new version of cache_dirs that documents (and fixes) the previously undocumented "-p" option that allows you to set the "cache_pressure" kernel tunable parameter via a command line option.  It is attached to the first post in this thread.  The new version is 1.6.7.

Type

cache_dirs -V

to see your version number. 

 

You do not need to replace your existing version unless you plan on experimenting with the -p option. (and trying different cache_pressure tunable values)

 

Joe L.

[MartinQ]

Corrected version attached to prior post.

Joe, are you referring to the attachment on the first post (now v1.6.7) or the previous post where you attached 'spinup_when_accessed.zip'  ?

 

I don't see this issue addressed in the version notes:

 

Version 1.6.7 - Added cache_pressure to "usage" statement, fixed bug where it reverted to 10 after being invoked through "at" when used with the -w option.

[Joe L.]

Corrected version attached to prior post.

Joe, are you referring to the attachment on the first post (now v1.6.7) or the previous post where you attached 'spinup_when_accessed.zip'  ?

 

I don't see this issue addressed in the version notes:

 

Version 1.6.7 - Added cache_pressure to "usage" statement, fixed bug where it reverted to 10 after being invoked through "at" when used with the -w option.

this refers to "cache_dirs" 

 

The version attached to the first post in this thread has the updated cache_dirs.zip.  Basically, it just documents what was there all along, and fixes a bug that did exist if you used -p and -w in the same command line.  If you never used -p, the bug did not affect you.

[PeteAron]

I apologize for asking a stupid question, but I cannot figure out how to install cache_dirs.  I have been searching for hours.  I was interested in the unmenu plugin, but I see I dont know how to create a directory - I thought "md" was the command.  I take this as a sign that I am an idiot and i need to ask for help to do this.

 

Could someone point me directly to install instructions?  The wiki page refers to 5 separate threads.  It is very confusing.

 

thanks!

[Joe L.]

I apologize for asking a stupid question, but I cannot figure out how to install cache_dirs.  I have been searching for hours.  I was interested in the unmenu plugin, but I see I dont know how to create a directory - I thought "md" was the command.

It is the command for MS-DOS, but not for linux/unix.

 

The command for unMENU would be

mkdir /boot/unmenu

 

Apparently, you did not look here:

http://code.google.com/p/unraid-unmenu/

or here

http://lime-technology.com/forum/index.php?topic=5568.0

Both give the "mkdir" command.

 

I take this as a sign that I am an idiot and i need to ask for help to do this.

 

Could someone point me directly to install instructions?  The wiki page refers to 5 separate threads.  It is very confusing.

 

thanks!

Which "wiki" page are you referring to.  Since the wiki can be edited by anyone, it is  impossible for me to know where you looked.  It probably was not this one

http://lime-technology.com/wiki/index.php/UnRAID_Add_Ons#UnMENU

 

Joe L.

[PeteAron]

Joe,

 

Actually I was not referring to Unmenu - I have used that for a long time.  I was referring to cache_dirs. 

 

This thread talks about an unmenu plugin:

 

http://lime-technology.com/forum/index.php?topic=19790.msg195098#msg195098

 

this is where I was trying to making a new directory. 

 

WRT the multiple posts on installation, the cache_dirs wiki in your OP in this thread:

 

http://lime-technology.com/forum/index.php?topic=4500.msg40570#msg40570

 

refers to this wiki,

 

http://lime-technology.com/wiki/index.php?title=Improving_unRAID_Performance#Keep_directory_entries_cached

 

I will give mkdir a shot and follow the instructions in the first thread I linked.

 

Edit:  that wiki seems to have been modified today - there were a few threads referenced on customizing cache_dirs after the two bullet points, now they are not there. 

 

the OP in this thread discusses a lot about how cache_dirs works but not how to actually install it.  I can't find cache_dirs in the unmenu package manager; thus my confusion.

 

Thank you.

 

I apologize for asking a stupid question, but I cannot figure out how to install cache_dirs.  I have been searching for hours.  I was interested in the unmenu plugin, but I see I dont know how to create a directory - I thought "md" was the command.

It is the command for MS-DOS, but not for linux/unix.

 

The command for unMENU would be

mkdir /boot/unmenu

 

Apparently, you did not look here:

http://code.google.com/p/unraid-unmenu/

or here

http://lime-technology.com/forum/index.php?topic=5568.0

Both give the "mkdir" command.

 

I take this as a sign that I am an idiot and i need to ask for help to do this.

 

Could someone point me directly to install instructions?  The wiki page refers to 5 separate threads.  It is very confusing.

 

thanks!

Which "wiki" page are you referring to.  Since the wiki can be edited by anyone, it is  impossible for me to know where you looked.  It probably was not this one

http://lime-technology.com/wiki/index.php/UnRAID_Add_Ons#UnMENU

 

Joe L.

[Joe L.]

There is no entry in unmenu's package manager for cache_dirs.    It is a separate utility.

 

The person who "borrowed" my code for cache_dirs and included it in their plugin did it without my explicit permission.  (I wish they did not include my code, as it now confuses the user base.)

 

It was me who updated the wiki today.  The threads describing customization are still there, but moved later where there is less confusion.    you must realize the program wen through a lot of changes and versions as we learned what worked and what did.  Some of the wiki links were no longer best as they pointed to older versions.

 

I know there is a plugin...  ask the author for guidance on it.  I have nothing to do with it or how to install it.  I've never used it.  I think it installs itself when you install the plugin.

 

I've stayed away from the plugin system as it is NOWEHERE NEAR FINAL how it will work with unRAID.  It is poorly implemented, there is no plugin manager, and more often than not, poorly created plugins either stop the array from stopping, or prevent it from starting.  The entire "event" drive script is single threaded, and will stop the entire server from operating if any plugin misbehaves.    It desperately needs a rewrite.  It barely works... and as often as not, any issue stops the correct operation of the server starting/stopping.    It is my intention, once there is a true plugin manager in 5.X, to possibly work on fixing the issues. 

 

I do not intend to sound off like this, so please accept my apology in advance.  Frankly, I'm a bit offended by the statement in the plugin thread you linked to:

Please report any problems with the plugin here. If you have specific issues or feature requests for the cache_dirs script, please direct them here http://lime-technology.com/forum/index.php?topic=4500.0

To me, it is saying...  I stoletook Joe's code, and included it in my plugin.  If you have any issues, or want improvements, see Joe, as I have no idea what it does.

 

For now, This thread http://lime-technology.com/forum/index.php?topic=4500.0

and this wiki entry http://www.lime-technology.com/wiki/index.php/Improving_unRAID_Performance#Keep_directory_entries_cached are most current.

 

Joe L.

 

 

[PeteAron]

Joe, thanks for your help.  I am not complaining and I am sensitive to the amount of flak you get from folks (some of us need to read Dale Carnegie's book IMO).  The problem is that my ignorance gets in the way of implementing things outside of unmenu =).  I was confused, as you say, by the plugin thread, and I have learned to stick with the basic unmenu plugins.  Some of the behavior I've seen makes sense now that you've pointed out to me the single threaded nature of the process.  I'll continue to be conservative with it. 

 

I should be able now to implement cache_dirs on my server.  Thanks again.

 

 

[olympia]

Hello Joe L.

 

I am trying to troubleshoot a spin down issue, detailed here:

http://lime-technology.com/forum/index.php?topic=25250.msg222150#msg222150

 

I have 6GB memory. Would it be possible by any chance that cache_dirs go mad by the high memory issue on rc10 (I don't seem to have the slow down issue though) and could somehow cause such issue? It also occured once, that cache_dirs was - I beleive - crashing overnight, because it disappeard from the running processes when I checked in the morning without anything in the log.

[Joe L.]

Hello Joe L.

 

I am trying to troubleshoot a spin down issue, detailed here:

http://lime-technology.com/forum/index.php?topic=25250.msg222150#msg222150

 

I have 6GB memory. Would it be possible by any chance that cache_dirs go mad by the high memory issue on rc10 (I don't seem to have the slow down issue though)

 

No, it is just a shell script. that invokes the linux"find" command every few seconds.

 

It can be killed off by the kernel if there is insufficient memory, but you would see evidence in the syslog. 

[olympia]

Thanks Joe L., I finally sorted it was the good-old smarthistory which caused the weird behaviour with the spin downs.

[Heretic]

Hi, i'm trying to figure out why my drives always spin up around 04:41

does cache directories do extra scheduled scans?

if not I have to look elsewhere

 

 

Jan 28 04:41:00 Whaaat kernel: mdcmd (2021): spindown 0 (Routine)
Jan 28 04:41:01 Whaaat kernel: mdcmd (2022): spindown 1 (Routine)
Jan 28 04:41:01 Whaaat kernel: mdcmd (2023): spindown 3 (Routine)
Jan 28 04:41:03 Whaaat kernel: mdcmd (2024): spindown 4 (Routine)
Jan 28 04:41:04 Whaaat kernel: mdcmd (2025): spindown 5 (Routine)
Jan 28 04:41:04 Whaaat kernel: mdcmd (2026): spindown 6 (Routine)
Jan 28 04:41:05 Whaaat kernel: mdcmd (2027): spindown 7 (Routine)

 

 

thanks for your time

[RobJ]

Hi, i'm trying to figure out why my drives always spin up around 04:41

does cache directories do extra scheduled scans?

if not I have to look elsewhere

 

Jan 28 04:41:00 Whaaat kernel: mdcmd (2021): spindown 0 (Routine)
Jan 28 04:41:01 Whaaat kernel: mdcmd (2022): spindown 1 (Routine)
Jan 28 04:41:01 Whaaat kernel: mdcmd (2023): spindown 3 (Routine)
Jan 28 04:41:03 Whaaat kernel: mdcmd (2024): spindown 4 (Routine)
Jan 28 04:41:04 Whaaat kernel: mdcmd (2025): spindown 5 (Routine)
Jan 28 04:41:04 Whaaat kernel: mdcmd (2026): spindown 6 (Routine)
Jan 28 04:41:05 Whaaat kernel: mdcmd (2027): spindown 7 (Routine)

 

That looks like one hour after the mover runs, so you probably have the spindown timer set to one hour.  So the question for you becomes - why are all of my drives except Disk 2 spinning up for the mover?

[Heretic]

thanks that should help i'll dive into the settings to see what going on

 

disk2 also spins up and down like to rest for some reason it didn't in the little sample log i posted not sure why.

 

the cache disks contains only a .torrent folder

 

i'll try to leave the mover info blank to see if that disables the mover.

 

i think i read unraid 5 has cache only shares maybe that helps

 

since this is all oftopic now and 5.0  is around the corner i'll just wait for that. again thanks for pointing me in the right direction

 

 

 

[sambo]

Just figured out the same issue as Heretic.

I have disable this crontab line:

 

# Run daily cron jobs at 4:40 every day:

#40 4 * * * /usr/bin/run-parts /etc/cron.daily 1> /dev/null

 

 

Is it the good way to fix this issue?

I dont have a cache drive between

 

Thanks

 

 

Jan 27 05:37:05 Tower kernel: mdcmd (55): spindown 3

Jan 27 05:37:06 Tower kernel: mdcmd (56): spindown 4

Jan 28 05:34:06 Tower kernel: mdcmd (79): spindown 3

Jan 28 05:34:06 Tower kernel: mdcmd (80): spindown 4

Jan 29 05:34:51 Tower kernel: mdcmd (84): spindown 3

Jan 29 05:35:22 Tower kernel: mdcmd (85): spindown 4

Feb  1 05:33:25 Tower kernel: mdcmd (92): spindown 3

Feb  1 05:33:26 Tower kernel: mdcmd (93): spindown 4

Feb  3 05:33:16 Tower kernel: mdcmd (98): spindown 2

Feb  4 05:33:36 Tower kernel: mdcmd (104): spindown 3

Feb  4 05:33:37 Tower kernel: mdcmd (105): spindown 4

Feb  5 05:33:45 Tower kernel: mdcmd (106): spindown 3

 

 

 

[RobJ]

Just figured out the same issue as Heretic.

I have disable this crontab line:

 

# Run daily cron jobs at 4:40 every day:

#40 4 * * * /usr/bin/run-parts /etc/cron.daily 1> /dev/null

 

Is it the good way to fix this issue?

I dont have a cache drive between

 

Jan 27 05:37:05 Tower kernel: mdcmd (55): spindown 3

Jan 27 05:37:06 Tower kernel: mdcmd (56): spindown 4

Jan 28 05:34:06 Tower kernel: mdcmd (79): spindown 3

Jan 28 05:34:06 Tower kernel: mdcmd (80): spindown 4

Jan 29 05:34:51 Tower kernel: mdcmd (84): spindown 3

Jan 29 05:35:22 Tower kernel: mdcmd (85): spindown 4

Feb  1 05:33:25 Tower kernel: mdcmd (92): spindown 3

Feb  1 05:33:26 Tower kernel: mdcmd (93): spindown 4

Feb  3 05:33:16 Tower kernel: mdcmd (98): spindown 2

Feb  4 05:33:36 Tower kernel: mdcmd (104): spindown 3

Feb  4 05:33:37 Tower kernel: mdcmd (105): spindown 4

Feb  5 05:33:45 Tower kernel: mdcmd (106): spindown 3

 

Unfortunately that disables all daily cron jobs.  It would be better to examine /etc/cron.daily and determine what may be running that is causing the disk usage.  Also, if cron was set to run daily jobs at 4:40am and your spindown delay is one hour, then the drives would spin down after 5:40am, not before that.  What is your default spindown delay?  If one hour, then it looks like something is running at 4:30am and accessing for several minutes Disk 3 and Disk 4, and once in a while Disk 2.  It ran for 4 to 7 minutes last week, but only about 3 and a half minutes this week.

[Joe L.]

Just figured out the same issue as Heretic.

I have disable this crontab line:

 

# Run daily cron jobs at 4:40 every day:

#40 4 * * * /usr/bin/run-parts /etc/cron.daily 1> /dev/null

 

Is it the good way to fix this issue?

I dont have a cache drive between

 

Jan 27 05:37:05 Tower kernel: mdcmd (55): spindown 3

Jan 27 05:37:06 Tower kernel: mdcmd (56): spindown 4

Jan 28 05:34:06 Tower kernel: mdcmd (79): spindown 3

Jan 28 05:34:06 Tower kernel: mdcmd (80): spindown 4

Jan 29 05:34:51 Tower kernel: mdcmd (84): spindown 3

Jan 29 05:35:22 Tower kernel: mdcmd (85): spindown 4

Feb  1 05:33:25 Tower kernel: mdcmd (92): spindown 3

Feb  1 05:33:26 Tower kernel: mdcmd (93): spindown 4

Feb  3 05:33:16 Tower kernel: mdcmd (98): spindown 2

Feb  4 05:33:36 Tower kernel: mdcmd (104): spindown 3

Feb  4 05:33:37 Tower kernel: mdcmd (105): spindown 4

Feb  5 05:33:45 Tower kernel: mdcmd (106): spindown 3

 

Unfortunately that disables all daily cron jobs.  It would be better to examine /etc/cron.daily and determine what may be running that is causing the disk usage.  Also, if cron was set to run daily jobs at 4:40am and your spindown delay is one hour, then the drives would spin down after 5:40am, not before that.  What is your default spindown delay?  If one hour, then it looks like something is running at 4:30am and accessing for several minutes Disk 3 and Disk 4, and once in a while Disk 2.  It ran for 4 to 7 minutes last week, but only about 3 and a half minutes this week.

It is not really a mystery Rob.

 

The "mover" command on 4.7 invokes "sync" which will spin up all the drives. 

 

The  "mover" command on 5.X is completely different in how it woks and does not invoke "sync"

There is a unMENU package that when installed will add logic to the "mover" script on 4.7 that will cause it to only invoke "sync" if a file is actually moved.

 

Joe L.