trurl Posted January 14, 2013 Share Posted January 14, 2013 ...It would be great if the disks started up in anticipation of a file being opened. How would it anticipate a file being opened? Quote Link to comment
Helmonder Posted January 14, 2013 Share Posted January 14, 2013 This will not work for several reasons: 1) If cache_dirs would spin up a disk you were browing on it would spin up all disks if you access a share that is spread over several disks, something that you do not want (or at least, the majority of users will not) 2) cache_dirs, though great in what it does, is basically just using a standard linux function, directories are kept in memory to speed up browsing, this is however removed from memory if the memory is needed somewhere else. Cache_dirs continuasly access the directories making the system believe they are constantly used and therefor the info stays in memory. However, this might help you: I to have a share that IF I browse will allmost allways also result in a file beiing accessed, for me this is my Music share (itunes). I have excluded this drive from cache_dirs, this means that the disk spins up as soon as the share is accessed. I have also made sure that this share is only on one disk so it does not spin up stuff unnecessary. Quote Link to comment
Joe L. Posted January 14, 2013 Author Share Posted January 14, 2013 ...It would be great if the disks started up in anticipation of a file being opened. How would it anticipate a file being opened? I use a different script that uses "lsof" to detect open files anticipating access. It works pretty well for me. spinup_when_accessed.sh It tracks activity browsing directories and spins up the affiliated drives. The script works by using lsof on /mnt/user to find the files and folders being accessed and then gets a list of the disks with the same top level folders (the user-shares) and then spins up the affiliated disks if they are not currently spinning. As soon as one of my media players performs a directory listing, or accesses a file, the affiliated drives are spun up. When I press "Play" the drives are ready and the media player does not time-out. A copy of the script is attached. type spinup_when_accessed.sh -? to see the usage help. Edit: fixed reported bug creating file named "0" on flash drive. New version attached. New version number is 1.4 type spinup_when_accessed.sh -v to see version number. Joe L. spinup_when_accessed.zip Quote Link to comment
dalben Posted January 14, 2013 Share Posted January 14, 2013 I use a different script that uses "lsof" to detect open files anticipating access. It works pretty well for me. spinup_when_accessed.sh It tracks activity browsing directories and spins up the affiliated drives. The script works by using lsof on /mnt/user to find the files and folders being accessed and then gets a list of the disks with the same top level folders (the user-shares) and then spins up the affiliated disks if they are not currently spinning. As soon as one of my media players performs a directory listing, or accesses a file, the affiliated drives are spun up. When I press "Play" the drives are ready and the media player does not time-out. A copy of the script is attached. type spinup_when_accessed.sh -? to see the usage help. Joe L. I'll have a look at this, thanks. It's doing exactly what I was after by the sound of it. Have a share set to spin up on directory access. I'm not sure if you're still developing cache_dirs but if this solution could be another option in that, it would be lovely. Chose which shares to ignore, which to cache only, and which to cache and spin up. I'm still trying to work out what xaminder found so funny though. Quote Link to comment
Helmonder Posted January 15, 2013 Share Posted January 15, 2013 I would also use this on selected shares.. Quote Link to comment
sambo Posted January 20, 2013 Share Posted January 20, 2013 Hi, I still have issue, sometimes disk spinup and i can see it on my ram graph. I just set my swappiness from 60 to 95, lets see if its better Someone already played with this value? Quote Link to comment
beckp Posted January 20, 2013 Share Posted January 20, 2013 Joe, There is an error in spinup_when_accessed.sh that could cause a premature flash drive failure. I noticed that the flash drive LED was constantly flashing, therefore the drive was constantly being accessed. I didn't seen that before. Reviewing the flash drive I noticed a file named "0" was the newest file. Line 316 is the offending line. It creates a file with the name "0". if [ $exclude_share_count > 0 ] it should be if [ $exclude_share_count -gt 0 ] PB Quote Link to comment
Joe L. Posted January 21, 2013 Author Share Posted January 21, 2013 Joe, There is an error in spinup_when_accessed.sh that could cause a premature flash drive failure. I noticed that the flash drive LED was constantly flashing, therefore the drive was constantly being accessed. I didn't seen that before. Reviewing the flash drive I noticed a file named "0" was the newest file. Line 316 is the offending line. It creates a file with the name "0". if [ $exclude_share_count > 0 ] it should be if [ $exclude_share_count -gt 0 ] PB Thanks. Corrected version of spinup_when_accessed.sh attached to prior post. Quote Link to comment
Joe L. Posted January 21, 2013 Author Share Posted January 21, 2013 I just posted a new version of cache_dirs that documents (and fixes) the previously undocumented "-p" option that allows you to set the "cache_pressure" kernel tunable parameter via a command line option. It is attached to the first post in this thread. The new version is 1.6.7. Type cache_dirs -V to see your version number. You do not need to replace your existing version unless you plan on experimenting with the -p option. (and trying different cache_pressure tunable values) Joe L. Quote Link to comment
MartinQ Posted January 21, 2013 Share Posted January 21, 2013 Corrected version attached to prior post. Joe, are you referring to the attachment on the first post (now v1.6.7) or the previous post where you attached 'spinup_when_accessed.zip' ? I don't see this issue addressed in the version notes: Version 1.6.7 - Added cache_pressure to "usage" statement, fixed bug where it reverted to 10 after being invoked through "at" when used with the -w option. Quote Link to comment
Joe L. Posted January 21, 2013 Author Share Posted January 21, 2013 Corrected version attached to prior post. Joe, are you referring to the attachment on the first post (now v1.6.7) or the previous post where you attached 'spinup_when_accessed.zip' ? I don't see this issue addressed in the version notes: Version 1.6.7 - Added cache_pressure to "usage" statement, fixed bug where it reverted to 10 after being invoked through "at" when used with the -w option. this refers to "cache_dirs" The version attached to the first post in this thread has the updated cache_dirs.zip. Basically, it just documents what was there all along, and fixes a bug that did exist if you used -p and -w in the same command line. If you never used -p, the bug did not affect you. Quote Link to comment
PeteAron Posted January 21, 2013 Share Posted January 21, 2013 I apologize for asking a stupid question, but I cannot figure out how to install cache_dirs. I have been searching for hours. I was interested in the unmenu plugin, but I see I dont know how to create a directory - I thought "md" was the command. I take this as a sign that I am an idiot and i need to ask for help to do this. Could someone point me directly to install instructions? The wiki page refers to 5 separate threads. It is very confusing. thanks! Quote Link to comment
Joe L. Posted January 21, 2013 Author Share Posted January 21, 2013 I apologize for asking a stupid question, but I cannot figure out how to install cache_dirs. I have been searching for hours. I was interested in the unmenu plugin, but I see I dont know how to create a directory - I thought "md" was the command. It is the command for MS-DOS, but not for linux/unix. The command for unMENU would be mkdir /boot/unmenu Apparently, you did not look here: http://code.google.com/p/unraid-unmenu/ or here http://lime-technology.com/forum/index.php?topic=5568.0 Both give the "mkdir" command. I take this as a sign that I am an idiot and i need to ask for help to do this. Could someone point me directly to install instructions? The wiki page refers to 5 separate threads. It is very confusing. thanks! Which "wiki" page are you referring to. Since the wiki can be edited by anyone, it is impossible for me to know where you looked. It probably was not this one http://lime-technology.com/wiki/index.php/UnRAID_Add_Ons#UnMENU Joe L. Quote Link to comment
PeteAron Posted January 21, 2013 Share Posted January 21, 2013 Joe, Actually I was not referring to Unmenu - I have used that for a long time. I was referring to cache_dirs. This thread talks about an unmenu plugin: http://lime-technology.com/forum/index.php?topic=19790.msg195098#msg195098 this is where I was trying to making a new directory. WRT the multiple posts on installation, the cache_dirs wiki in your OP in this thread: http://lime-technology.com/forum/index.php?topic=4500.msg40570#msg40570 refers to this wiki, http://lime-technology.com/wiki/index.php?title=Improving_unRAID_Performance#Keep_directory_entries_cached I will give mkdir a shot and follow the instructions in the first thread I linked. Edit: that wiki seems to have been modified today - there were a few threads referenced on customizing cache_dirs after the two bullet points, now they are not there. the OP in this thread discusses a lot about how cache_dirs works but not how to actually install it. I can't find cache_dirs in the unmenu package manager; thus my confusion. Thank you. I apologize for asking a stupid question, but I cannot figure out how to install cache_dirs. I have been searching for hours. I was interested in the unmenu plugin, but I see I dont know how to create a directory - I thought "md" was the command. It is the command for MS-DOS, but not for linux/unix. The command for unMENU would be mkdir /boot/unmenu Apparently, you did not look here: http://code.google.com/p/unraid-unmenu/ or here http://lime-technology.com/forum/index.php?topic=5568.0 Both give the "mkdir" command. I take this as a sign that I am an idiot and i need to ask for help to do this. Could someone point me directly to install instructions? The wiki page refers to 5 separate threads. It is very confusing. thanks! Which "wiki" page are you referring to. Since the wiki can be edited by anyone, it is impossible for me to know where you looked. It probably was not this one http://lime-technology.com/wiki/index.php/UnRAID_Add_Ons#UnMENU Joe L. Quote Link to comment
Joe L. Posted January 22, 2013 Author Share Posted January 22, 2013 There is no entry in unmenu's package manager for cache_dirs. It is a separate utility. The person who "borrowed" my code for cache_dirs and included it in their plugin did it without my explicit permission. (I wish they did not include my code, as it now confuses the user base.) It was me who updated the wiki today. The threads describing customization are still there, but moved later where there is less confusion. you must realize the program wen through a lot of changes and versions as we learned what worked and what did. Some of the wiki links were no longer best as they pointed to older versions. I know there is a plugin... ask the author for guidance on it. I have nothing to do with it or how to install it. I've never used it. I think it installs itself when you install the plugin. I've stayed away from the plugin system as it is NOWEHERE NEAR FINAL how it will work with unRAID. It is poorly implemented, there is no plugin manager, and more often than not, poorly created plugins either stop the array from stopping, or prevent it from starting. The entire "event" drive script is single threaded, and will stop the entire server from operating if any plugin misbehaves. It desperately needs a rewrite. It barely works... and as often as not, any issue stops the correct operation of the server starting/stopping. It is my intention, once there is a true plugin manager in 5.X, to possibly work on fixing the issues. I do not intend to sound off like this, so please accept my apology in advance. Frankly, I'm a bit offended by the statement in the plugin thread you linked to: Please report any problems with the plugin here. If you have specific issues or feature requests for the cache_dirs script, please direct them here http://lime-technology.com/forum/index.php?topic=4500.0 To me, it is saying... I stoletook Joe's code, and included it in my plugin. If you have any issues, or want improvements, see Joe, as I have no idea what it does. For now, This thread http://lime-technology.com/forum/index.php?topic=4500.0 and this wiki entry http://www.lime-technology.com/wiki/index.php/Improving_unRAID_Performance#Keep_directory_entries_cached are most current. Joe L. Quote Link to comment
PeteAron Posted January 22, 2013 Share Posted January 22, 2013 Joe, thanks for your help. I am not complaining and I am sensitive to the amount of flak you get from folks (some of us need to read Dale Carnegie's book IMO). The problem is that my ignorance gets in the way of implementing things outside of unmenu =). I was confused, as you say, by the plugin thread, and I have learned to stick with the basic unmenu plugins. Some of the behavior I've seen makes sense now that you've pointed out to me the single threaded nature of the process. I'll continue to be conservative with it. I should be able now to implement cache_dirs on my server. Thanks again. Quote Link to comment
olympia Posted January 25, 2013 Share Posted January 25, 2013 Hello Joe L. I am trying to troubleshoot a spin down issue, detailed here: http://lime-technology.com/forum/index.php?topic=25250.msg222150#msg222150 I have 6GB memory. Would it be possible by any chance that cache_dirs go mad by the high memory issue on rc10 (I don't seem to have the slow down issue though) and could somehow cause such issue? It also occured once, that cache_dirs was - I beleive - crashing overnight, because it disappeard from the running processes when I checked in the morning without anything in the log. Quote Link to comment
Joe L. Posted January 25, 2013 Author Share Posted January 25, 2013 Hello Joe L. I am trying to troubleshoot a spin down issue, detailed here: http://lime-technology.com/forum/index.php?topic=25250.msg222150#msg222150 I have 6GB memory. Would it be possible by any chance that cache_dirs go mad by the high memory issue on rc10 (I don't seem to have the slow down issue though) No, it is just a shell script. that invokes the linux"find" command every few seconds. It can be killed off by the kernel if there is insufficient memory, but you would see evidence in the syslog. Quote Link to comment
olympia Posted January 28, 2013 Share Posted January 28, 2013 Thanks Joe L., I finally sorted it was the good-old smarthistory which caused the weird behaviour with the spin downs. Quote Link to comment
Heretic Posted January 30, 2013 Share Posted January 30, 2013 Hi, i'm trying to figure out why my drives always spin up around 04:41 does cache directories do extra scheduled scans? if not I have to look elsewhere Jan 28 04:41:00 Whaaat kernel: mdcmd (2021): spindown 0 (Routine) Jan 28 04:41:01 Whaaat kernel: mdcmd (2022): spindown 1 (Routine) Jan 28 04:41:01 Whaaat kernel: mdcmd (2023): spindown 3 (Routine) Jan 28 04:41:03 Whaaat kernel: mdcmd (2024): spindown 4 (Routine) Jan 28 04:41:04 Whaaat kernel: mdcmd (2025): spindown 5 (Routine) Jan 28 04:41:04 Whaaat kernel: mdcmd (2026): spindown 6 (Routine) Jan 28 04:41:05 Whaaat kernel: mdcmd (2027): spindown 7 (Routine) thanks for your time Quote Link to comment
RobJ Posted January 30, 2013 Share Posted January 30, 2013 Hi, i'm trying to figure out why my drives always spin up around 04:41 does cache directories do extra scheduled scans? if not I have to look elsewhere Jan 28 04:41:00 Whaaat kernel: mdcmd (2021): spindown 0 (Routine) Jan 28 04:41:01 Whaaat kernel: mdcmd (2022): spindown 1 (Routine) Jan 28 04:41:01 Whaaat kernel: mdcmd (2023): spindown 3 (Routine) Jan 28 04:41:03 Whaaat kernel: mdcmd (2024): spindown 4 (Routine) Jan 28 04:41:04 Whaaat kernel: mdcmd (2025): spindown 5 (Routine) Jan 28 04:41:04 Whaaat kernel: mdcmd (2026): spindown 6 (Routine) Jan 28 04:41:05 Whaaat kernel: mdcmd (2027): spindown 7 (Routine) That looks like one hour after the mover runs, so you probably have the spindown timer set to one hour. So the question for you becomes - why are all of my drives except Disk 2 spinning up for the mover? Quote Link to comment
Heretic Posted January 31, 2013 Share Posted January 31, 2013 thanks that should help i'll dive into the settings to see what going on disk2 also spins up and down like to rest for some reason it didn't in the little sample log i posted not sure why. the cache disks contains only a .torrent folder i'll try to leave the mover info blank to see if that disables the mover. i think i read unraid 5 has cache only shares maybe that helps since this is all oftopic now and 5.0 is around the corner i'll just wait for that. again thanks for pointing me in the right direction Quote Link to comment
sambo Posted February 8, 2013 Share Posted February 8, 2013 Just figured out the same issue as Heretic. I have disable this crontab line: # Run daily cron jobs at 4:40 every day: #40 4 * * * /usr/bin/run-parts /etc/cron.daily 1> /dev/null Is it the good way to fix this issue? I dont have a cache drive between Thanks Jan 27 05:37:05 Tower kernel: mdcmd (55): spindown 3 Jan 27 05:37:06 Tower kernel: mdcmd (56): spindown 4 Jan 28 05:34:06 Tower kernel: mdcmd (79): spindown 3 Jan 28 05:34:06 Tower kernel: mdcmd (80): spindown 4 Jan 29 05:34:51 Tower kernel: mdcmd (84): spindown 3 Jan 29 05:35:22 Tower kernel: mdcmd (85): spindown 4 Feb 1 05:33:25 Tower kernel: mdcmd (92): spindown 3 Feb 1 05:33:26 Tower kernel: mdcmd (93): spindown 4 Feb 3 05:33:16 Tower kernel: mdcmd (98): spindown 2 Feb 4 05:33:36 Tower kernel: mdcmd (104): spindown 3 Feb 4 05:33:37 Tower kernel: mdcmd (105): spindown 4 Feb 5 05:33:45 Tower kernel: mdcmd (106): spindown 3 Quote Link to comment
RobJ Posted February 8, 2013 Share Posted February 8, 2013 Just figured out the same issue as Heretic. I have disable this crontab line: # Run daily cron jobs at 4:40 every day: #40 4 * * * /usr/bin/run-parts /etc/cron.daily 1> /dev/null Is it the good way to fix this issue? I dont have a cache drive between Jan 27 05:37:05 Tower kernel: mdcmd (55): spindown 3 Jan 27 05:37:06 Tower kernel: mdcmd (56): spindown 4 Jan 28 05:34:06 Tower kernel: mdcmd (79): spindown 3 Jan 28 05:34:06 Tower kernel: mdcmd (80): spindown 4 Jan 29 05:34:51 Tower kernel: mdcmd (84): spindown 3 Jan 29 05:35:22 Tower kernel: mdcmd (85): spindown 4 Feb 1 05:33:25 Tower kernel: mdcmd (92): spindown 3 Feb 1 05:33:26 Tower kernel: mdcmd (93): spindown 4 Feb 3 05:33:16 Tower kernel: mdcmd (98): spindown 2 Feb 4 05:33:36 Tower kernel: mdcmd (104): spindown 3 Feb 4 05:33:37 Tower kernel: mdcmd (105): spindown 4 Feb 5 05:33:45 Tower kernel: mdcmd (106): spindown 3 Unfortunately that disables all daily cron jobs. It would be better to examine /etc/cron.daily and determine what may be running that is causing the disk usage. Also, if cron was set to run daily jobs at 4:40am and your spindown delay is one hour, then the drives would spin down after 5:40am, not before that. What is your default spindown delay? If one hour, then it looks like something is running at 4:30am and accessing for several minutes Disk 3 and Disk 4, and once in a while Disk 2. It ran for 4 to 7 minutes last week, but only about 3 and a half minutes this week. Quote Link to comment
Joe L. Posted February 8, 2013 Author Share Posted February 8, 2013 Just figured out the same issue as Heretic. I have disable this crontab line: # Run daily cron jobs at 4:40 every day: #40 4 * * * /usr/bin/run-parts /etc/cron.daily 1> /dev/null Is it the good way to fix this issue? I dont have a cache drive between Jan 27 05:37:05 Tower kernel: mdcmd (55): spindown 3 Jan 27 05:37:06 Tower kernel: mdcmd (56): spindown 4 Jan 28 05:34:06 Tower kernel: mdcmd (79): spindown 3 Jan 28 05:34:06 Tower kernel: mdcmd (80): spindown 4 Jan 29 05:34:51 Tower kernel: mdcmd (84): spindown 3 Jan 29 05:35:22 Tower kernel: mdcmd (85): spindown 4 Feb 1 05:33:25 Tower kernel: mdcmd (92): spindown 3 Feb 1 05:33:26 Tower kernel: mdcmd (93): spindown 4 Feb 3 05:33:16 Tower kernel: mdcmd (98): spindown 2 Feb 4 05:33:36 Tower kernel: mdcmd (104): spindown 3 Feb 4 05:33:37 Tower kernel: mdcmd (105): spindown 4 Feb 5 05:33:45 Tower kernel: mdcmd (106): spindown 3 Unfortunately that disables all daily cron jobs. It would be better to examine /etc/cron.daily and determine what may be running that is causing the disk usage. Also, if cron was set to run daily jobs at 4:40am and your spindown delay is one hour, then the drives would spin down after 5:40am, not before that. What is your default spindown delay? If one hour, then it looks like something is running at 4:30am and accessing for several minutes Disk 3 and Disk 4, and once in a while Disk 2. It ran for 4 to 7 minutes last week, but only about 3 and a half minutes this week. It is not really a mystery Rob. The "mover" command on 4.7 invokes "sync" which will spin up all the drives. The "mover" command on 5.X is completely different in how it woks and does not invoke "sync" There is a unMENU package that when installed will add logic to the "mover" script on 4.7 that will cause it to only invoke "sync" if a file is actually moved. Joe L. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.