KentBrockman Posted December 27, 2009 Share Posted December 27, 2009 Wont mediabrowser be accessing the mymovies.xml files in each folder to verify it is current? That is corret. My media frontend (SageTV) also reads the cover images of all my videos. So the caching script that I am using copies all .jpg files it finds to /dev/null Purko Purko, Could you please explain how you modified this script? I would love to have a way to give media browser instant access to the.xml and folder.jpg files so that all my drives don't have to spin up when browsing my library. Thanks Quote Link to comment
starcat Posted January 7, 2010 Share Posted January 7, 2010 I am noticing some strange phenomenom. When I call cache_dirs -w -B from the go script, my drives do not spin down. When I kill the process and start it manually on the cmdline, same syntax, they will properly spin down after the specified amount of time. Any hints as to what might be causing this behaviour? For what it matters I am calling cache_dirs -w -B right before calling unMenu in my go script. Quote Link to comment
Joe L. Posted January 7, 2010 Author Share Posted January 7, 2010 I am noticing some strange phenomenom. When I call cache_dirs -w -B from the go script, my drives do not spin down. When I kill the process and start it manually on the cmdline, same syntax, they will properly spin down after the specified amount of time. Any hints as to what might be causing this behaviour? For what it matters I am calling cache_dirs -w -B right before calling unMenu in my go script. I've got no idea... One difference I can think of is the search PATH used for commands might be different. (Causing the shell to traverse different directories when looking for the commands to run) My cache_dirs lives in /boot/custom/bin. I invoke it like this in my "go" script. my drives spin down just fine. I add /boot/custom/bin to the execution path in the "go" script. (This is my whole "go" script.) #!/bin/bash # Start the Management Utility /usr/local/sbin/emhttp & PATH=$PATH:/boot/custom/bin cache_dirs -w -e Pictures -e data -d 3 fromdos < /boot/custom/bin/powerdown > /sbin/powerdown chmod u+x /sbin/powerdown fromdos < /boot/custom/etc/rc.d/S30-inittab-powerdown | sh fromdos < /boot/custom/etc/rc.d/rc.local_startup | sh echo "/boot/unmenu/uu" | at now + 1 minute What version of cache_dirs are you running? What version of unRAID are you running? Other than that, something is either 1. Accessing the disks, keeping them spinning or 2. Not finding the needed directory information in the buffer cache, so it has to spin up a drive to get the data. or 3. Your observation, although true, might have nothing to do with the actual issue. It might be something entirely un-related to how you invoke it. Quote Link to comment
starcat Posted January 7, 2010 Share Posted January 7, 2010 I have cache_dirs 1.6.4 and unRAID 4.5 final. fuser don't show any processes on the /mnt/disk*. Also, I have about 130k files and 4GB memory. I don't have $PATH in my go. I just start /boot/scripts/cache_dirs -w -B Is it ok to have two cache_dir processes running? The one starts the other, it has its PID as parents PID in the second. root 300 1 0 Jan07 pts/1 00:00:00 /bin/bash /boot/scripts/cache_dirs -w -B root 3874 300 0 00:38 pts/1 00:00:00 /bin/bash /boot/scripts/cache_dirs -w -B Quote Link to comment
Joe L. Posted January 8, 2010 Author Share Posted January 8, 2010 I have cache_dirs 1.6.4 and unRAID 4.5 final. fuser don't show any processes on the /mnt/disk*. Also, I have about 130k files and 4GB memory. I don't have $PATH in my go. I just start /boot/scripts/cache_dirs -w -B Is it ok to have two cache_dir processes running? The one starts the other, it has its PID as parents PID in the second. root 300 1 0 Jan07 pts/1 00:00:00 /bin/bash /boot/scripts/cache_dirs -w -B root 3874 300 0 00:38 pts/1 00:00:00 /bin/bash /boot/scripts/cache_dirs -w -B I'm running the same versions. I only cache three levels deep. I only have 512 meg of RAM in my server and under 5000 files in my Movies and Music folders. You have lots more files you are attempting to cache, but far more memory too. Yes, it is normal to see two processes. part of the script is run in a sub-shell. (The while loop is run as a sub-shell) Quote Link to comment
starcat Posted January 8, 2010 Share Posted January 8, 2010 It used to work when I first installed/started it by hand. I added it to the go script and then after time had to reboot adding a new drive. Then I noticed that the drives won't spin down. Killed, started by hand and it worked again. Restarted again the server, same - no spin down. However, since last manual restart drives are still spinning and I don't know why. Quote Link to comment
Joe L. Posted January 8, 2010 Author Share Posted January 8, 2010 I have cache_dirs 1.6.4 and unRAID 4.5 final. fuser don't show any processes on the /mnt/disk* fuser is a snapshot in time. If at that instant in time no access was occurring, then nothing will show. It is not able to show you things that happened 10 minutes ago and lasted a fraction of a second as a directory or file was accessed.. For that, we need something that can track all file and directory accesses, and log all of them. Quote Link to comment
starcat Posted January 8, 2010 Share Posted January 8, 2010 lsof? It too doesn't show any processes being on /mnt/disk* Edit: I was also wondering what is the default access mode for files on unRAID? I have ownership set for everything to root:root and access rights to 750 for dirs and 640 for files. Can remember that I was changing this lately... But shouldn't have to do anything with the strange behaviour described above?! Edit2: Well, it is definitely cache_dirs spinning up the drives. I sync'ed on the cmdline a couple of times and them spun down the array from unRAID Main. After some time, the disks went up again, one at a time corresponding to the find processes started from cache_dirs... Edit3: I am running unraid_notify, can this be the cause? It says it won't send notification if disks are spun down though, wouldn't do anything to them to spin them up. Quote Link to comment
Guzzi Posted January 8, 2010 Share Posted January 8, 2010 I have cache_dirs 1.6.4 and unRAID 4.5 final. fuser don't show any processes on the /mnt/disk*. Also, I have about 130k files and 4GB memory. I don't have $PATH in my go. I just start /boot/scripts/cache_dirs -w -B Is it ok to have two cache_dir processes running? The one starts the other, it has its PID as parents PID in the second. root 300 1 0 Jan07 pts/1 00:00:00 /bin/bash /boot/scripts/cache_dirs -w -B root 3874 300 0 00:38 pts/1 00:00:00 /bin/bash /boot/scripts/cache_dirs -w -B I'm running the same versions. I only cache three levels deep. I only have 512 meg of RAM in my server and under 5000 files in my Movies and Music folders. You have lots more files you are attempting to cache, but far more memory too. Yes, it is normal to see two processes. part of the script is run in a sub-shell. (The while loop is run as a sub-shell) Hi Joe, I also have lots of files and directories on my server and experience a lot of reading on the disks. I upgraded memory to make sure, it is sufficient to hold all dir-infos. Nevertheless my disks DO spindown (all), but often reread all contents. Might be caused, when I write to the array so the buffers are reused for writing? So my question (I am Linux noob....): Is there a way/tools to check/monitor what's happening? currently I used "free" to check memory usage and the mgmt page to see read operations on the drives. So I can see how many reads I had for first run of cache_dirs - and after some days of operation I see 10 or 20 times the amount, although there was no "real" read on the disks. Thanks, Guzzi Edit: I use S3 and wake the server via WOL - does cache_dirs a reread after S3? Does UnRaid flush the buffers after S3-wake? Hmmm, that would explain it - how can I verify this and is there a way to avoid it? Here is my "free" output: total used free shared buffers cached Mem: 4017452 2313480 1703972 0 248792 1391780 -/+ buffers/cache: 672908 3344544 Swap: 0 0 0 Do I read right, that mem extension from 2 to 4 GB wastn't necessary? Quote Link to comment
Guzzi Posted January 28, 2010 Share Posted January 28, 2010 Hi Joe L., I have another question: What is the usecase for the "-u" parameter? I understand it reads and caches the usershare - but I thought, this is already working, if the disk dirs are cached? If -u is enabled - would the excludes still work? (Because I have mapped dirs in the usershare, that I do not want to cache) Thanks, Guzzi Quote Link to comment
Joe L. Posted January 28, 2010 Author Share Posted January 28, 2010 Hi Joe L., I have another question: What is the usecase for the "-u" parameter? I understand it reads and caches the usershare - but I thought, this is already working, if the disk dirs are cached? If -u is enabled - would the excludes still work? (Because I have mapped dirs in the usershare, that I do not want to cache) Thanks, Guzzi It would be used for Experimentation only. User-shares are in memory already. No need to cache since access is quick regardless. It was made an option since user-shares are not "cached" specifically by cache_dirs by default. When we were first trying to identify what we needed to do to keep from spinning up a disk some thought the user-shares mith need to be cached too. They are not unless you use the -u option, but access is fast regardless. If you have a lot of ram you can experiment with using the -u option or not. General opinion is it will not make a difference (other than to cause the CPU to do more work). Joe L. Quote Link to comment
Guzzi Posted January 28, 2010 Share Posted January 28, 2010 Thanks for explanation - I currently have it installed and checked with htop - so CPU usage seems pretty high, but can't say if it was the same before without -u. Will just let it run some days and see "how it feels". rgds, Guzzi Quote Link to comment
Joe L. Posted January 28, 2010 Author Share Posted January 28, 2010 Thanks for explanation - I currently have it installed and checked with htop - so CPU usage seems pretty high, but can't say if it was the same before without -u. Will just let it run some days and see "how it feels". rgds, Guzzi A lot will depend on how many files & folders you have... basically, you are now doing a "find" on twice as many. It might even be worse, as the entries from the disk shares you want to stay in the cache, to keep them from spinning down, might be displaced by the extra ones read by scanning the user share's twin copies. Quote Link to comment
omkar Posted January 29, 2010 Share Posted January 29, 2010 Thank you Joe for creating this! I no longer have any pauses when using my media player to browse my unRaid server I put the script in a folder like you indicated on an earlier post in this thread and added the appropriate lines (to set the path and kick off the script) in my go file. After rebooting, the array automatically started a parity check. Is that normal? Once it completed, everything was working just fine. Quote Link to comment
Joe L. Posted January 29, 2010 Author Share Posted January 29, 2010 Thank you Joe for creating this! I no longer have any pauses when using my media player to browse my unRaid server I put the script in a folder like you indicated on an earlier post in this thread and added the appropriate lines (to set the path and kick off the script) in my go file. After rebooting, the array automatically started a parity check. Is that normal? Once it completed, everything was working just fine. The parity check is only normal if you did not stop the array cleanly before rebooting. You might want to post a syslog. If you think you stopped the array before rebooting, and the parity check was unexpected, then you might want to post a syslog to see if there was a different reason for the parity check. Joe L. Quote Link to comment
Guzzi Posted January 29, 2010 Share Posted January 29, 2010 Thanks for explanation - I currently have it installed and checked with htop - so CPU usage seems pretty high, but can't say if it was the same before without -u. Will just let it run some days and see "how it feels". rgds, Guzzi A lot will depend on how many files & folders you have... basically, you are now doing a "find" on twice as many. It might even be worse, as the entries from the disk shares you want to stay in the cache, to keep them from spinning down, might be displaced by the extra ones read by scanning the user share's twin copies. so this is the result after running the box overnight: All disks are spun down and memoryusage is as follows: root@TOWER:/# free total used free shared buffers cached Mem: 4017452 3866908 150544 0 247984 2939772 -/+ buffers/cache: 679152 3338300 Swap: 0 0 0 I ususally access the server for files via the [TOWER-Root] - path = /mnt/user entry in smb-extra.conf to get all usershares on one drive; this access (from windows SMB1) is fast and no disks spin up. Can you comment on how to read the output of free-command? I check the same without -u parameter to compare later... THanks, Guzzi Quote Link to comment
omkar Posted January 29, 2010 Share Posted January 29, 2010 Thank you Joe for creating this! I no longer have any pauses when using my media player to browse my unRaid server I put the script in a folder like you indicated on an earlier post in this thread and added the appropriate lines (to set the path and kick off the script) in my go file. After rebooting, the array automatically started a parity check. Is that normal? Once it completed, everything was working just fine. The parity check is only normal if you did not stop the array cleanly before rebooting. You might want to post a syslog. If you think you stopped the array before rebooting, and the parity check was unexpected, then you might want to post a syslog to see if there was a different reason for the parity check. Joe L. That explains it then. I did not stop the array via the web interface. I only pressed the Reset button on my MD-1510. Thanks! Quote Link to comment
Joe L. Posted January 29, 2010 Author Share Posted January 29, 2010 Thank you Joe for creating this! I no longer have any pauses when using my media player to browse my unRaid server I put the script in a folder like you indicated on an earlier post in this thread and added the appropriate lines (to set the path and kick off the script) in my go file. After rebooting, the array automatically started a parity check. Is that normal? Once it completed, everything was working just fine. The parity check is only normal if you did not stop the array cleanly before rebooting. You might want to post a syslog. If you think you stopped the array before rebooting, and the parity check was unexpected, then you might want to post a syslog to see if there was a different reason for the parity check. Joe L. That explains it then. I did not stop the array via the web interface. I only pressed the Reset button on my MD-1510. Thanks! Don't know how to say this any other way, but don't just hit the reset button unless absolutely necessary and you have no way to reboot otherwise. It might cause you to lose data... if the files being written to disks had not yet been flushed from the disk buffer cache to the physical disks they might be mangled/partially written, or lost/never written. Always stop the array first using the "Stop" button on the web-interface. Then you can reset/reboot/power down as needed. When you power back up, the array will know it shut down cleanly and not need to perform a full parity check. Joe L. Quote Link to comment
omkar Posted January 29, 2010 Share Posted January 29, 2010 That explains it then. I did not stop the array via the web interface. I only pressed the Reset button on my MD-1510. Thanks! Don't know how to say this any other way, but don't just hit the reset button unless absolutely necessary and you have no way to reboot otherwise. It might cause you to lose data... if the files being written to disks had not yet been flushed from the disk buffer cache to the physical disks they might be mangled/partially written, or lost/never written. Always stop the array first using the "Stop" button on the web-interface. Then you can reset/reboot/power down as needed. When you power back up, the array will know it shut down cleanly and not need to perform a full parity check. Joe L. Thanks for the explanation. Got it now! Quote Link to comment
tyrindor Posted February 14, 2010 Share Posted February 14, 2010 Can someone give me a standard command to cache everything on my unRAID server? I'm currently using: /boot/cache_dirs -d 10 -w However, disk 1 likes to spin up when browsing certain directories (like pictures, or program installers). Is this normal? Quote Link to comment
JDGJr Posted February 14, 2010 Share Posted February 14, 2010 Is it possible to use the 'exclude_dir' setting to exclude an entire drive? I have a non-array drive on the system that'd i'd rather not cache. Thanks, Joe, for another useful tool! Quote Link to comment
xamindar Posted March 16, 2010 Share Posted March 16, 2010 Just tried this. It seemed to run as it spun up all my disks and they were grinding away for a while. Then I ran "cache_dirs -q" to stop it and started it with "cache_dirs -F" so I can see output but now it just sits there on a blank line forever, no output, nothing. It seems to run fine without the -F option so I guess it's no big deal. Just thought it was supposed to spit back info as it ran. Quote Link to comment
Joe L. Posted March 16, 2010 Author Share Posted March 16, 2010 Just tried this. It seemed to run as it spun up all my disks and they were grinding away for a while. Then I ran "cache_dirs -q" to stop it and started it with "cache_dirs -F" so I can see output but now it just sits there on a blank line forever, no output, nothing. It seems to run fine without the -F option so I guess it's no big deal. Just thought it was supposed to spit back info as it ran. You know... you are the first to describe this behavior since I posted the last version of cache_dirs. Guess what. In adding the feature to suspend the script during the "mover" scripts operation, I broke cache_dirs. It is supposed to spit_back info as it ran. It is broken if you do not have a cache drive defined, as you discovered, a major part of the loop is skipped and no detail is printed. Since last October, over 200 people have downloaded it, and you are the first to notice the problem. (indirectly, at least) Thanks so much for the report of the problem you were having. I'm just attached a newer version of cache_dirs to the first post in its thread. I'll go hang my head in shame for my logic oversight.... for a few minutes anyway. Version 1.6.5 is now attached to the first post in this thread. It should work as you expected. Joe L. Quote Link to comment
sosdk Posted March 16, 2010 Share Posted March 16, 2010 I really like this feature a lot - only problem I've found is that my array has grow to big to have everything in the 1GB of RAM my system holds.... Guess I need to buy 4GB instead. Quote Link to comment
Joe L. Posted March 16, 2010 Author Share Posted March 16, 2010 I really like this feature a lot - only problem I've found is that my array has grow to big to have everything in the 1GB of RAM my system holds.... Guess I need to buy 4GB instead. That is why you can limit the number of levels cached, and why you can exclude folders where the spin-up delay is not an issue. I have only 512 Meg of ram in my server. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.