June 4, 200917 yr I've been doing some more reading/investigating... I was reading this site: http://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/ and found this reply at the bottom of the article: What about deadlocks when invoking a shell script which itself forks a child process? From the real shell, it returns right away as expected; but, from Subprocess, it deadlocks until the child process dies. Cheers, Matt
June 4, 200917 yr I've been doing some more reading/investigating... I was reading this site: http://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/ and found this reply at the bottom of the article: What about deadlocks when invoking a shell script which itself forks a child process? From the real shell, it returns right away as expected; but, from Subprocess, it deadlocks until the child process dies. Cheers, Matt Very interesting... as it seems that the child process is indeed waiting for something for it to continue. Joe L.
June 14, 200917 yr I've tried to read through this entire thread, but I can't seem to find an answer to my problem. Here's the output I'm getting: root@Tower:/mnt# /boot/cache_dirs -v -F Executing find /mnt/disk4/Backup Executing find /mnt/disk1/Movies Executing find /mnt/disk2/Movies Executing find /mnt/disk3/Movies Executing find /mnt/disk4/Movies Executing find /mnt/disk5/Movies Executing find /mnt/disk6/Movies Executing find /mnt/disk7/Movies Executing find /mnt/disk8/Movies Executing find /mnt/disk9/Movies Executing find /mnt/disk1/TV Executing find /mnt/disk2/TV Executing find /mnt/disk4/TV Executing find /mnt/disk5/TV Executing find /mnt/disk6/TV Executed find in 0.120009 seconds, weighted avg=0.120009 seconds, now sleeping 5 seconds Executing find /mnt/disk4/Backup Executing find /mnt/disk1/Movies Executing find /mnt/disk2/Movies Executing find /mnt/disk3/Movies Executing find /mnt/disk4/Movies Executing find /mnt/disk5/Movies Executing find /mnt/disk6/Movies Executing find /mnt/disk7/Movies Executing find /mnt/disk8/Movies Executing find /mnt/disk9/Movies Executing find /mnt/disk1/TV Executing find /mnt/disk2/TV Executing find /mnt/disk4/TV Executing find /mnt/disk5/TV Executing find /mnt/disk6/TV Executed find in 0.120884 seconds, weighted avg=0.120592 seconds, now sleeping 4 seconds In other words, it's not caching anything. If I open a drive that's spun down, it spins up. What am I doing wrong? Thanks!
June 14, 200917 yr I've tried to read through this entire thread, but I can't seem to find an answer to my problem. Here's the output I'm getting: root@Tower:/mnt# /boot/cache_dirs -v -F Executing find /mnt/disk4/Backup Executing find /mnt/disk1/Movies Executing find /mnt/disk2/Movies Executing find /mnt/disk3/Movies Executing find /mnt/disk4/Movies Executing find /mnt/disk5/Movies Executing find /mnt/disk6/Movies Executing find /mnt/disk7/Movies Executing find /mnt/disk8/Movies Executing find /mnt/disk9/Movies Executing find /mnt/disk1/TV Executing find /mnt/disk2/TV Executing find /mnt/disk4/TV Executing find /mnt/disk5/TV Executing find /mnt/disk6/TV Executed find in 0.120009 seconds, weighted avg=0.120009 seconds, now sleeping 5 seconds Executing find /mnt/disk4/Backup Executing find /mnt/disk1/Movies Executing find /mnt/disk2/Movies Executing find /mnt/disk3/Movies Executing find /mnt/disk4/Movies Executing find /mnt/disk5/Movies Executing find /mnt/disk6/Movies Executing find /mnt/disk7/Movies Executing find /mnt/disk8/Movies Executing find /mnt/disk9/Movies Executing find /mnt/disk1/TV Executing find /mnt/disk2/TV Executing find /mnt/disk4/TV Executing find /mnt/disk5/TV Executing find /mnt/disk6/TV Executed find in 0.120884 seconds, weighted avg=0.120592 seconds, now sleeping 4 seconds In other words, it's not caching anything. If I open a drive that's spun down, it spins up. What am I doing wrong? Thanks! First, make sure you are using the newest version 1.5 of the cache_dirs script. Then, what makes you think nothing is being cached? The output of all the "find" commands is sent to /dev/null. Nothing will show on the screen other than the statement of what it is running and on which disk. .12 seconds seems a reasonable time. If you wish to see what it is doing underneath you can install inotifywatch. What we've discovered (back around this post) was the directories were being cached, but the "find" command was optimizing itself by not reading in every leaf inode. I then added the ability to add in options to the executed commands so we could force it to not to optimize... You just need to invoke the cache_dirs command as follows and it will probably work as you desired (I'll probably make the -noleaf option the default in the next version of cache_dirs, but for now, you can just add it using the "-a" option.) cache_dirs -a "-noleaf" or to run in the foreground cache_dirs -a "-noleaf" -v -F Joe L.
June 14, 200917 yr Then, what makes you think nothing is being cached? The fact that the drive spins up when I open a folder! I tried the noleaf option and it now works, the drives no longer spin up when folders are opened. Thanks for that! This is a very useful utility. Are you saying the noleaf option is required for my use? FYI, my array is 7TB, 51,657 files in 3,318 folders, mainly Blu-ray rips.
June 14, 200917 yr Are you saying the noleaf option is required for my use? I'm saying it is required for everybody's use, not just you. It will be the "default" in the next version... at that point you won't need to explicitly use it on the command line, but until then... it is required.
June 15, 200917 yr -noleaf. ahh!!!! That's the reason my own version of caching with find was not working as well. Now I understand. Thanks for the hint!
June 16, 200917 yr Joe any chance of a new version with the new defaults and changes?. (trying not to sound pushy Yes... there is chance... Here is version 1.6 of cache_dirs. I'll leave the old version attached to the older post for a short while, then I think it is time for a thread in the Customizations Forum for its long-term home. There are quite a few new options and improvements in Version 1.6 First, I fixed the bug described earlier where if "-q" was used, and cache_dirs not currently running, it started running in error. I added the missing "exit" Then I fixed the bug where "find" optimized its scan of directories and did not scan some nodes. I now have -noleaf the default behavior for the "find" command (you can always use -a "" to disable if you want to experiment without it). I added a "-z" option to write the initial log entry to the syslog as a single concise line if desired instead of multiple lines. Next, I changed vfs_cache_pressure setting to be 1 instead of 0 by default, and added a "-p cache_pressure" option to allow you to set the old value if you like, or to experiment with other values. The reason the default changed is that with cache_pressure=0, and a small amount of RAM, and a large number of directory and file entries, you could run out of RAM trying to put all the directory information in RAM. Once this happens, processes are killed to free memory instead of available cached RAM being freed. (tough trying to regain control when your login shells are being terminated by the kernel, and emhttp terminated) In response to at least one user having a small PANIC attack when they were confronted with many "Unformatted" disks after they attempted to stop their unRAID array, and the cache_dirs "find" was scanning their disks at the moment, and a disk could not be un-mounted, but all the others were... I added logic to force all disks "busy" if cache_dirs is running. I force disks to be "busy" by creating a process using each of the data disks as their current-working-directory. These use no space on the disks, but prevent unRAID from un-mounting them. As a result, pressing "Stop" on the management interface will not stop the array, in fact, it will return doing nothing to stop the server... and no disk will be un-mounted, therefore no disk will show up (inaccurately) as un-formatted on the management web-page. The unRAID management console does add a syslog entry stating it could not stop the array. I look for that each time through the loop... if it is there in the last 100 lines of the syslog, I suspend the cache_dirs loop and terminate those extra processes, and therefore it permits the "Stop" button to work normally when you press it a second time. Each time through the cache_dirs loop, if an attempt to stop the array is detected in the last 100 lines of the syslog, those same processes I created are all terminated and the cache_dirs loop paused for 120 seconds. Pressing "Stop" on the management console a second time will then succeed and no disk will fail to be un-mounted as a result of a cache_dirs "find" scanning the directory tree. Yes, you can still catch the loop if you are really lucky, but odds are much better now you wont see "Unformatted" If you stop the array from the management console and then press "Start" to re-start it, the cache_dirs will resume after a minute or two. You do not need to re-start it by hand. (The process of stopping and re-starting the array writes more than 100 lines to the syslog, so the line showing it could not be stopped earlier will not be detected, and the loop will continue its normal processing. When running this version you will see one or two new processes per disk in your array in the process list. These are only keeping the disk from being un-mounted. Disks can still go to sleep if unused for their sleep duration settings. I added new -B option to revert to the old behavior and not force disks busy if by chance this new feature causes problems for some users. If you like the way it was, where you might see "Unformatted" disks when stopping the array if your timing was lucky, then add "-B" to the command line arguments. Based on a request, I now allow min seconds to be equal to max seconds in loop delay range parameter. I added run-time-logging, log name = /var/log/cache_dirs.log If the file exists, it will be logged to. If not, logging will stop. To make it easier, once cache_dirs is running, you can use (from another command prompt) cache_dirs -l on and cache_dirs -l off to enable or disable the log. (Those two commands just create or remove the log file, but make it easier on some to perform that task) Give it a try and let me know if you see anything I should change. Edit:Newest version is attached here: http://lime-technology.com/forum/index.php?topic=4500.0 It locks the "cache" drive from showing as un-mounted when an attempt to stop the array is made. Joe L.
June 16, 200917 yr Would the command to start it from the GO script still be the same, like: /boot/custom/bin/cache_dirs no need for any additional switches for the defaults and provided it's located there of course
June 16, 200917 yr Would the command to start it from the GO script still be the same, like: /boot/custom/bin/cache_dirs no need for any additional switches for the defaults and provided it's located there of course yes... from the linux command line that is all that is needed. However, I would always use the -w option if from a "go" script to ensure it waits till the array is on-line.
June 17, 200917 yr Well I don't know why, but this doesn't work for me. I've installed it and rebooted, then allowed it to run before spinning down all the drives. Next I browse folders from my windows box, and multiple drives spin up, with the obligatory delay in browsing as each one does. Could it be that explorer is requesting information that isn't cached?
June 17, 200917 yr Well I don't know why, but this doesn't work for me. I've installed it and rebooted, then allowed it to run before spinning down all the drives. Next I browse folders from my windows box, and multiple drives spin up, with the obligatory delay in browsing as each one does. Could it be that explorer is requesting information that isn't cached? I have the same issue! I guess you are using the -l on option also, right? Well, mine I got it working by issuing two command line: /boot/cache_dirs then followed by /boot/cache_dirs -l on By doing this no more multple drives spin up. Joe L. will probably look at this issue later on.
June 17, 200917 yr Well I don't know why, but this doesn't work for me. I've installed it and rebooted, then allowed it to run before spinning down all the drives. Next I browse folders from my windows box, and multiple drives spin up, with the obligatory delay in browsing as each one does. Could it be that explorer is requesting information that isn't cached? I have the same issue! I guess you are using the -l on option also, right? Well, mine I got it working by issuing two command line: /boot/cache_dirs then followed by /boot/cache_dirs -l on By doing this no more multple drives spin up. Joe L. will probably look at this issue later on. Enabling/disabling the logging is not supposed to have anything to do with the scan of the directories. You must use the -l on/off option as a second command, after initially starting the cache_dirs program with the normal options you desire, exactly as SuperPerry described. (I think I need to add logic to make that more clear, and provide an error message if you try.) You do need to be aware that if you turn on logging, and leave it permanently enabled, then given enough time you could eventually crash your server when the log uses up all the available RAM. Now... it all depends on how fast you are looping, and how much RAM you have available. On my server I am logging about 33000 bytes of content per hour. That would use 250 Meg of ram in just over 300 days if I did my math right... I have about 250 Meg free, so I'm not really worried, as I reboot more than once a year or so... If you have 4 Gig of RAM, you'll probably not have any worry, at least not for a few months, unless you loop and log at a faster rate than me. Joe L.
June 17, 200917 yr Well I don't know why, but this doesn't work for me. I've installed it and rebooted, then allowed it to run before spinning down all the drives. Exactly what command line did you use when it was invoked? Next I browse folders from my windows box, and multiple drives spin up, with the obligatory delay in browsing as each one does. Could it be that explorer is requesting information that isn't cached? It is always possible. If you have the time and interest, you might download and install the inotifywatch package and see exactly what is being accessed.
June 17, 200917 yr You do need to be aware that if you turn on logging, and leave it permanently enabled, then given enough time you could eventually crash your server when the log uses up all the available RAM. Are you logging through syslog with the logger program? From what I remember the logrotate bug was fixed so in theory you should not have issues with it.
June 17, 200917 yr You do need to be aware that if you turn on logging, and leave it permanently enabled, then given enough time you could eventually crash your server when the log uses up all the available RAM. Are you logging through syslog with the logger program? From what I remember the logrotate bug was fixed so in theory you should not have issues with it. No, I am not logging to the syslog... but that might be a way to avoid any issue at all. cache_dirs is just writing to a file of its own and not using the logger program for the continuous run-time-statistics enabled by "cache_dirs -l on".. It will take a very long time to run out of RAM (months or possibly years if you have a lot of RAM), but it can happen. To clear out the log all you need do is stop logging and re-start it. No need to reboot the server.
June 18, 200917 yr Exactly what command line did you use when it was invoked? /boot/custom/bin/cache_dirs -w It is always possible. If you have the time and interest, you might download and install the inotifywatch package and see exactly what is being accessed. Will investigate that, thanks.
June 21, 200917 yr Joe L., thanks for another great script. I had been planning on implementing this for a while and finally decided to do it while adding another drive. I've had some initial issues, and I'd like to know if I'm doing something wrong. Granted, at this point, this is all from memory, but I think this is pretty accurate. 1. Now that my unRAID runs months on end without reboots, I wanted to make sure I could get cache_dirs working prior to adding new drive. I added it to my /boot/custom/bin and invoked it from there. I was my idle disks spin up which I think is normal given that it probably needed to cache all my directories at this point. 2. I tried to stop the array from the management page. Although I pressed the button to stop the array a number of times (i.e., more than ten) over a good period of time (i.e., probably across a total of 5 minutes), the array never stopped. I'm grateful for no "Unformatteds" during this time, but I'm wondering if there's something wrong, or if I should have tried a little longer. Note, I don't think I had explicitly stopped VMware Server at this point, but it's open files are on my cache drive, so I don't know if this affected the stop attempts. 3. I did a /boot/custom/bin/cache_dirs -q to stop the program followed by a "Stop" on the array management page, and I got a number of "Unformatteds" across the array. 4. I forced a reboot from the commandline (via telnet), and think the array restarted without a parity check. 5. unRAID came back up and cache_dirs started automatically, but I still need to stop the array in order to add my new disk. This time, I stopped VMware Server, but still, I can't stop the array by pressing "Stop" multiple times. After stopping cache_dirs via a user script via unMENU, pressing stop now says the cache is unformatted but the array is still started. 6. I don't think the next forced reboot started a parity check, but after playing around with it, I noticed that looking at open files via unMENU, even after killing cache_dirs, there seemed to be a find command running that would restart even if I killed that specific PID. At some point, a parity check initiated. At this point, I simple disabled the automatic start-on-boot for cache_dirs, so I can get my new drive installed and cleared. After it clears, I'll run a parity check. After that, I can do some real testing if needed on the cache_dirs issues I've been having. However, I'm betting that I've just done something stupid that someone can easily point out. thanks, mm
June 21, 200917 yr Quote from: musicmann on June 20, 2009, 11:34:33 PM Joe L., thanks for another great script. I had been planning on implementing this for a while and finally decided to do it while adding another drive. I've had some initial issues, and I'd like to know if I'm doing something wrong. Granted, at this point, this is all from memory, but I think this is pretty accurate. 1. Now that my unRAID runs months on end without reboots, I wanted to make sure I could get cache_dirs working prior to adding new drive. I added it to my /boot/custom/bin and invoked it from there. I was my idle disks spin up which I think is normal given that it probably needed to cache all my directories at this point. 2. I tried to stop the array from the management page. Although I pressed the button to stop the array a number of times (i.e., more than ten) over a good period of time (i.e., probably across a total of 5 minutes), the array never stopped. I'm grateful for no "Unformatteds" during this time, but I'm wondering if there's something wrong, or if I should have tried a little longer. Note, I don't think I had explicitly stopped VMware Server at this point, but it's open files are on my cache drive, so I don't know if this affected the stop attempts. The script only looks at the last 100 lines of the syslog to try to detect the "Stop" attempt. If it sees specific text it will enter a 120 second sleep after first killing the processes it created to force the drives "busy" It will only look for that text between full scans of your directories. So, if you are logging more to the syslog, it might miss the line it was looking for because it was more than 100 lines back... Or, if your version of unRAID does not output the specific text it is looking for ("devices still in use"), it will never find it, so it will not pause for you to shut down the array. If your syslog is HUGE it will take a while just to get the tail of the last 100 lines... so there is that possible delay. However... The processes keeping the drives busy all look for the "lock file" created by this process. Removing it (/var/lock/cache_dirs.LCK), either manually, or with cache_dirs -q should have had them all exit, but only after they get to it in their respective loops. The child processes have only a 2 second sleep, so they will exit quickly. The main cache_dirs could be taking a very long time just to scan your directories and files. (How long is your normal scan average cycle time?) Now, it is still possible to have a busy drive. If you have unMENU you might be able to identify the process holding a drive busy. 3. I did a /boot/custom/bin/cache_dirs -q to stop the program followed by a "Stop" on the array management page, and I got a number of "Unformatteds" across the array. Yup, something else was holding a drive from being un-mounted. From the command line I think you can kill it by typing: fuser -km /mnt/disk3 (assuming it is drive 3 that was the one that was NOT reported as un-formatted... remember, the ones still showing OK on the management interface are those that were unable to be un-mounted. So, if all the drives but disk3 show un-formatted, you want to use "fuser -km /mnt/disk3" If disk2 was the only one not showing un-formatted, then use "fuser -km /mnt/disk2" etc... Tom has said he will be adding that same "fuser" command to the unRAID emhttp process in a future version... so at that time the extra processes holding a disk busy will be killed automatically. Until then, we'll have to deal with it. 4. I forced a reboot from the commandline (via telnet), and think the array restarted without a parity check. 5. unRAID came back up and cache_dirs started automatically, but I still need to stop the array in order to add my new disk. This time, I stopped VMware Server, but still, I can't stop the array by pressing "Stop" multiple times. After stopping cache_dirs via a user script via unMENU, pressing stop now says the cache is unformatted but the array is still started. Sounds to me like the cache drive was not mounted... Probably some interaction from vmware. 6. I don't think the next forced reboot started a parity check, but after playing around with it, I noticed that looking at open files via unMENU, even after killing cache_dirs, there seemed to be a find command running that would restart even if I killed that specific PID. At some point, a parity check initiated. Most of the "find" commands are in a loop in these scripts. You need to kill the PARENT process of the find command, otherwise, it will issue another find command. At this point, I simple disabled the automatic start-on-boot for cache_dirs, so I can get my new drive installed and cleared. After it clears, I'll run a parity check. After that, I can do some real testing if needed on the cache_dirs issues I've been having. However, I'm betting that I've just done something stupid that someone can easily point out. thanks, mm You have to realize, we have NO good way at this point in time to know when the array is being stopped and to terminate processes prior to them being un-mounted. This script is a complete "hack" at trying to work within those constraints. If you have unMENU you can usually see the process IDs holding a disk busy and take action. Learn to live with "unformatted" when you are running non-standard scripts and attempt to stop the array. It is not a problem... It is a warning (to those who know the disks are not really un-formatted) that one of your other disks could not be un-mounted. You can always do a ps -ef to see all the running processes... and take action as necessary to terminate your extra processes and scripts. That combined with the output of the fuser command is the best we have at this point. Sorry... but if you have ideas on how to make this better... I'd love the input. Joe L.
June 22, 200917 yr btw, just wanted to say thank you for this awesome user script; really makes a huge difference in browsing my shares. look forward to any future releases also are there any other tweaks I can do to make use of the 8gb of ram in my unraid server (it was cheap so i figured why not) ; I'd love to make use of as much as possible to gain any other performance from the system.
June 22, 200917 yr 5. unRAID came back up and cache_dirs started automatically, but I still need to stop the array in order to add my new disk. This time, I stopped VMware Server, but still, I can't stop the array by pressing "Stop" multiple times. After stopping cache_dirs via a user script via unMENU, pressing stop now says the cache is unformatted but the array is still started. I re-read this, and it hit me. When you pressed "Stop" unRAID was able to un-mount the cache drive... because I did not have a process keeping it "busy" like I did with the other data disks. Therefore, it showed as "unformatted" (can you tell I don't use a cache drive?) You can fix it by cd'ing to the directory with the cache_dirs file and running sed -i -e "s/\(for i in \/mnt\/disk\* $\)/\\1 \/mnt\/cache/" cache_dirs The "stream edit" command will fix the two lines that currently say for i in /mnt/disk* The stream edit changes them to say for i in /mnt/disk* /mnt/cache The lines that are changed are at line 385 and line 511 of the 1.6 version of cache dirs. It will not solve all your problems, but it might help a tiny bit by keeping the cache drive from being un-mounted when all the others are "busy" Joe L.
June 22, 200917 yr Thanks, Joe. This afternoon, I was able to stop cache_dirs and eventually get all the processes stopped for a clean array stop. I'll update the script. I'm new to this, so I think I might not have been giving my setup time to go through all the files. Is there a way to check how long it takes to scan all the directories?
June 22, 200917 yr Thanks, Joe. This afternoon, I was able to stop cache_dirs and eventually get all the processes stopped for a clean array stop. I'll update the script. I'm new to this, so I think I might not have been giving my setup time to go through all the files. Is there a way to check how long it takes to scan all the directories? Easiest way is to use the -F option to run it in the foreground from the command line. It will then print statistics to the screen. You will need to stop it first cache_dirs -q followed by re-starting it adding the -F option to any you currently use. cache_dirs -F -e dir ..... JOe L.
Archived
This topic is now archived and is closed to further replies.