April 17, 200917 yr The way I use unraid (and probably most others) is that I read content much more than I add (write) content. I have noticed that (understandably), multiple drives must spin up to retrieve a directory listing when the contents of that directory are split across multiple drives. I remember seeing someone issue an ls -R against the user shares root presumably to cache the directory structure in memory. Is anyone using this and does it help, or is there another way to accomplish this. I'm trying to avoid spinning up drives just to view directory listings.
April 17, 200917 yr Many people use it. It helps by essentially stoping you ever spinning up drives unless you are reading or writing actual data. I tried running without it on 4.5beta3 during my testing and it made unRAID feel dog slow. Woulndnt be without it now. Here is my very simple script #!/bin/sh while [ 1 ] do if [ -d /mnt/user/ ]; then #echo "Exists"; ls -R /mnt/user >/dev/null 2>&1 # Modify sleep time (in seconds) as needed below sleep 10 fi done
April 17, 200917 yr Many people use it. It helps by essentially stoping you ever spinning up drives unless you are reading or writing actual data. I tried running without it on 4.5beta3 during my testing and it made unRAID feel dog slow. Woulndnt be without it now. Here is my very simple script #!/bin/sh while [ 1 ] do if [ -d /mnt/user/ ]; then #echo "Exists"; ls -R /mnt/user >/dev/null 2>&1 # Modify sleep time (in seconds) as needed below sleep 10 fi done NAS, how does this work on shares that have lots and lots of files? From my understanding this will fill up the buffers and kind of backfire without a huge amount of memory and 64-bit support. I have not tried it yet as I power down my system on a regular basis. I am hoping a better more permanent solution can be implemented at some time. I come from the Microsoft side of things mainly and their Indexing is a perfect example of persistent caching. Does Linux have something similar?
April 17, 200917 yr Its not perfect but dont worry about all the 64bit and buffers talk... thats in search of a perfect solution which this will never be. it will make a HUGE difference though. Imlaunch it in go script with nohup. Webotech has a much better version of this but its too complicated for me at this point
April 17, 200917 yr Author Its not perfect but dont worry about all the 64bit and buffers talk... thats in search of a perfect solution which this will never be. it will make a HUGE difference though. Imlaunch it in go script with nohup. Webotech has a much better version of this but its too complicated for me at this point I went ahead and added the script shown on the post above and it seems to work fine, can you point me the the one by Webotech so I can have a look? I did some forum searches but couldn't find it.
April 18, 200917 yr can you point me the the one by Webotech so I can have a look? I did some forum searches but couldn't find it. It's in this thread, Read the whole thing, there is a segment that is required at the top of the script that is not in the whole thread. One of these days I'll post the whole thing on googlecode. I was in the process of testing if ls -lR was better the a find -ls. http://lime-technology.com/forum/index.php?topic=3137.msg26264#msg26264
April 18, 200917 yr Author can you point me the the one by Webotech so I can have a look? I did some forum searches but couldn't find it. It's in this thread, Read the whole thing, there is a segment that is required at the top of the script that is not in the whole thread. One of these days I'll post the whole thing on googlecode. I was in the process of testing if ls -lR was better the a find -ls. http://lime-technology.com/forum/index.php?topic=3137.msg26264#msg26264 Before I read that thread the thought had occurred to me whether ls -R would cache all the file info such as size/date since the command does not display that info. I'm guessing that the underlying library/system call being used by ls must read all those things and cause them to be cached even though the ls -R command does not print them, I tested this by putting the ls -R script from this thread in my go script and them spinning down the drives and then doing an ls -alR through telnet, this pretty much prints everything about a file that an SMB client would see when getting a listing and no drives spun up, so I don't think that the find version should be necessary.
April 20, 200917 yr Here is my very simple script #!/bin/sh while [ 1 ] do if [ -d /mnt/user/ ]; then #echo "Exists"; ls -R /mnt/user >/dev/null 2>&1 # Modify sleep time (in seconds) as needed below sleep 10 fi done NAS, I'd like to suggest moving the sleep 10 out of the if, below the fi. Right now, while it is waiting for the array to start, it will be looping very rapidly, potentially slowing the startup down. There's no need to test for /mnt/user more than once every 10 seconds.
April 20, 200917 yr Your quite right Ive no idea how it snuck back in there. Sleep moved. The test for /mnt/user does a passable job at seeing if the array is down. If someone knows of at slicker way of querying status directly let me know. #!/bin/sh while [ 1 ] do if [ -d /mnt/user/ ]; then #echo "Exists"; ls -R /mnt/user >/dev/null 2>&1 # Modify sleep time (in seconds) as needed below fi sleep 10 done
April 20, 200917 yr Before I read that thread the thought had occurred to me whether ls -R would cache all the file info such as size/date since the command does not display that info. I'm guessing that the underlying library/system call being used by ls must read all those things and cause them to be cached even though the ls -R command does not print them, I tested this by putting the ls -R script from this thread in my go script and them spinning down the drives and then doing an ls -alR through telnet, this pretty much prints everything about a file that an SMB client would see when getting a listing and no drives spun up, so I don't think that the find version should be necessary. I actually wrote a C program to do "just the" stat() call to save on having to print something then ignore it in /dev/null. This proved to work only at a partial level because ftw(file tree walk) does not handle files over 4GB correctly. Actually it calls stat, but needs to call stat64. This is a library call which is not easy to replace. Ugh. So I got side tracked with my C daemon. I'm going to code my own recursive file tree walk so I can control the whole thing one of these days. In any case the ls -R and find both do a stat call to capture file name information. This is what gets saved in the cache. I'm used to coding programs to make the most out of resources without waste. This is my nature. I had a 30 million dollar business delivering a huge amounts of data on a Pentium 90 for years because of my miserly programming. No matter what option chosen, find, ls -R, recursive file tree walk.. The kernel still needs the vm.vfs_cache_pressure=0
April 20, 200917 yr The current behavior of most of the variations of the 'ls -R' script is to cache the User Share directory entries (ls -R /mnt/user). That seems wasteful and duplicative to me, as it first caches the directory entries of the User share file system, which are already in memory (!) because it is essentially a virtual file system, and then caches the corresponding directory entries from the physical disks. Why cache something that is already in memory? Would it not be better to only cache the entries stored on 'physical' storage media? We really don't care if the ones from the memory file system 'expire' from the cache. Agreed, it makes a much simpler script to just load /mnt/user, and it makes the script much easier to explain. But it should not be hard to create a fairly generic script that generates a target list of cached directories, from the folders within /mnt/user, and then loops through the available unRAID drives looking for them. Something like this pseudo-bash (!): #!/bin/sh while [ 1 ] do if [ -d /mnt/user/ ]; then # is array started (?) if ( -z $sharelist ); then # is sharelist initialized yet sharelist = shares from 'ls /mnt/user' # needs loop to cut and add folders to sharelist fi for disk in /mnt/disk* if ( $disk == '/mnt/disk0' ); then # skip parity drive continue fi do share = next from $sharelist # loop to extract each share name if ( -d "$disk/$share" ); then # does share exist on this drive? ls -R $disk/$share >/dev/null 2>&1 # cache its directory entries fi done fi sleep 10 # don't process more than once each 10 seconds (or as set by user) done
April 20, 200917 yr Here is something to try... [pre] while true do sleep 10 [ ! -d /mnt/user ] && continue ls -1 /mnt/user | while read dir do dir=`basename "$dir"` # Exclude the "data" directory from cache [ "$dir" = "data" ] && continue for i in /mnt/disk* do [ "$i" = "disk0" ] && continue [ ! -d "$i"/"$dir" ] && continue ls -R "$i"/"$dir" >/dev/null 2>&1 done done done [/pre] It will not cache the /mnt/user structure, but will do it for the corresponding disk structure. Note I have a "data" directory on some of my disks I did not want to put into cache, so I exclude it. Warning: I'm adding this note everywhere I made a post regarding this technique of repeatedly accessing directory entries to keep them in the cache. When accessing the disks, using "ls" or "find" or anything else, they are "busy" and unable to be un-mounted. If even one disk is unable to be un-mounted you will not be able to "stop" the array without being caught by a surprise... If what I describe here happens to you, do not panic. What will happen when you press "Stop" is those disks that can be un-mounted will be... the "busy" disk will not be able to be un-mounted. (normally all disks are un-mounted) When unRAID is unable to un-muont all the drives then management interface will not stop the array. Instead, it will show all the disks that could be un-mounted (incorrectly) as Unformatted. The "busy" disk will show as normal. This is frightening when you first encounter it as the management interface even presents a "format" button (whatever you do, do NOT press it). All you need do is press "Stop" once more (or as needed) to catch the added script between its scans when it is sleeping and not accessing a disk. As I said, whatever you do, do not press the "Format" button, as it would try to re-format all the disks it incorrectly thinks are unformatted. (they are not unformatted, just un-mounted, so their file-systems not visible. This is a bug in the display logic of the management interface.) As some point in the future the unRAID interface will be enhanced to terminate these add-on scripts prior to stopping the array, but as of 4.5-beta6 it does not, and you will see the unformatted display in error if you catch a script in the middle of a scan. Again, all you need do is press "Stop" once more (or as needed) to catch the added script between its scans when it is sleeping and not accessing a disk. Joe L. Joe L.
April 20, 200917 yr Wow! Superb! I can take a little credit for the idea, but now this amateur is stepping back, the pro's are here! I've tried using the various Bash manuals online, but they are terrible, very haphazard and disorganized. I have to agree with someone else who said they had to constantly revert to trial and error when trying to write a bash script! I am quite sure I would have learned more, especially would have learned more practical and useful stuff, if I had spent the time studying many of the scripts offered on the forums, especially Joe's and WeeboTech's. This script is an excellent example, highly educational for me. Now we just need to 'polish' it a bit for general use, especially for new users, and add a few instructions as to how to install it. Even if we comment it out, the skipping of the data folder will serve as a good example for the similar needs of other users.
April 21, 200917 yr Author Before I read that thread the thought had occurred to me whether ls -R would cache all the file info such as size/date since the command does not display that info. I'm guessing that the underlying library/system call being used by ls must read all those things and cause them to be cached even though the ls -R command does not print them, I tested this by putting the ls -R script from this thread in my go script and them spinning down the drives and then doing an ls -alR through telnet, this pretty much prints everything about a file that an SMB client would see when getting a listing and no drives spun up, so I don't think that the find version should be necessary. I actually wrote a C program to do "just the" stat() call to save on having to print something then ignore it in /dev/null. This proved to work only at a partial level because ftw(file tree walk) does not handle files over 4GB correctly. Actually it calls stat, but needs to call stat64. This is a library call which is not easy to replace. Ugh. So I got side tracked with my C daemon. I'm going to code my own recursive file tree walk so I can control the whole thing one of these days. In any case the ls -R and find both do a stat call to capture file name information. This is what gets saved in the cache. I'm used to coding programs to make the most out of resources without waste. This is my nature. I had a 30 million dollar business delivering a huge amounts of data on a Pentium 90 for years because of my miserly programming. No matter what option chosen, find, ls -R, recursive file tree walk.. The kernel still needs the vm.vfs_cache_pressure=0 The reason I say that the find version may not be necessary is because it seems to be the slowest of the options. I tried playing with the args to ls to see how little output I could get it to write hoping to speed things up. I tried, for example to turn off the color coding, this did help a bit IF I was eriting to the console, but redirecting to null made any shortening of the output pointless. It seems that the time ls takes printing in in the printf or fprintf or sprintf or whatever it uses or again the system calls that are in turn used by C. As for as your philosophy abut optimal coding, I agree. I think all programmers should be forced to only test apps on hardware that meets the minimum requirements for a project because of the fact that developers USUALLY have access to the opposite they tend not to see the waste very clearly.
April 21, 200917 yr Here is something to try... [pre] while true do sleep 10 [ ! -d /mnt/user ] && continue ls -1 /mnt/user | while read dir do dir=`basename "$dir"` # Exclude the "data" directory from cache [ "$dir" = "data" ] && continue for i in /mnt/disk* do [ "$i" = "disk0" ] && continue [ ! -d "$i"/"$dir" ] && continue ls -R "$i"/"$dir" >/dev/null 2>&1 done done done [/pre] It will not cache the /mnt/user structure, but will do it for the corresponding disk structure. Note I have a "data" directory on some of my disks I did not want to put into cache, so I exclude it. Joe L. Brilliant stuff. I have been thinking about this for some time now. How about an adaptive timer rather than a flat 10 seconds. Basically the logic would be grep dentry from /proc/slabinfo. If dentry <<< xxx set timer to 0 (this means the first loop is instant) Do the ls loop then read dentry into a starting variable. Start loop again If dentry xx% < starting variable shrink timer If dentry xx% NOT < starting variable shrink timer set timer back to 10 seconds. Thats the basic premise that should help catch conditions where dentry is being flushed quicker than a flat timer
April 24, 200917 yr I attempted to track the dentry cache numbers and figure out when it was being purged, but I had no real success. The numbers reported in /proc/slabinfo were not really useful. Instead, I added logic to time how long the series of ls -R commands take, and adjust the loop sleep time accordingly. If the ls -R starts to take longer, the sleep interval is decreased. if it takes less time, it is increased. I set the min and max at 0 and 10 seconds. On my server, the loop eventually seems to sleep about 6 or 7 seconds. If I do some heavy file-system activity (copying an 8Gig file from a disk to /dev/null) the loop interval drops down to between 2 and 4 seconds. This script is a lot more complicated in that it creates a "lock-file" to ensure you do not run more than one at a time. It also puts itself in the background and disassociates itself from the terminal, so you can just invoke it without "nohup" or "&" To stop it, just remove the lock-file it creates, and it will notice it is removed and terminate itself. rm /var/lock/cache_dirs The only changes you might need is if you have directories you do not need cached. In my case, I have a "data" directory I use for file-system backups of other PCs and other misc files. I do not cache it. I put a second "your_dir_to_exclude" line in place... you can edit it, or add additional lines to exclude other directories. Have fun. Joe L. lockfile="/var/lock/cache_dirs" if [ -f "${lockfile}" ]; then # The file exists so read the PID # to see if it is still running lock_pid=`head -n 1 "${lockfile}"` pid_running=`ps -p "${lock_pid}" | grep ${lock_pid}` if [ -z "${pid_running}" ]; then # The process is not running # Echo current PID into lock file echo $$ > "${lockfile}" else echo "`basename $0` is already running [${lock_pid}]" exit 0 fi else echo $$ > "${lockfile}" fi num_seconds=10 prior_elapsed_time=0 sysctl vm.vfs_cache_pressure=0 >/dev/null 2>&1 while [ -f "$lockfile" ] do sleep $num_seconds start_time=`date +%s%N` [ ! -d /mnt/user ] && continue ls -1 /mnt/user 2>/dev/null | while read dir do dir=`basename "$dir"` # Exclude the "data" directory from being processed, don't need it in memory [ "$dir" = "data" ] && continue [ "$dir" = "your_dir_to_exclude" ] && continue for i in /mnt/disk* do # disk0 is the user share, don't process it, as it is already in memory [ "$i" = "disk0" ] && continue # If the directory does not exist on this disk, don't do recursive "ls -R" [ ! -d "$i"/"$dir" ] && continue # Perform a recursive "ls -R" on /mnt/disk??/share ls -R "$i"/"$dir" >/dev/null 2>&1 done done end_time=`date +%s%N` # track how long the recursive "ls" is taking. If it starts to take longer it must be # because it has to read more from the physical disk. If so, adjust the timing to # perform the "ls" more frequently. elapsed_time=$(($end_time-$start_time)) [ $prior_elapsed_time -lt $elapsed_time -a $num_seconds -gt 0 ] && num_seconds=$(($num_seconds-1)) [ $prior_elapsed_time -gt $elapsed_time -a $num_seconds -lt 10 ] && num_seconds=$(($num_seconds+1)) prior_elapsed_time=$elapsed_time done & # while loop was put into background, now disown it # so it will continue to run when you log off # to get it to stop, type: rm /var/lock/cache_dirs echo $! > "${lockfile}" disown %% Warning: I'm adding this note everywhere I made a post regarding this technique of repeatedly accessing directory entries to keep them in the cache. When accessing the disks, using "ls" or "find" or anything else, they are "busy" and unable to be un-mounted. If even one disk is unable to be un-mounted you will not be able to "stop" the array without being caught by a surprise... If what I describe here happens to you, do not panic. What will happen when you press "Stop" is those disks that can be un-mounted will be... the "busy" disk will not be able to be un-mounted. (normally all disks are un-mounted) When unRAID is unable to un-muont all the drives then management interface will not stop the array. Instead, it will show all the disks that could be un-mounted (incorrectly) as Unformatted. The "busy" disk will show as normal. This is frightening when you first encounter it as the management interface even presents a "format" button (whatever you do, do NOT press it). All you need do is press "Stop" once more (or as needed) to catch the added script between its scans when it is sleeping and not accessing a disk. As I said, whatever you do, do not press the "Format" button, as it would try to re-format all the disks it incorrectly thinks are unformatted. (they are not unformatted, just un-mounted, so their file-systems not visible. This is a bug in the display logic of the management interface.) As some point in the future the unRAID interface will be enhanced to terminate these add-on scripts prior to stopping the array, but as of 4.5-beta6 it does not, and you will see the unformatted display in error if you catch a script in the middle of a scan. Again, all you need do is press "Stop" once more (or as needed) to catch the added script between its scans when it is sleeping and not accessing a disk. Joe L.
April 24, 200917 yr Joe- Why use 'ls' and not 'find'? I think using find's -depth option would result in the higher level inodes being 'touched' later, hence be less subject to being replaced in the dentry cache. Also, could use -maxdepth level to limit how far down the directory tree you go - it may not be necessary to cache the entire tree in memory. Take for example a 'Movies' directory. You only need to cache the entries for 'Movies' and each 'Movies/Title' directory. Beyond that probably not necessary for normal "browsing". Perhaps use the 'split level' as a hint for how far down the tree to go?? Just thinking out loud here
April 24, 200917 yr Just thinking out loud here Good Idea's!! I prefer find myself. It's coming along nice Joe.
April 24, 200917 yr Agreed, coming along great! I'm very leery of basing anything on the Split Level settings, for a number of reasons, and for that matter, even messing with the -maxdepth. Part of the reason for this exercise was to give new users a script that for the most part needed no customization/editing. First, you have to deal with Split Level values of 0, 99 or greater, and the new non-numeric values. Then you have to worry about whether the user understood the Split Level setting well enough to set it correctly (not at all guaranteed), and then you don't know for sure what level the user's scanning software will go to. It seems logical that a decent 'Media Center' app would browse for the movie title names only, but I definitely would not want to count on that, some may scan deeper. And last is the non-trivial matter that every User Share may have a different Split Level setting. You could initially include a depth limiting method, defaulting to fairly high, and allow interested users to customize it if they wish, but it still would need to be customizable for each User Share. I can't see how these directory nodes can take that much memory. Is it worth the extra effort?
April 24, 200917 yr I would definitely like to see some type of level limiting feature. I believe the perception of a performance issue could be resolved by defaulting the depth to 2 as this allows a user to see the root folder, then at least 1 folder underneath. For the most part this would get me to where I need to be on the drive, then I could wait for the spin up time. Of course allowing the user to tweak this setting is preferred. Thanks for all of your hard work Joe.
April 24, 200917 yr I can't see how these directory nodes can take that much memory. Is it worth the extra effort? I'm in agreement, a user customizable maxdepth parameter has a certain value to it and coding is easier then split level. I guess if it was going to be that advanced to base something on a timer, split level, drive sleep, etc, etc... Then perhaps emhttp could have an internal timer that could use system callas and do the directory scanning. In the meantime an environment variable or external config file can contain a max depth parameter and if defined use it, if not do not define anything. Now about memory, From what I saw in a little bit of kernel reading, the Dentry cache hash table is set at boot time. I could be wrong in this, but it was something I noticed (and looks to be a configurable option). If my machine wasn't so busy I would reboot with new parameters to test it. dcache.c __setup("dhash_entries=", set_dhash_entries); On one machine it is 500 on another it is 128. I don't know if this has something that could help nor do I know exactly how it is used.
April 24, 200917 yr We've talked elsewhere of adding more logging of addons and tweaks. This seems an excellent candidate. It would be very helpful to see what exactly is going to be cached (perhaps just the top level folders), and any depth settings, and timer settings. A user would have a better idea of what to adjust, from the logged config and the actual spin up behavior.
April 24, 200917 yr I like the idea of depth of ls/find but i question its usefulness. If a user was rebooting his server every night then sure but in every other scenario what we have is one long HDD scrape followed by hopefully very few/no more. I think the real focus of effort should be where Webotech is going namely how do we keep our precious cache. This makes the assumption that cache takes up very little memory but do we really care if a level 2 scrape takes 2 MB and a full scrape takes 16.? Surely the added beenfits of less spin ups, cooler server and disks and massive improveent in real reponse for the end user is worth that trivial amount of RAM. Im sure someone will come up with a scenario where partial scrapes are of benefit but I stick by my guns that the real action should be in cache preservation
April 27, 200917 yr Joe- Why use 'ls' and not 'find'? I started with "find" then switched to "ls -R" thinking it would be more efficient. As it turns out, I think "find" is actually quicker. I think using find's -depth option would result in the higher level inodes being 'touched' later, hence be less subject to being replaced in the dentry cache. I looked at the source code for "find" and it seems the traversal of the inodes is still from the top of the tree, it is just the printing or exec'ing that is done bottom up, and it creates its own in-memory tree at that, so, as far as I can see, I don't think the higher level "nodes" are actually touched later. Also, could use -maxdepth level to limit how far down the directory tree you go - it may not be necessary to cache the entire tree in memory. Take for example a 'Movies' directory. You only need to cache the entries for 'Movies' and each 'Movies/Title' directory. Beyond that probably not necessary for normal "browsing". Perhaps use the 'split level' as a hint for how far down the tree to go?? Just thinking out loud here The maxdepth limit does not help too much in my case, since most of my media is only two levels down, but it might help some. I've modified my script once more... This time you can specify on the command line the max and min time to sleep between directory scans. The loop will automatically adjust the sleep time based on the average duration of a directory scan vs. the time of the last scan. You can also specify an alternate "find -maxdepth" vs. "ls -R." You can specify the "maxdepth" for the scan, and lastly, you can specify your own command to use instead of "find" or "ls." It will be invoked with the disk/directory as its argument for each of your disks, for each of your user-shares. One last feature, by default it detaches itself from your terminal session and allows you to log off with it running. By using the "-F" option, you can run it in the foreground where it will print to the screen the directory scan time, the average scan time, and the directory scan interval as it runs. You can watch it adjust the timing as you perform various tasks on the server and put the disks into use by playing a movie. (In the background it does not print the statistics) Unless you exclude certain share folders, it will cache all /mnt/disk* top-level folder hierarchies. You can exclude specific shares by using the -e share_name_to_exclude option. You can have multiple -e options on the command line. Example: If I wanted to exclude a folder named "joe" and another named "unraid config files" I would invoke with: -e "joe" -e "unraid config files" Put the share name to exclude in quotes if it has any embedded spaces like this: -e "Junk Movies" You can get help by typing cache_dirs -h root@Tower:/boot/custom/bin# cache_dirs -h Usage: ./cache_dirs [-m min_seconds] [-M max_seconds] [-F] [-d maxdepth] [-f] [-c command] [-e exclude_dir][-e exclude_dir] -m NN = minimum seconds to wait between directory scans (default=0) -M NN = maximum seconds to wait between directory scans (default=10) -F = do NOT run in background, run in Foreground and print statistics as it loops and scans -f = use "find" instead of "ls -R" -d NN = use "find -maxdepth NN" instead of "ls -R" (-f is not needed in addition, it is assumed if -d is used) -c command = use command instead of "ls -R" (command should be quoted if it has embedded spaces) -e exclude_dir (may be repeated as many times as desired) To test, I usually invoke it as cache_dirs -F -d 3 -e "data" Using "-d" automatically infers the use of "find -maxdepth" so you do not need to specify -f as well. That results in the following (since the command is not put into the background) [pre] root@Tower:/boot/custom/bin# cache_dirs -F -d 3 -e data Executed find -maxdepth 3 in 0.191483 seconds, avg=0.191483 seconds, now sleeping 5 seconds Executed find -maxdepth 3 in 0.191763 seconds, avg=0.191623 seconds, now sleeping 4 seconds Executed find -maxdepth 3 in 0.190594 seconds, avg=0.191280 seconds, now sleeping 5 seconds Executed find -maxdepth 3 in 0.528078 seconds, avg=0.275479 seconds, now sleeping 4 seconds Executed find -maxdepth 3 in 1.056556 seconds, avg=0.431695 seconds, now sleeping 3 seconds Executed find -maxdepth 3 in 0.377409 seconds, avg=0.422647 seconds, now sleeping 4 seconds Executed find -maxdepth 3 in 0.327844 seconds, avg=0.409104 seconds, now sleeping 5 seconds Executed find -maxdepth 3 in 0.343205 seconds, avg=0.400867 seconds, now sleeping 6 seconds Executed find -maxdepth 3 in 0.805775 seconds, avg=0.445856 seconds, now sleeping 5 seconds Executed find -maxdepth 3 in 0.396309 seconds, avg=0.440902 seconds, now sleeping 6 seconds Executed find -maxdepth 3 in 0.353097 seconds, avg=0.432919 seconds, now sleeping 7 seconds Executed find -maxdepth 3 in 0.403112 seconds, avg=0.430435 seconds, now sleeping 8 seconds Executed find -maxdepth 3 in 0.474028 seconds, avg=0.433789 seconds, now sleeping 7 seconds Executed find -maxdepth 3 in 0.803757 seconds, avg=0.460215 seconds, now sleeping 6 seconds Executed find -maxdepth 3 in 0.323222 seconds, avg=0.451082 seconds, now sleeping 7 seconds Executed find -maxdepth 3 in 0.795963 seconds, avg=0.472637 seconds, now sleeping 6 seconds Executed find -maxdepth 3 in 0.347227 seconds, avg=0.465260 seconds, now sleeping 7 seconds Executed find -maxdepth 3 in 0.873279 seconds, avg=0.487928 seconds, now sleeping 6 seconds [/pre] To have the command run in the background, just leave off the "-F" Now you can choose to use ls -R, or find, or anything else you desire. On my server, the loop delay goes to zero seconds when I'm doing lots of I/O moving things around on disks and works its way back up to 9 or 10 seconds when activity stops or slows. The "-F" option lets you see what is happening. If you put it into the background, it can be killed by removing the lockfile: rm /var/lock/cache_dirs.LCK or, in the new version, type: cache_dirs -q [/b]If invoked in the foreground, you can stop it by typing Control-C Have fun, A "zipped" copy of the newest version of "cache_dirs" is attached here: http://lime-technology.com/forum/index.php?topic=3666.msg33498#msg33498 Lots of things have been improved. Most of the suggestions are incorporated. Warning: I'm adding this note everywhere I made a post regarding this technique of repeatedly accessing directory entries to keep them in the cache. When accessing the disks, using "ls" or "find" or anything else, they are "busy" and unable to be un-mounted. If even one disk is unable to be un-mounted you will not be able to "stop" the array without being caught by a surprise... If what I describe here happens to you, do not panic. What will happen when you press "Stop" is those disks that can be un-mounted will be... the "busy" disk will not be able to be un-mounted. (normally all disks are un-mounted) When unRAID is unable to un-muont all the drives then management interface will not stop the array. Instead, it will show all the disks that could be un-mounted (incorrectly) as Unformatted. The "busy" disk will show as normal. This is frightening when you first encounter it as the management interface even presents a "format" button (whatever you do, do NOT press it). All you need do is press "Stop" once more (or as needed) to catch the added script between its scans when it is sleeping and not accessing a disk. As I said, whatever you do, do not press the "Format" button, as it would try to re-format all the disks it incorrectly thinks are unformatted. (they are not unformatted, just un-mounted, so their file-systems not visible. This is a bug in the display logic of the management interface.) As some point in the future the unRAID interface will be enhanced to terminate these add-on scripts prior to stopping the array, but as of 4.5-beta6 it does not, and you will see the unformatted display in error if you catch a script in the middle of a scan. Again, all you need do is press "Stop" once more (or as needed) to catch the added script between its scans when it is sleeping and not accessing a disk. Joe L. Joe L.
Archived
This topic is now archived and is closed to further replies.