August 19, 200916 yr If I detect an attempt is made to stop the array, I use the bg_processes array to terminate the child processes keeping each disk busy, so I know I can add a "trap" for SIGKILL to do the same with relative ease. What do you think? This is what is needed. I should add, I have not looked at the code for dir_cache. In my case, it would not help me. My system is too busy with torrents for this technique to be effective. I usually do something like: trap "some shell commands;rm -f $pidfile;" HUP INT QUIT TERM EXIT You cannot trap kill -9, but the others are the most comment to stop scripts.
August 19, 200916 yr trap "some shell commands;rm -f $pidfile;" HUP INT QUIT TERM EXIT You cannot trap kill -9, but the others are the most comment to stop scripts. When cache_dirs is run in the foreground, I currently use this trap "rm -f $lockfile; kill $background_pid 2>/dev/null; exit" INT TERM EXIT I don't have anything but the lockfile in place when it is run in the background. I know all about kill -9 ... but I've seen plenty of times where it was ignored at the device driver level. (Yeah.. I wrote some unix device driver code at one point in my career... ) kill works on processes in user-space. It does not help you out of a deadlock in kernel space (ask me how I know ) Joe L.
August 20, 200916 yr Idea for Joe's excellent tool. Parse /proc/wharever for inode count. Save the last value after scan complete to a permanent file. Use that for APPROX % complete option.
August 20, 200916 yr It is not a surprise... The cache_dirs script constantly loops and scans the disks every few seconds. Put the extra line at the top of the powerdown script. In fact, put two, add a "sleep 10" after it to allow time for the process to exit. Use the full path to the cache_dirs script, otherwise it might not be in the current PATH. Oh yes, depending on when you got the cache_dirs script... get a current one. the current version forces ALL disks to be busy so you can't accidentally see "Unformatted" disks on the management console. The "Stop" button will have no effect the first time you attempt to use it, but will be effective a second time if pressed within a few minutes of the first. /boot/cache_dirs -q sleep 10 Thx for the tip.
September 6, 200916 yr Wow. This has made a HUGE difference to browsing my files. Thanks!! Edit: I tend to turn my unraid box off after use (generally used for watching movies one or two evenings a week). Last night I was unable to stop the array 'normally'. Think I read in this thread something about pressing stop twice but cannot find it.. can someone point me to the thread or explain how I safely shut-down? Thanks
September 7, 200916 yr How do you "normally" shut down? The process was coded to look in the syslog for your attempt to stop the array and to stop its process that keeps the disks from being un-mounted. This takes a few seconds, as the cache_dirs has to wake up first. It sleeps up to 10 seconds per loop... So... press "Stop" wait 10 - 15 seconds, press "Stop" again. Or... type cache_dirs -q on the command line and you can shut down as normal.
September 7, 200916 yr How do you "normally" shut down? The process was coded to look in the syslog for your attempt to stop the array and to stop its process that keeps the disks from being un-mounted. This takes a few seconds, as the cache_dirs has to wake up first. It sleeps up to 10 seconds per loop... So... press "Stop" wait 10 - 15 seconds, press "Stop" again. Or... type cache_dirs -q on the command line and you can shut down as normal. Sorry, I use the IE interface, press "stop" and wait for the "powerdown" (?) button to appear to safely power-down. I will do as you suggest. Many thanks.
September 24, 200916 yr I normally shut down my unRaid by pressing the power buttom on it. Will that still work with this add-on ? Have read the tread but not 100% sure.
September 24, 200916 yr Yes. But you might have to push it twice. The cache_dirs script constantly loops and scans the disks every few seconds. Put the extra line at the top of the powerdown script. In fact, put two, add a "sleep 10" after it to allow time for the process to exit. Use the full path to the cache_dirs script, otherwise it might not be in the current PATH. Oh yes, depending on when you got the cache_dirs script... get a current one. the current version forces ALL disks to be busy so you can't accidentally see "Unformatted" disks on the management console. The "Stop" button will have no effect the first time you attempt to use it, but will be effective a second time if pressed within a few minutes of the first. /boot/cache_dirs -q sleep 10
October 11, 200916 yr I install powerdown with a package and I think I'm better off adding "cache_dirs -q" programmaticly, on every boot, after the package installs. #installing package(s) for powerdown" CTRLALTDEL=YES LOGSAVE=30 installpkg /boot/packages/powerdown-1.02-noarch-unRAID.tgz #bind physical power button to powerdown script sysctl -w kernel.poweroff_cmd="/sbin/powerdown" sed -i -e "s/init 0/powerdown/" /etc/acpi/acpi_handler.sh ... <bunch of other stuff> ... #v1.6.1 of cache_dirs - caches file list to avoid drive spinup while browsing /boot/cache_dirs -d 5 -m 3 -M 5 -w #stop caching directories in powerdown" { rm /sbin/powerdown && awk '{ sub(/^#!\/bin\/bash.*$/,"#!/bin/bash\n/boot/cache_dirs -q\nsleep 10")}1'> /sbin/powerdown; } < /sbin/powerdown What this avoids is you modifying the 1.02 version of powerdown and putting it on flash and overwriting /sbin/powerdown on every boot. When 1.04 comes out, if you're not careful (and I'll forget this stuff faster then I learned it) you'll still be overwriting /sbin/powerdown with the 1.02 version.
October 11, 200916 yr I'm running 4.5 beta 6 - where do I find the poowerdown script ? I run 4.4.2 But I think it's an add on package in any version. Start here: http://lime-technology.com/wiki/index.php?title=UnRAID_Add_Ons#Power_Down
October 13, 200916 yr With the release of unRAID 4.5b7, the unRAID web-management console no longer shows Unformatted when attempting to stop an array with busy disks. Instead, it just displays an "Unmounting" status on the disks that are busy and unable to be un-mounted. You can see what it looks like in this post: http://lime-technology.com/forum/index.php?topic=4484.msg40363#msg40363 It also no longer gives a user any opportunity to press the "Stop" button on the user-interface a second time... as it will wait for all disks to be free and constantly attempt to un-mount them until they are successfully un-mounted. For users of cache_dirs, through version 1.6.2, you must then log in via telnet and stop cache_dirs yourself. Once you do, the array will then stop as instructed. An updated version of cache_dirs is now available to look for the new status messages being logged in the syslog in 4.5b7, and it will stop itself. It is attached to this post. http://lime-technology.com/forum/index.php?topic=4500.0 In the interim, if you attempt to stop the array while running any of the older versions, you will be presented with a screen filled with "Unmounting" descriptions until you log in and terminate the process by typing cache_dirs -q Joe L.
October 14, 200916 yr I just started playing around with this tonight. It is working pretty good. I have a question though. I ran "cache_dirs -d 4". I navigated down 2 levels, "\Tower\TV\I want that\". With all disk spun down it was relatively instant to go down to that depth. Oddly enough though, when I navigated back up to "\Tower\TV" I had to wait for a disk to spin up. Why would that be? unRAID 4.5-beta7 cache_dir 1.6.3 Browsing \tower using Windows 7.
October 14, 200916 yr With all disk spun down it was relatively instant to go down to that depth. Oddly enough though, when I navigated back up to "\Tower\TV" I had to wait for a disk to spin up. Why would that be? It depends on how many disk/files there are vs how much ram you have. Even with 4G, there is still a finite amount of hash'ed dentry's.
October 14, 200916 yr With all disk spun down it was relatively instant to go down to that depth. Oddly enough though, when I navigated back up to "\Tower\TV" I had to wait for a disk to spin up. Why would that be? It depends on how many disk/files there are vs how much ram you have. Even with 4G, there is still a finite amount of hash'ed dentry's. I got 2Gigs of RAM, which I thought would be plenty, but I get what your saying. Thanks.
October 14, 200916 yr Where are the cache_dirs command line parameters all documented? I believe if you run it as "cache_dirs -help" you get the parameters documentation. That is unless your looking for more than that, which I know I could use. I've seen bits of more explanation scatter about this thread.
October 14, 200916 yr Where are the cache_dirs command line parameters all documented? type cache_dirs -h Or, you can always read the source... it is just a shell script, and it has a fair amount of comments. It appears as if the built-in help does not list the "-B" option that will cause the cache_dirs program to omit making all the disks "busy" That option might be used with the latest version of unRAID, as it no longer shows "Unformatted" when unable to un-mount a file system currently being scanned. Consider it as a un-documented feature. Joe L.
October 14, 200916 yr With all disk spun down it was relatively instant to go down to that depth. Oddly enough though, when I navigated back up to "\Tower\TV" I had to wait for a disk to spin up. Why would that be? It depends on how many disk/files there are vs how much ram you have. Even with 4G, there is still a finite amount of hash'ed dentry's. I got 2Gigs of RAM, which I thought would be plenty, but I get what your saying. Thanks. The 2 gigs of ram are shared between the operating system and all the data being read or written to the disks. A small portion of that ram represents the data blocks holding directory entries, and it is freed and re-used if not referenced, if they are the least recently used blocks in the disk cache and the cache needs a block because you are reading, or writing to a data disk. (playing a movie, calculating parity, copying files to the server) Also... Windows like to cache its own files (they call it an indexed search) it might have been scanning your files on the server below 4 directories. One more thing... are you sure it was running? You typed "cache_dir" in your post and its name is "cache_dirs" You can type ps -ef and see it in the process list if it is.
October 14, 200916 yr The 2 gigs of ram are shared between the operating system and all the data being read or written to the disks. A small portion of that ram represents the data blocks holding directory entries, and it is freed and re-used if not referenced (if they are the least recently used blocks in the disk cache and the cache needs a block because you are reading, or writing to a data disk. (playing a movie, calculating parity, copying files to the server) Also... Windows like to cache its own files (they call it an indexed search) it might have been scanning your files on the server below 4 directories. One more thing... are you sure it was running? You typed "cache_dir" in your post and its name is "cache_dirs" You can type ps -ef and see it in the process list if it is. It was a mistype. It definitely was running. To confirm it, I killed the process, spun down the disks and cleared the statistics, then re-ran it. I saw the disks spin up and reads happen. I then spun down the disks and accessed the directory of \\tower. I didn't think windows did indexing on remote shares, but it most likely is a windows thing. Either way, the initial browsing is still much faster.
October 28, 200916 yr Hi, I'm trying use cache_dirs with the new beta7, In my process list I have 8 cache_dir entries: root 1874 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 1878 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 1880 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 1883 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 1886 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 1893 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 1898 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 1923 1898 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 7536 1923 0 20:11 pts/0 00:00:00 find /mnt/disk1/Media -noleaf root 7538 1880 0 20:11 pts/0 00:00:00 sleep 2 root 7540 1886 0 20:11 pts/0 00:00:00 sleep 2 root 7543 1883 0 20:11 pts/0 00:00:00 sleep 2 root 7544 1878 0 20:11 pts/0 00:00:00 sleep 2 root 7546 1893 0 20:11 pts/0 00:00:00 sleep 2 root 7548 1874 0 20:11 pts/0 00:00:00 sleep 2 The lock file references 1874. Below is the cache dir portion of my syslog. I ran it manually using this command. [b]root@Tower:/flash/config# /boot/cache_dirs -w[/b] cache_dirs process ID 1898 started, To terminate it, type: cache_dirs -q root@Tower:/flash/config# cd /var root@Tower:/var# cd lock root@Tower:/var/lock# ls cache_dirs.LCK root@Tower:/var/lock# cat * 1898 root@Tower:/var/lock# Any thoughts? Thanks syslog Oct 27 20:00:20 Tower cache_dirs: ============================================== Oct 27 20:00:20 Tower cache_dirs: command-args=-w Oct 27 20:00:20 Tower cache_dirs: vfs_cache_pressure=10 Oct 27 20:00:20 Tower cache_dirs: max_seconds=10, min_seconds=1 Oct 27 20:00:20 Tower cache_dirs: max_depth=9999 Oct 27 20:00:20 Tower cache_dirs: command=find -noleaf Oct 27 20:00:20 Tower cache_dirs: version=1.6.4 Oct 27 20:00:20 Tower cache_dirs: ---------- caching directories --------------- Oct 27 20:00:20 Tower cache_dirs: Backup-ReadOnly Oct 27 20:00:20 Tower cache_dirs: Backups Oct 27 20:00:20 Tower cache_dirs: Downloads Oct 27 20:00:20 Tower cache_dirs: Media Oct 27 20:00:20 Tower cache_dirs: Others Oct 27 20:00:20 Tower cache_dirs: Sage Oct 27 20:00:20 Tower cache_dirs: ftp Oct 27 20:00:20 Tower cache_dirs: ---------------------------------------------- Oct 27 20:00:20 Tower cache_dirs: cache_dirs process ID 1898 started, To terminate it, type: cache_dirs -q
October 28, 200916 yr Hi, I'm trying use cache_dirs with the new beta7, In my process list I have 8 cache_dir entries: root 1874 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 1878 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 1880 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 1883 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 1886 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 1893 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 1898 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 1923 1898 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 7536 1923 0 20:11 pts/0 00:00:00 find /mnt/disk1/Media -noleaf root 7538 1880 0 20:11 pts/0 00:00:00 sleep 2 root 7540 1886 0 20:11 pts/0 00:00:00 sleep 2 root 7543 1883 0 20:11 pts/0 00:00:00 sleep 2 root 7544 1878 0 20:11 pts/0 00:00:00 sleep 2 root 7546 1893 0 20:11 pts/0 00:00:00 sleep 2 root 7548 1874 0 20:11 pts/0 00:00:00 sleep 2 The lock file references 1874. Below is the cache dir portion of my syslog. I ran it manually using this command. [b]root@Tower:/flash/config# /boot/cache_dirs -w[/b] cache_dirs process ID 1898 started, To terminate it, type: cache_dirs -q root@Tower:/flash/config# cd /var root@Tower:/var# cd lock root@Tower:/var/lock# ls cache_dirs.LCK root@Tower:/var/lock# cat * 1898 root@Tower:/var/lock# Any thoughts? Thanks syslog Oct 27 20:00:20 Tower cache_dirs: ============================================== Oct 27 20:00:20 Tower cache_dirs: command-args=-w Oct 27 20:00:20 Tower cache_dirs: vfs_cache_pressure=10 Oct 27 20:00:20 Tower cache_dirs: max_seconds=10, min_seconds=1 Oct 27 20:00:20 Tower cache_dirs: max_depth=9999 Oct 27 20:00:20 Tower cache_dirs: command=find -noleaf Oct 27 20:00:20 Tower cache_dirs: version=1.6.4 Oct 27 20:00:20 Tower cache_dirs: ---------- caching directories --------------- Oct 27 20:00:20 Tower cache_dirs: Backup-ReadOnly Oct 27 20:00:20 Tower cache_dirs: Backups Oct 27 20:00:20 Tower cache_dirs: Downloads Oct 27 20:00:20 Tower cache_dirs: Media Oct 27 20:00:20 Tower cache_dirs: Others Oct 27 20:00:20 Tower cache_dirs: Sage Oct 27 20:00:20 Tower cache_dirs: ftp Oct 27 20:00:20 Tower cache_dirs: ---------------------------------------------- Oct 27 20:00:20 Tower cache_dirs: cache_dirs process ID 1898 started, To terminate it, type: cache_dirs -q When you "cat" the lock-file itself, it showed 1898, when you looked in the syslog, it showed the process as 1898. Where do you see a lock file referencing 1874? As far as I can see from what you posted, all is normal.
October 28, 200916 yr Looking at your process list I can see these parent-child relationships. Parent process, and owner of the lock-file. root 1898 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w Child process, running loop that will scan the directories. root 1923 1898 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w Grand-Child process, performing "find" to get directory entries into cache. root 7536 1923 0 20:11 pts/0 00:00:00 find /mnt/disk1/Media -noleaf Processes used to hold disks busy. Each of these will run until the lockfile is removed or the process whose ID is in the lockfile is terminated. Each has a child process sleeping for 2 seconds, looping for 2 seconds more each time through the loop if the lockfile exists, exiting if it does not.. root 1874 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 7548 1874 0 20:11 pts/0 00:00:00 sleep 2 root 1878 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 7544 1878 0 20:11 pts/0 00:00:00 sleep 2 root 1880 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 7538 1880 0 20:11 pts/0 00:00:00 sleep 2 root 1883 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 7543 1883 0 20:11 pts/0 00:00:00 sleep 2 root 1886 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 7540 1886 0 20:11 pts/0 00:00:00 sleep 2 root 1893 1 0 20:00 pts/0 00:00:00 /bin/bash /boot/cache_dirs -w root 7546 1893 0 20:11 pts/0 00:00:00 sleep 2
Archived
This topic is now archived and is closed to further replies.