June 23, 200917 yr Sorry to be thick here but I have a simple question. At home I have a Movies share so my Go command is /boot/custom/bin/cache_dirs -w -i Movies which only caches the Movies Share. On a test box at work we have a share named "Shared" with a subfolder called "Movies". How can I just cache that Movies subfolder without everything else in the Shared share? Would it just be /boot/custom/bin/cache_dirs -w -i Shared/Movies? Thanks, Scott
June 23, 200917 yr Sorry to be thick here but I have a simple question. At home I have a Movies share so my Go command is /boot/custom/bin/cache_dirs -w -i Movies which only caches the Movies Share. On a test box at work we have a share named "Shared" with a subfolder called "Movies". How can I just cache that Movies subfolder without everything else in the Shared share? Would it just be /boot/custom/bin/cache_dirs -w -i Shared/Movies? Thanks, Scott the cache_dirs program is not written to include/exclude sub-directories of a user-share. It might be possible to modify it for your own needs by editing the following line and hard-coding your own directory At line 348, there is a call to a function to build the directory list to be scanned. You can add a line after it to force a specific folder (or folders separated by spaces) dir_list=`build_dir_list` <-- Existing line dir_list="Shared/Movies" <-- line to be added If there are more than one directory, then it would look like this: dir_list="Shared/Movies Shared/Music" <-- alternate sample line to be added Once you do that, the -i and -e options will not work... but you will have your specific list. Joe L.
June 23, 200917 yr ok cool...just tried it although not sure how to know if it is working or not but I trust you
June 23, 200917 yr ok cool...just tried it although not sure how to know if it is working or not but I trust you Run it in the foreground with the -F -v options and it will tell you what it is scanning. Joe L.
June 25, 200917 yr I also tried this script and found it fully working when inside UnRAID itself (ls -R in each disk share works perfectly) and also when using Windows command prompt to list contents of a mapped drives. However when using Windows Explorer to access either mapped drives or UNC-paths, I get disk spinning up whenever reaching file level. To me it seems that Windows Explorer is always somehow accessing the file contents, at least with my Windows XP setup. This might partly explain why some people are reporting disks still spinning up. But in my case the behaviour is consistent and repeatable (drive always spins up if there is a file, not directory, within the visible directory) that I wonder what is the normal case. Should cache_dirs work with the Windows Explorer? This is no problem for me since my main use case is Meedio doing media file imports which work correctly with cache_dirs.
June 25, 200917 yr I am/was using the "Details" view with default settings (Name, Size, Type, Date Modified fields visible). The system I'm currently using has variety of sw installed so it is likely that at least one of them has "hooked" to the explorer and trying to access the file contents. My point is, that people should not use Windows Explorer trying to verify whether cache_dirs is working or not. Or if they do and it seems not to be working properly they should try next windows command prompt.
June 25, 200917 yr very sensible advice. Just as an fyi on XP SP2 i have no explorer problems that I have detected so far.
June 26, 200917 yr Just as an fyi on XP SP2 i have no explorer problems that I have detected so far. I use XP pro SP3 and i also have no problems for now.
June 26, 200917 yr I also tried this script and found it fully working when inside UnRAID itself (ls -R in each disk share works perfectly) and also when using Windows command prompt to list contents of a mapped drives. However when using Windows Explorer to access either mapped drives or UNC-paths, I get disk spinning up whenever reaching file level. To me it seems that Windows Explorer is always somehow accessing the file contents, at least with my Windows XP setup. This might partly explain why some people are reporting disks still spinning up. But in my case the behaviour is consistent and repeatable (drive always spins up if there is a file, not directory, within the visible directory) that I wonder what is the normal case. Should cache_dirs work with the Windows Explorer? This is no problem for me since my main use case is Meedio doing media file imports which work correctly with cache_dirs. I have a utility (a dll maybe, I can't remember offhand) that will give details of video files such as bitrates and audio track type and picture size and this info is available in Windows Explorer. It reads the data if a file is selected so I'd bet this utility would cause the drives to spin up. It's main use is to provide more info in Media Browser which is handy to have. Peter
June 28, 200917 yr Has anyone noticed frequent parity checks being triggered whilst using this script?
June 28, 200917 yr No. I don't have that kind of behavior. No automatic parity check since i use the script. edit : today, 08/19/2009, I did. The syslog says that disk 6 was unmounted last night. I suppose this is the reason. Aug 18 18:25:39 tvixhd4 status[12562]: No active PIDS on the array Aug 18 18:25:40 tvixhd4 rc.unRAID[12666]: Killing active pids on the array drives Aug 18 18:25:40 tvixhd4 rc.unRAID[12728]: Umounting the drives Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: /dev/md1 umounted Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: /dev/md2 umounted Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: /dev/md3 umounted Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: /dev/md4 umounted Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: /dev/md5 umounted Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: umount: /mnt/disk6: device is busy Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: umount: /mnt/disk6: device is busy Aug 18 18:25:42 tvixhd4 rc.unRAID[12732]: /dev/md7 umounted Aug 18 18:25:42 tvixhd4 rc.unRAID[12732]: /dev/md8 umounted Aug 18 18:25:42 tvixhd4 rc.unRAID[12757]: Stopping the Array Aug 18 18:25:42 tvixhd4 kernel: mdcmd (10): stop Aug 18 18:25:42 tvixhd4 kernel: md: 2 devices still in use.
June 28, 200917 yr Has anyone noticed frequent parity checks being triggered whilst using this script? I've never seen any induced parity checks... What I have seen with this script, with cache-pressure set to zero (the older script did this), and a smaller amount of installed RAM (I have 512 Meg), you could run out of RAM and the OOM (out-of-memory) process in the kernel will start killing off processes... this can cripple your server's ability to be seen on the network, and eventually require you to reboot, possibly without a clean shutdown. This will result in a parity check when you reboot since you did not shut down cleanly. Other than that, if you are experiencing parity checks, and you did not press the "Check Parity" button, and not scheduled one to be performed at a regular interval, then your server is rebooting on its own, or you are rebooting as a result of a momentary power blip, and the parity check is occurring because it was not shut down cleanly. It could also be as a result of bad memory, or memory that has the voltage, timing , or clock speed incorrectly set in the BIOS. a memory test is in order... and manually verify the BIOS settings are appropriate for your specific memory strips. Do not rely on "auto" settings as they frequently get it wrong. A post of your syslog is in order if you desire assistance interpreting what else might be happening. Instructions under "Troubleshooting" in the wiki. Joe L.
August 2, 200916 yr I figure it is appropriate to post the fixed version of cache_dirs that will "lock" the cache drive in addition to the data drives to prevent it from showing as "Unformatted" when the array is stopped. This is version 1.6.3 and it is attached here http://lime-technology.com/forum/index.php?topic=3666.msg40393#msg40393 Joe L.
August 2, 200916 yr I figure it is appropriate to post the fixed vjavascript:void(0);ersion of cache_dirs that will "lock" the cache drive in addition to the data drives to prevent it from showing as "Unformatted" when the array is stopped. This is version 1.6.1 and it is attached. Joe L. Still a newbie and noticed that whenever I just look at my Movies share all the drive must spin up. So this looks like what I need to stop this. So I just copy the cache_dirs file to my root folder then edit my go file (as shown) in the config folder restart and I'll be set? EDIT: Also Joe the date on the file downloaded from your link is Jun 25. This is version 1.6.1?
August 2, 200916 yr Looks good to me. About the only option you really need is the -w -m and -M already have defaults that will work in most situations of 1 second and 10 seconds. The loop will automatically adjust itself somewhere between those limits based on the time it takes to scan the directories. The way you have it, it will not loop more than once every 3 seconds, nor any longer than every 5 seconds. The -d 5 will limit it to caching directory entries to the top 5 levels of folders. (typically fine for most people) If you do not specify it, it will scan everything. Joe L.
August 2, 200916 yr I have another question regarding RAM usage: Is there an easy way to check how much RAM DirCache requires or uses after full scan of directories? What would happen, if RAM is not sufficient? Probably it would drop older cache entries - thus always "trashing" the dircache. Is this something I can "check" or monitor? I just want to make sure, if my box has enough RAM (and no idea, how much RAM dircache allocates per file and/or directory ...) Has anybody ever had problems with RAM Cache and S3-sleepmode (or more specific: after wakening up the machine)? Thanks, Guzzi
August 2, 200916 yr I have another question regarding RAM usage: Is there an easy way to check how much RAM DirCache requires or uses after full scan of directories? What would happen, if RAM is not sufficient? Probably it would drop older cache entries - thus always "trashing" the dircache. Is this something I can "check" or monitor? I just want to make sure, if my box has enough RAM (and no idea, how much RAM dircache allocates per file and/or directory ...) Has anybody ever had problems with RAM Cache and S3-sleepmode (or more specific: after wakening up the machine)? Thanks, Guzzi type free before you run it, and then again after it has run for a while. Nobody has enough RAM to buffer a 1TB disk... If you play a 4Gig movie, then all its disk blocks will be cached... or at least as much as will fit in memory. If you check parity, then you are reading every block in your server... If there is not enough RAM the least-recently-used blocks will be freed from the cache and used. Is for this reason the cache_dirs program must constantly scan... to keep the directory blocks from being the least-recently used.
August 2, 200916 yr Sure, I can't cache the content of one or more TB disks - but to my understanding cachedir only keeps the directoies and their entries of the files in memory - file access itself requires access to the disk. If I get e.g.: total used free shared buffers cached Mem: 968992 940572 28420 0 22016 854944 -/+ buffers/cache: 63612 905380 Swap: 0 0 0 .. this looks to me, that all memory is used - so I cannot be sure, if cachedirs can keep all info in the RAM, right? What would happen, if not? I assume, that the disks would not spin down, because with constantly re-reading it would require access to the disks again, when old dirinfos have been dropped!? So is it right, that the best indication that all dirinfo fits into ram is when all disks spin down and stay there? (Above is from a running machine, so I might need to reboot to get full info that is not "poisoned" by other accesses...
August 2, 200916 yr Sure, I can't cache the content of one or more TB disks - but to my understanding cachedir only keeps the directoies and their entries of the files in memory - file access itself requires access to the disk. If I get e.g.: total used free shared buffers cached Mem: 968992 940572 28420 0 22016 854944 -/+ buffers/cache: 63612 905380 Swap: 0 0 0 .. this looks to me, that all memory is used - so I cannot be sure, if cachedirs can keep all info in the RAM, right? What would happen, if not? I assume, that the disks would not spin down, because with constantly re-reading it would require access to the disks again, when old dirinfos have been dropped!? So is it right, that the best indication that all dirinfo fits into ram is when all disks spin down and stay there? (Above is from a running machine, so I might need to reboot to get full info that is not "poisoned" by other accesses... cache_dirs accesses the directory listings, same as you would if you were browsing them to find a piece of music to play, or a movie to watch. It is just that you don't browse every 5 seconds. You are correct, you show 1 Gig of ram, almost all (855 Meg) of it being used for buffer cache. You are right, if the disks spin up when a cache_dirs scan occurs, then its directory entries were displaced out of cache by something that needed the memory. Odds are you will not run into this with as much memory as you have very often. If you want to clear the buffer memory cache, without rebooting, you can type this: sync; echo 3 > /proc/sys/vm/drop_caches Joe L.
August 2, 200916 yr [...] You are correct, you show 8 Gig of ram, almost all of it being used. [...] Thanks for clarification! ... the only problem with this machine is - it's only 1 GB, not 8 ;-) edit: Well, not really a problem - I just rebooted and waited until reads on all disks finished - takes about half of the RAM, so it should be fine. I was worried because of many pictures and thumbs on the box, but seems to be no problem: total used free shared buffers cached Mem: 968992 504740 464252 0 144068 138008 -/+ buffers/cache: 222664 746328 Swap: 0 0 0
August 19, 200916 yr Has anyone noticed frequent parity checks being triggered whilst using this script? It did it today. I'm not sure it is cache-dirs related but i have no other explications. Last night, I did a clean shutdown as always with the powerdown script. Today, when i started the unraid, it did a parity check. Yesterday syslog says : Aug 18 18:25:39 tvixhd4 status[12562]: No active PIDS on the array Aug 18 18:25:40 tvixhd4 rc.unRAID[12666]: Killing active pids on the array drives Aug 18 18:25:40 tvixhd4 rc.unRAID[12728]: Umounting the drives Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: /dev/md1 umounted Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: /dev/md2 umounted Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: /dev/md3 umounted Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: /dev/md4 umounted Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: /dev/md5 umounted Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: umount: /mnt/disk6: device is busy Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: umount: /mnt/disk6: device is busy Aug 18 18:25:42 tvixhd4 rc.unRAID[12732]: /dev/md7 umounted Aug 18 18:25:42 tvixhd4 rc.unRAID[12732]: /dev/md8 umounted Aug 18 18:25:42 tvixhd4 rc.unRAID[12757]: Stopping the Array Aug 18 18:25:42 tvixhd4 kernel: mdcmd (10): stop Aug 18 18:25:42 tvixhd4 kernel: md: 2 devices still in use. The unraid box was the only one still in the lan, so i don't think it was accessed from anything. That leave me with one explanation : the cache_dirs script. I know I can stop it with typing cache_dirs -q I'd like to add that line into the powerdown script so it will to it for sure before stopping the array and the unraid. Any suggestion where to put the line ? I tough i put it here : logger "Umounting the drives" cache_dirs -q for disk in /mnt/disk* do /bin/umount ${disk} done or logger "Killing active pids on the array drives" cache_dirs -q for fs in /mnt/user /mnt/disk* do if [ ! -d ${fs} ] ; then continue ; fi for pid in $(fuser -cu $fs 2>/dev/null) do ps --no-headers -fp ${pid} kill -TERM ${pid} # sleep 1 #if kill -0 ${pid} 2>/dev/null # then kill -9 ${pid} #fi done done 2>&1 | logger
August 19, 200916 yr Has anyone noticed frequent parity checks being triggered whilst using this script? It did it today. I'm not sure it is cache-dirs related but i have no other explications. Last night, I did a clean shutdown as always with the powerdown script. Today, when i started the unraid, it did a parity check. Yesterday syslog says : Aug 18 18:25:39 tvixhd4 status[12562]: No active PIDS on the array Aug 18 18:25:40 tvixhd4 rc.unRAID[12666]: Killing active pids on the array drives Aug 18 18:25:40 tvixhd4 rc.unRAID[12728]: Umounting the drives Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: /dev/md1 umounted Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: /dev/md2 umounted Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: /dev/md3 umounted Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: /dev/md4 umounted Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: /dev/md5 umounted Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: umount: /mnt/disk6: device is busy Aug 18 18:25:41 tvixhd4 rc.unRAID[12732]: umount: /mnt/disk6: device is busy Aug 18 18:25:42 tvixhd4 rc.unRAID[12732]: /dev/md7 umounted Aug 18 18:25:42 tvixhd4 rc.unRAID[12732]: /dev/md8 umounted Aug 18 18:25:42 tvixhd4 rc.unRAID[12757]: Stopping the Array Aug 18 18:25:42 tvixhd4 kernel: mdcmd (10): stop Aug 18 18:25:42 tvixhd4 kernel: md: 2 devices still in use. The unraid box was the only one still in the lan, so i don't think it was accessed from anything. That leave me with one explanation : the cache_dirs script. I know I can stop it with typing cache_dirs -q I'd like to add that line into the powerdown script so it will to it for sure before stopping the array and the unraid. Any suggestion where to put the line ? I tough i put it here : logger "Umounting the drives" cache_dirs -q for disk in /mnt/disk* do /bin/umount ${disk} done or logger "Killing active pids on the array drives" cache_dirs -q for fs in /mnt/user /mnt/disk* do if [ ! -d ${fs} ] ; then continue ; fi for pid in $(fuser -cu $fs 2>/dev/null) do ps --no-headers -fp ${pid} kill -TERM ${pid} # sleep 1 #if kill -0 ${pid} 2>/dev/null # then kill -9 ${pid} #fi done done 2>&1 | logger It is not a surprise... The cache_dirs script constantly loops and scans the disks every few seconds. Put the extra line at the top of the powerdown script. In fact, put two, add a "sleep 10" after it to allow time for the process to exit. Use the full path to the cache_dirs script, otherwise it might not be in the current PATH. Oh yes, depending on when you got the cache_dirs script... get a current one. the current version forces ALL disks to be busy so you can't accidentally see "Unformatted" disks on the management console. The "Stop" button will have no effect the first time you attempt to use it, but will be effective a second time if pressed within a few minutes of the first. /boot/cache_dirs -q sleep 10
August 19, 200916 yr Perhaps I'll have to add some tweaking to the rc.unRAID script. It does do a kill on any pid which is using a drive. In fact it does 3. Perhaps that is an older version. here is what is in my current version. It should have also logged what was active on the array. logger "Killing active pids on the array drives" for fs in /mnt/user /mnt/disk* do [ ! -d ${fs} ] && continue for signal in TERM TERM KILL do for pid in $(fuser -cu $fs 2>/dev/null) do ps --no-headers -fp ${pid} if kill -0 ${pid} 2>/dev/null then kill -${signal} ${pid} fi done [ ! -z "${pid}" ] && sleep 1 done done 2>&1 | logger Does the cache_dirs script have a trap to remove the pidfile and kill any active children if it receives a SIGTERM or SIGKILL
August 19, 200916 yr Perhaps I'll have to add some tweaking to the rc.unRAID script. Does the cache_dirs script have a trap to remove the pidfile and kill any active children if it receives a SIGTERM or SIGKILL It has a trap, but only when run in the foreground... So no... it does not have a trap... at least not yet. The current version spawns on child shell per disk, and that shell "cd's" to the disk to keep it busy to prevent an "Unformatted" display if you attempt to stop the array while it is running. They in turn sleep and loop while a lock file exists. fuser looks like this: (from /usr/bin/lsof /dev/md*) COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME cache_dir 1818 root cwd DIR 9,1 424 2 /mnt/disk1 cache_dir 1820 root cwd DIR 9,10 320 2 /mnt/disk10 cache_dir 1822 root cwd DIR 9,11 504 2 /mnt/disk11 cache_dir 1824 root cwd DIR 9,2 344 2 /mnt/disk2 cache_dir 1826 root cwd DIR 9,3 288 2 /mnt/disk3 cache_dir 1828 root cwd DIR 9,4 336 2 /mnt/disk4 cache_dir 1830 root cwd DIR 9,5 376 2 /mnt/disk5 cache_dir 1832 root cwd DIR 9,6 368 2 /mnt/disk6 cache_dir 1834 root cwd DIR 9,7 288 2 /mnt/disk7 cache_dir 1836 root cwd DIR 9,8 424 2 /mnt/disk8 sleep 21517 root cwd DIR 9,3 288 2 /mnt/disk3 sleep 21518 root cwd DIR 9,4 336 2 /mnt/disk4 sleep 21519 root cwd DIR 9,1 424 2 /mnt/disk1 sleep 21520 root cwd DIR 9,6 368 2 /mnt/disk6 sleep 21521 root cwd DIR 9,10 320 2 /mnt/disk10 sleep 21522 root cwd DIR 9,8 424 2 /mnt/disk8 sleep 21523 root cwd DIR 9,5 376 2 /mnt/disk5 sleep 21524 root cwd DIR 9,2 344 2 /mnt/disk2 sleep 21525 root cwd DIR 9,11 504 2 /mnt/disk11 sleep 21526 root cwd DIR 9,7 288 2 /mnt/disk7 The code used is # Create a child process with a "current working directory" on each of the disks # These will prevent their un-mounting, and they will prevent an unexpected surprise # of "unformatted" disks showing in the management web-page when some disks are un-mounted # and other disks are not able to be un-mounted, because they were "busy" # This guarantees that all the disks will be "busy" until the lockfile is removed # or these child processes are terminated. No disk may then be un-mounted until these # processes terminate. They will all self-terminate if the lockfile is removed. if [ "$force_disk_busy_flag" = "yes" ] then a=0 for i in /mnt/disk* /mnt/cache do [ ! -d $i ] && continue ( cd $i; while test -f "$lockfile"; do sleep 2; done ) & bg_process[$a]=$! a=$(($a+1)) done fi If I detect an attempt is made to stop the array, I use the bg_processes array to terminate the child processes keeping each disk busy, so I know I can add a "trap" for SIGKILL to do the same with relative ease. What do you think? Joe L.
Archived
This topic is now archived and is closed to further replies.