queeg Posted May 9, 2010 Share Posted May 9, 2010 What about -prune option? Would that work to exclude sub dirs? EDIT: Here's an example that I just figured out. It prints everything except the packages directory. find /boot/config \( -name packages -prune \) -o -name "*" -print And one that prints everything except subdirs ending in "old" or the packages directory. find /boot \( -name "*old" -prune -o -name packages -prune \) -o -name "*" -print Cool. I'll try to integrate this into it. Quote Link to comment
Joe L. Posted May 9, 2010 Author Share Posted May 9, 2010 Actually, the ability to add arguments is built in with the "-a" option. An example: cache_dirs -a '-noleaf -name Steam -prune -o -print' Or, if you have more than one directory to exclude: cache_dirs -a '-noleaf \( -name Steam -prune -o -name data -prune \) -o -print' Your example would be cache_dirs -a '-noleaf \( -name "*old" -prune -o -name packages -prune \) -o -print' No need to hack at cache_dirs at all. It was designed to be flexible. When using the -a option you need to use the -noleaf at the beginning of the -a option argument, and -print at the end to get correct syntax. The entire -a option argument is enclosed in single quotes as in -a '-noleaf ....... -print' Individual shell wild-cards "*" can be enclosed in double quotes as in the "*old" example above. Not too difficult... Joe L. Quote Link to comment
queeg Posted May 10, 2010 Share Posted May 10, 2010 Actually, the ability to add arguments is built in with the "-a" option. An example: cache_dirs -a '-noleaf -name Steam -prune -o -print' Or, if you have more than one directory to exclude: cache_dirs -a '-noleaf \( -name Steam -prune -o -name data -prune \) -o -print' Your example would be cache_dirs -a '-noleaf \( -name "*old" -prune -o -name packages -prune \) -o -print' No need to hack at cache_dirs at all. It was designed to be flexible. When using the -a option you need to use the -noleaf at the beginning of the -a option argument, and -print at the end to get correct syntax. The entire -a option argument is enclosed in single quotes as in -a '-noleaf ....... -print' Individual shell wild-cards "*" can be enclosed in double quotes as in the "*old" example above. Not too difficult... Joe L. That sounds easier. I had added a new -x parameter but trying to put it into all the find's get me tangled up. This is what I'm going to use. /boot/cache_dirs -w -u -d 3 -a '-noleaf \( -name "ftproot" -prune -o -name "lost+found" -prune -o -name "iTunes" -prune \) -o -print' Quote Link to comment
queeg Posted May 10, 2010 Share Posted May 10, 2010 It doesn't seem to be working. The drive spins up going down the first directory leve. /boot/cache_dirs -w -u -d 3 -a '-noleaf \( -name "ftproot" -prune -o -name "lost+found" -prune -o -name "iTunes" -prune \) -o -print' Quote Link to comment
BRiT Posted May 10, 2010 Share Posted May 10, 2010 The first time the script runs, the drive has to be spun up to get the directory listings. If you mean the drive spuns up after the script is running and the drive has idled out to sping down, how are you browsing the files? If you're using Windows Explorer, make sure it's not trying to read the content of the files to display thumbnails. Quote Link to comment
queeg Posted May 10, 2010 Share Posted May 10, 2010 What I meant to say was I spun up the drives, ran the cache_dirs command, allowed the drives to spin down (actually just came back 7 hours later) and from windows I explored the directories while also watching the drives on the unRAID menu. After browsing down past the first directory the drive spun up. So I don't think cache_dirs with those parameters is working. Quote Link to comment
Joe L. Posted May 10, 2010 Author Share Posted May 10, 2010 What I meant to say was I spun up the drives, ran the cache_dirs command, allowed the drives to spin down (actually just came back 7 hours later) and from windows I explored the directories while also watching the drives on the unRAID menu. After browsing down past the first directory the drive spun up. So I don't think cache_dirs with those parameters is working. As already said, it depends on what you are browsing... If you have thumbnail views enabled, windows is looking at file contents, not just directory listings. Quote Link to comment
queeg Posted May 10, 2010 Share Posted May 10, 2010 The (Win7) explorer is set to Titles which is the default I think. Edit: Tiles (not Titles) Quote Link to comment
Joe L. Posted May 10, 2010 Author Share Posted May 10, 2010 The (Win7) explorer is set to Titles which is the default I think. Edit: Tiles (not Titles) Yes, "tiles" will show you small images of jpg files, and images of documents. It will be examining file content, not just directory data. cache_dirs is not opening every file to read them... Solution, change the view you are using to a file listing mode. Joe L. Quote Link to comment
queeg Posted May 10, 2010 Share Posted May 10, 2010 A little testing and here is what I see. It seems that even though explorer defaults to Tile it actually does request file info unless you manually set it to Tile. Open explorer browse top level folder: works browse any subdir: spins up drive. Open explorer Change view from Tiles to List browse top level folder: works browse any subdir: works change view from List back to Tiles browse other subdirs: continues to work. In any case, cache_dirs is working correctly with the added -a parameter so I'm happy and thanks Joe. Quote Link to comment
Joe L. Posted May 10, 2010 Author Share Posted May 10, 2010 A little testing and here is what I see. It seems that even though explorer defaults to Tile it actually does request file info unless you manually set it to Tile. Open explorer browse top level folder: works browse any subdir: spins up drive. Open explorer Change view from Tiles to List browse top level folder: works browse any subdir: works change view from List back to Tiles browse other subdirs: continues to work. In any case, cache_dirs is working correctly with the added -a parameter so I'm happy and thanks Joe. Don't be fooled by your test. Once a "file" is read, it is in the cache until it is replaced by something else needing the cache. To perform your test properly, you'll need to empty the cache between tests. To do that, type this in between each test: sync; echo 3 > /proc/sys/vm/drop_caches Quote Link to comment
paperclip Posted May 30, 2010 Share Posted May 30, 2010 Joe L., thanks for your great script. I do have a questions about the -d (max depth) parameter. Does the numbering start at 0 or 1? So if I had a "Music" share with the following structure: Music/The Eagles/Hotel California/Track01.mp3 What would be the appropriate max depth value? Does it even matter? I mean, if my folder structure didn't have a max depth exceeding 3, but I set cache_dirs to look 5 folders deep, would there be a performance hit? Thanks. Quote Link to comment
Joe L. Posted May 30, 2010 Author Share Posted May 30, 2010 Joe L., thanks for your great script. I do have a questions about the -d (max depth) parameter. Does the numbering start at 0 or 1? So if I had a "Music" share with the following structure: Music/The Eagles/Hotel California/Track01.mp3 What would be the appropriate max depth value? The argument is passed to the "find" command. Its manual page says -maxdepth levels Descend at most levels (a non-negative integer) levels of direc- tories below the command line arguments. `-maxdepth 0' means only apply the tests and actions to the command line arguments. You can experiment by changing the number and testing with this sample "find" command and see what it prints: find /mnt/disk*/Music -maxdepth 1 The depth is relative to the starting path, so a maxdepth of 0 would only cache the /mnt/disk*/Music folders and nothing in them or below them. To cache the hierarchy you described you would need at least a depth of 3. Does it even matter? I mean, if my folder structure didn't have a max depth exceeding 3, but I set cache_dirs to look 5 folders deep, would there be a performance hit? Thanks. Since you have no really deep hierarchies, there is no reason to specify any -d option. There is no penalty of using 5, or using the default of 999. Edit: fixed directory name in example Quote Link to comment
paperclip Posted May 30, 2010 Share Posted May 30, 2010 You can experiment by changing the number and testing with this sample "find" command and see what it prints: find /mnt/disk*/Mp3 -maxdepth 1 The depth is relative to the starting path, so a maxdepth of 0 would only cache the /mnt/disk*/Music folders and nothing in them or below them. To cache the hierarchy you described you would need at least a depth of 3. Since you have no really deep hierarchies, there is no reason to specify any -d option. There is no penalty of using the default of 999. Joe L., thanks for this information. Quote Link to comment
starcat Posted June 2, 2010 Share Posted June 2, 2010 Now, if your "browsing" of the directory is writing to them (MAC OS writes ._DS files) or opening files to show thumbnail views (windows will do this) then yes, the disks will spin up, since the file contents are not cached. Writing of .DS_Store files on network shares can be prevented by running these: defaults write /Library/Preferences/com.apple.desktopservices DSDontWriteNetworkStores true defaults write com.apple.desktopservices DSDontWriteNetworkStores true see this for details: http://www.macosxhints.com/article.php?story=20051130083652119 Quote Link to comment
starcat Posted June 6, 2010 Share Posted June 6, 2010 I am starting cache_dirs from my go script with the following command, basically delaying startup until all directories and mount points are created. echo "/boot/scripts/cache_dirs -w -B -u -e /mnt/disk*/Data -e /mnt/user/Data" | at now + 10 minute However I am still getting following error, am I doing something wrong? ERROR: excluded directory "/mnt/disk*/Data" does not exist. ERROR: excluded directory "/mnt/user/Data" does not exist. cache_dirs process ID 16607 started, To terminate it, type: cache_dirs -q Quote Link to comment
Joe L. Posted June 6, 2010 Author Share Posted June 6, 2010 I am starting cache_dirs from my go script with the following command, basically delaying startup until all directories and mount points are created. echo "/boot/scripts/cache_dirs -w -B -u -e /mnt/disk*/Data -e /mnt/user/Data" | at now + 10 minute However I am still getting following error, am I doing something wrong? ERROR: excluded directory "/mnt/disk*/Data" does not exist. ERROR: excluded directory "/mnt/user/Data" does not exist. cache_dirs process ID 16607 started, To terminate it, type: cache_dirs -q You can only exclude top level directories using the "-e" option. You can use the -a option though and add a "prune" command to the "find" being invoked, and it would look like this: cache_dirs -w -B -u -a '-noleaf -name Data -prune -o -print' It was discussed a few post ago. http://lime-technology.com/forum/index.php?topic=4500.msg61520#msg61520 Looking closer at your example, Data is a top level directory. You only need give the share name, not the full path to it. Try this: echo "/boot/scripts/cache_dirs -w -B -u -e Data" | at now + 10 minute also verify the folder does exist with the exact same capitilization as in the directory. type ls -l /mnt/disk*/Data if it does not exist, then perhaps the directory name is lower case and only showing mixed case in Windows when viewed through SAMBA. Joe L. Quote Link to comment
rsd Posted June 15, 2010 Share Posted June 15, 2010 Hi Joe, I could greatly use some help with cache dirs. I recently upgraded my mobo to a biostar 760+ and upgraded to unraid 4.54. I call cache_dirs in my go script using: /boot/cache_dirs -d 5 -w -e "VMC" Everything was working for a few weeks but now twice in the past 3 days I got some strange error that hard froze unraid (can't telnet to it, can't refresh the unmenu/web, when I manually hook up a monitor to it no commands are acknolwedged, can't see it on the network etc). It doesn't freeze after a reboot or anything but some random time afterwards (hours/days). This is what the last few syslog lines read from Unmenu: Jun 15 05:51:33 Tower kernel: [] ? sys_read+0x3b/0x60 Jun 15 05:51:33 Tower kernel: [] ? syscall_call+0x7/0xb Jun 15 05:51:33 Tower kernel: Code: 55 f0 e8 8a 82 d5 c8 8d 50 01 29 f2 39 d3 7c 0b 8b 45 0c 89 d3 c7 00 01 00 00 00 8b 45 f0 89 d9 81 c6 a0 3d 3e f8 c1 e9 02 89 38 a5 89 d9 83 e1 03 74 02 f3 a4 5a 89 d8 5b 5e 5f 5d c3 55 89 Jun 15 05:51:33 Tower kernel: EIP: [] md_cmd_proc_read+0x41/0x54 [md_mod] SS:ESP 0068:c238bef4 Jun 15 05:51:33 Tower kernel: CR2: 00000000f83e8000 Jun 15 05:51:33 Tower kernel: ---[ end trace 6baff4c42c121ab7 ]-- I see some different error messages when I go to the unraid box itself, a lot of stuff but this is the part that makes me think it is something cache_dirs related: /boot/cache_dirs/: Line 479: 30756 killed $command "$i"/$share_dir" $depth > /dev/null 2>&1 I know my cache_dirs call is slightly modified from before (I didn't use to exclude the "VMC" folder), so I'll try just calling it with -w and see what happens. Any help is greatly appreciated! Quote Link to comment
Joe L. Posted June 15, 2010 Author Share Posted June 15, 2010 Looks more like you are running out of memory. cache_dirs attempts to set a server kernel variable to be more aggressive in keeping directory entries in cache memory. As part of its script, it invokes sysctl vm.vfs_cache_pressure=10 You can override this with the "-p" option to cache_dirs. As delivered by lime-technology the value of 100 is used. So, you can try adding -p 50 or -p 100 or anything higher than 10. When memory is running out Linux will terminate processes that are idle in order to attempt to free memory for programs requesting it. Typically, the web-management interface is one of those idle, so it gets killed and you lose communications. In the same was SAMBA also gets killed off. You can re-start emhttp with killall emhttp /usr/local/sbin/emhttp & You can re-start samba with /root/samba restart Usually this will allow you to re-gain communications with the server so you can cleanly shut it down and re-start it. Quote Link to comment
rsd Posted June 15, 2010 Share Posted June 15, 2010 Thanks Joe, you're the best! Quote Link to comment
NAS Posted June 15, 2010 Share Posted June 15, 2010 That exlains alot. cat /proc/sys/vm/vfs_cache_pressure 10 is overriding my value of 0 in in go script. I was scratching my head why disk were spinning up more. Now we have an answer. Quote Link to comment
Joe L. Posted June 15, 2010 Author Share Posted June 15, 2010 That exlains alot. cat /proc/sys/vm/vfs_cache_pressure 10 is overriding my value of 0 in in go script. I was scratching my head why disk were spinning up more. Now we have an answer. I've run out of memory with cache_pressure set to "0" on my server with 512Meg of RAM. It is for that reason I set it to 10. (and it is still possible to run out of RAM if I try hard enough) Default value on Linux is 100. Just use -p 0 to keep your alternative value from being changed Quote Link to comment
NAS Posted June 15, 2010 Share Posted June 15, 2010 Way ahead of you. Purely based on casual observations setting 0 REALLY makes a difference for me. Wouldn't it be nice if the kernel would allow you to reserve a portion of memory dedicated to this cache then swap based on cache pressure and all the other magic if more than this was needed. Oh to know a kernel developer/ Quote Link to comment
Joe L. Posted June 15, 2010 Author Share Posted June 15, 2010 Way ahead of you. Purely based on casual observations setting 0 REALLY makes a difference for me. Wouldn't it be nice if the kernel would allow you to reserve a portion of memory dedicated to this cache then swap based on cache pressure and all the other magic if more than this was needed. Oh to know a kernel developer/ how much RAM do you have in your server? I know I've run out even with a swap file defined with cache pressure of 10. Quote Link to comment
NAS Posted June 15, 2010 Share Posted June 15, 2010 2GB i believe. I havent really looked at ram usage but from a user POV everything seems to just work Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.