April 14, 200818 yr This something is that has been annoying me for a long time is that harddrives spin up when you just browse folders. Can't you just cache directory structure to memory?
April 14, 200818 yr See this post. It is configurable. More memory allows for more cache: http://lime-technology.com/forum/index.php?topic=1789.0
April 16, 200818 yr Author Doesn't work. Tried it. If I list usershare i've been before later time after drives have spundown most of the drives spin back up. Also i wish there would be way to cache all directories so if i just list content of user shares no drives would be spin up. Because most like i just want to access one file anyway and that's probably just in one drive so spinning up most of the drives feels like overkill. Oddly not all drive spin up, just most of them when i listed that user share.
April 16, 200818 yr Author And with trying i meant that i set "vm.vfs_cache_pressure" to 0 and checked it it was really set to 0.
April 16, 200818 yr After I set my vm.vfs_cache_pressure to 0 I did a find down the whole tree to insure all files were read and cached. find /mnt -type f -ls >/dev/null 2>&1 I also added the locate package to the system so that all drives are read every night using updatedb. The tendency to spin up is diminished, however it still does spin up for me every now and then.
April 16, 200818 yr I have found the same... problem is, almost any movie I play from the unRAID server is way larger than my memory size, so a directory listing is soon replaced in cache with blocks of data from what I'm watching. It is even worse, as my MG-35 network media player times-out when requesting a directory listing if it does not respond within a second or two and then removes the whole directory from its file browser, probably thinking it has been removed from the server. I ended up with a script to test if my media players are online, and if they are to spin up the drives for me. In that same script is a similar "ls -R" You could probably put a command like yours as a cron script run every minute. It will probably spin up drives as needed, but only if the directory listing was not in memory. Here is a part of what I have in my script. It is invoked when a media server responds to a ping after I spin up my drives. It does an "ls" command every 30 seconds during a total run time of 2400 seconds. As I said, if the directory listing is in memory, the "ls" is nearly instant, if not, it gets the data from the physical drives. [pre] seconds_to_cache_dir=2400 time_now=`date '+%s'` end_time=$(( $time_now + $seconds_to_cache_dir )) ls_interval=30 while true do ls -R /mnt/user >/dev/null 2>/dev/null sleep $ls_interval now=`date '+%s'` if [ $now -gt $end_time ] then break; fi done [/pre] Joe L.
April 28, 200818 yr I currently have 13 data drives in my array and 2GB of RAM. I've tried setting vm.vfs_cache_pressure to 0 but I'm still experiencing long delays while all the drives sequentially spin-up when I browse my shares. Is it possible to use the cache drive as a permanent repository for the shares directory trees so that I only have to wait for one drive to spin up? Can this be done via Linux scripting? - paul
April 28, 200818 yr You might be able to get by this with a swap file on the cache drive instead. (Name it with a . first to prevent it from being moved). I have 4GB of ram on my machine and with the kernel tuned, my drives do not spin up on a directory listing or s complete filesystem search down the whole /mnt/user tree. However, I DID install the slocate package. Upon bootup I install it, then submit a batch job to run in 90 seconds which will scan all of the mounted filesystems. This forces all files to be read an in the cache. I also run this job daily via cron. root@unraid:~# cd /boot/packages/ root@unraid:/boot/packages# ls -l slocate-3.1-i486-1.tgz -rwxr-xr-x 1 root root 30270 Apr 18 2006 slocate-3.1-i486-1.tgz* root@unraid:/boot/packages# cd /boot/config/rc.local/ root@unraid:/boot/config/rc.local# ls -C S00-syslog-update S20-init.identd S20-install-bind S81-rsync_boot_cron S99-syslog-save S01-blockdev S20-init.mt-daapd S30-inittab-additions S90-smb-shares S02-sync-etc S20-init.proftpd S30-inittab-powerdown S90-tmpfs S03-cpufreq S20-init.rsyncd S75-useradds S91-init.vfs_cache S10-installpkg S20-init.sshd S80-update_hosts S92-init.slocate root@unraid:/boot/config/rc.local# ls -l S92-init.slocate -rwxr-xr-x 1 root root 276 Apr 13 11:30 S92-init.slocate* root@unraid:/boot/config/rc.local# cat S92-init.slocate #!/bin/sh PACKAGE=slocate-3.1-i486-1 if [ ! -f /var/log/packages/$PACKAGE ] then installpkg /boot/packages/$PACKAGE.tgz fi rm -rf /usr/doc/slocate-3.1 rm -rf /usr/man/man1/updatedb.1.gz /usr/man/man1/slocate.1.gz batch <<-EOF sleep 90 /usr/bin/updatedb -c /etc/updatedb.conf EOF Here is how fast a find or search is without spinning up the drive. top - 14:02:04 up 8 days, 5:57, 2 users, load average: 0.06, 0.03, 0.01 Tasks: 71 total, 1 running, 70 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 3114428k total, 2388696k used, 725732k free, 73656k buffers Swap: 0k total, 0k used, 0k free, 1967976k cached root@unraid:/mnt/user/pub# time ls -lR . > /tmp/ls,out real 0m1.061s user 0m0.220s sys 0m0.260s root@unraid:/mnt/user/pub# time find . -type f | wc -l 22192 real 0m0.552s user 0m0.040s sys 0m0.060s root@unraid:/mnt/user# ls -l total 1025009 drwx--x--x 1 root root 88 Apr 12 05:38 Videos/ drwxr-xr-x 1 root root 48 Apr 13 11:48 backups/ drwxr-xr-x 1 root root 368 Apr 21 05:02 gatekeeper/ drwx------ 1 root root 136 Apr 21 13:59 images/ -rwx------ 1 root root 1048576000 Mar 17 22:35 iobw.tst* drwxrwsr-x 1 501 522 744 Mar 28 11:14 music/ drwxrwsr-x 1 501 ftp 3840 Apr 26 11:43 pub/ -rwxr--r-- 1 root root 45 Apr 23 18:21 rsync.sh* drwxrwxr-x 1 root 500 1064 Feb 15 16:50 shared/ drwxr-xr-x 1 root root 208 Apr 13 11:37 unraid/ drwx------ 1 root root 96 Apr 26 13:29 video/ root@unraid:/mnt/user# time find . -type f | wc -l find: ./gatekeeper/vmware/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs/VMs: Too many levels of symbolic links 590630 real 0m8.972s user 0m0.760s sys 0m1.210s
April 28, 200818 yr After I set my vm.vfs_cache_pressure to 0 I did a find down the whole tree to insure all files were read and cached. find /mnt -type f -ls >/dev/null 2>&1 I also added the locate package to the system so that all drives are read every night using updatedb. The tendency to spin up is diminished, however it still does spin up for me every now and then. Yes this is an annoying issue. As noted, setting vfs_cache_pressure to 0 does not completely solve it & in looking at the vm code, I can see that inodes will still get kicked out of cache. I don't think a swap file will solve this, because (correct me if wrong) only process pages are swapped out, not cache pages. But let's say the fastest network connection lets you r/w at 50MB/sec. Then it's going to take 5 sec to fill up 250MB of cache (which is 1/2 the recommended memory). If we have a background daemon that wakes up, say every 2 seconds, and executes a 'find /mnt' command, seems like that should keep inodes in cache. I don't think you need the slocate package. When I get a chance, I'll try to cobble together a script to do this, but perhaps WeeboTech or Joe L. will beat me to it
April 28, 200818 yr heh, I guess I have so much ram and so little usage that once slocate is run, I'm good. The benefit of slocate is when you want to find a specific file at a specific location, you can just run locate filesegment and it will be listed with the full path.
April 29, 200818 yr I would have to agree with WeeboTech regarding locate. Most unRAID boxes do nothing at all for the vast majority of the day. Anything that can be run independent of the user that speeds up the users real time usage is a win in my book. e.g. I would happily have my unRAID box run for an hour every night doing things if i sped up my browsing or finding of file during the day.
April 29, 200818 yr When I get a chance, I'll try to cobble together a script to do this, but perhaps WeeboTech or Joe L. will beat me to it I think a C daemon may serve better here. I can write something up. I have a daemon in C I can use as a starting model. I was playing with ftw() and it just may be more efficient. This way you don't have an interpreted shell forking find every few minutes. The real question is, How often do you think this should be done?
April 29, 200818 yr The real question is, How often do you think this should be done? It should be configurable, of course
April 29, 200818 yr Interesting in that I was looking at ftw yesterday too. The interval should be one parameter, the list of starting folders another. It should detach itself from the controlling terminal, and probably should prevent two copies of itself from running concurrently. "C" code would not be too difficult. In fact, the "man" page for "ftw" has half of it already written as sample use of ftw(). Joe L.
April 29, 200818 yr The real question is, How often do you think this should be done? It should be configurable, of course Oh yes... I currently do an "ls -R" every 30 seconds when my media players are detected online. (see shell script excerpt in previous post in this thread) You probably only need to do the scan of the folders once an hour or so, as once the drives are spinning, they most likely will respond fast enough already, and they take an hour to spin down once idle. Eventually, I think the directory inodes will all be in the buffer cache, and the physical disks will not be accessed for over an hour, and the disks will be able to spin down on their own. So... scan interval should be configurable, of course... Joe L.
April 29, 200818 yr quote The interval should be one parameter, the list of starting folders another. It should detach itself from the controlling terminal, and probably should prevent two copies of itself from running concurrently. "C" code would not be too difficult. In fact, the "man" page for "ftw" has half of it already written as sample use of ftw(). Agreed, in fact I was playing with it. Today I'll incorporate it into a daemon shell (Which follows all of the rules mentioned above). However, I would like some input on defaults. I.E. default scan time. Although as I mentioned earlier, use of slocate/locate package would be a good enhancement to unraid. Especially if there was a "find" interface in the http interface. You could type in a string and get back a list of the files, then have hyperlinks to the files it found. Boy sometimes I just wish so much we had apache as the http daemon and emhttp was just a control daemon that received messages via IP (or pipe). OK, I'm wishing out loud. Back to topic. ;-)
April 29, 200818 yr This is what I have so far. I've added other controls in there. You can specify a timeout value for the loop (default 180 seconds). You can specify one or more directories. (default: /mnt/user). If timeout is 0, the daemon will pause until a signal is sent. Either a continue or an increment of the timeout value. This is so you can stop the daemon in a job. The other controls allow you increment/decrement and run an urgent job. For me this will be a way to signal a manual execution of the slocate scan. I put it there in the hopes that one day we can run a cgi program from the http daemon and then trigger this manual scan. root@rgclws ~/src/ftwd >./ftwd -? ./ftwd: invalid option -- ? ftwd: Daemon to walk a file tree at intervals to insure directory inodes remain in cache Usage: ftwd [ flags ] timeout dir Where: -t timeout value to recycle -d directory to scan (mulple -d [dir] -d [dir] allowed if unswitched items present the following is assumed arg1 = timeout, subsequent args are directories if timeout is 0, daemon will wait for a -C continue for every loop -k Kills active daemon -q queries if daemon is active -S Stops daemon from cycling, but keeps in memory uses SIGSTOP. Does not touch timeout value -C continues daemon execution -j allows configuration of an urgent jobname for request -X Runs urgent job such as a slocate scan for new files -v Increment verbose execution to syslog -D increment debug level -T Test routine, shows how arguments are resolved default is to daemonize and start cyclic file tree walk timeout=180, directory=/mnt/user Here's how it looks when working. Note that there are commands run by me(root), then the associated syslog messages (IF -V is set). root@rgclws ~/src/ftwd >./ftwd ftwd: spawned (pid=11140) root@rgclws ~/src/ftwd >./ftwd -k ftwd: terminating (pid=11140) /var/log/syslog messages Apr 29 13:22:19 rgclws ftwd[11140]: File Tree Walk Daemon Started Apr 29 13:23:04 rgclws ftwd[11140]: Interrupt Requested Apr 29 13:23:04 rgclws ftwd[11140]: File Tree Walk Daemon Ended, RC: -1 root@rgclws ~/src/ftwd >./ftwd -T timeout: 180 urgent job: /etc/cron.daily/slocate.cron argc: 1 argv[0]: /mnt/user root@rgclws ~/src/ftwd >./ftwd -T 30 /mnt/user /tmp timeout: 30 urgent job: /etc/cron.daily/slocate.cron argc: 2 argv[0]: /mnt/user argv[1]: /tmp root@rgclws ~/src/ftwd >./ftwd -V 30 /mnt/user /tmp ftwd: spawned (pid=11703) root@rgclws ~/src/ftwd >./ftwd -V 3 /mnt/user /tmp ftwd: already running (pid=11703) (killed then restarted) root@rgclws ~/src/ftwd > >./ftwd -V 3 /mnt/user /tmp ftwd: spawned (pid=11899) messages (ONLY IF VERBOSE) Apr 29 13:25:11 rgclws ftwd[11899]: File Tree Walk Daemon Started Apr 29 13:25:14 rgclws ftwd[11899]: walking file tree. Apr 29 13:25:14 rgclws ftwd[11899]: Processing: /mnt/user Apr 29 13:25:14 rgclws ftwd[11899]: Processing: /tmp Apr 29 13:25:14 rgclws ftwd[11899]: walk complete. Apr 29 13:25:17 rgclws ftwd[11899]: walking file tree. Apr 29 13:25:17 rgclws ftwd[11899]: Processing: /mnt/user Apr 29 13:25:17 rgclws ftwd[11899]: Processing: /tmp Apr 29 13:25:17 rgclws ftwd[11899]: walk complete. Apr 29 13:25:20 rgclws ftwd[11899]: walking file tree. Apr 29 13:25:20 rgclws ftwd[11899]: Processing: /mnt/user Apr 29 13:25:20 rgclws ftwd[11899]: Processing: /tmp Apr 29 13:25:20 rgclws ftwd[11899]: walk complete. Apr 29 13:25:23 rgclws ftwd[11899]: walking file tree. Apr 29 13:25:23 rgclws ftwd[11899]: Processing: /mnt/user Apr 29 13:25:23 rgclws ftwd[11899]: Processing: /tmp Apr 29 13:25:23 rgclws ftwd[11899]: walk complete. root@rgclws ~/src/ftwd >./ftwd -X Apr 29 13:25:46 rgclws ftwd[11899]: urgent job requested. /etc/cron.daily/slocate.cron Apr 29 13:25:48 rgclws ftwd[11899]: urgent job request complete. rc=0 root@rgclws ~/src/ftwd >./ftwd -V -S ftwd: signal(20) sent successfully, (pid=11899) Apr 29 13:26:15 rgclws ftwd[11899]: Stop requested. timeout set to 0 Apr 29 13:26:15 rgclws ftwd[11899]: Paused... root@rgclws ~/src/ftwd >./ftwd -V -C ftwd: signal(18) sent successfully, (pid=11899) Apr 29 13:26:36 rgclws ftwd[11899]: Continue requested. timeout set to 3 Apr 29 13:26:36 rgclws ftwd[11899]: walking file tree. Apr 29 13:26:36 rgclws ftwd[11899]: Processing: /mnt/user Apr 29 13:26:36 rgclws ftwd[11899]: Processing: /tmp Apr 29 13:26:36 rgclws ftwd[11899]: walk complete. root@rgclws ~/src/ftwd >./ftwd -V -1 ftwd: signal(10) sent successfully, (pid=11899) root@rgclws ~/src/ftwd >./ftwd -V -1 ftwd: signal(10) sent successfully, (pid=11899) root@rgclws ~/src/ftwd >./ftwd -V -1 ftwd: signal(10) sent successfully, (pid=11899) root@rgclws ~/src/ftwd >./ftwd -V -1 ftwd: signal(10) sent successfully, (pid=11899) Apr 29 13:27:03 rgclws ftwd[11899]: timeout set to 8 Apr 29 13:27:03 rgclws ftwd[11899]: walking file tree. Apr 29 13:27:03 rgclws ftwd[11899]: Processing: /mnt/user Apr 29 13:27:03 rgclws ftwd[11899]: Processing: /tmp Apr 29 13:27:03 rgclws ftwd[11899]: walk complete. Apr 29 13:27:05 rgclws ftwd[11899]: timeout set to 13 Apr 29 13:27:05 rgclws ftwd[11899]: walking file tree. Apr 29 13:27:05 rgclws ftwd[11899]: Processing: /mnt/user Apr 29 13:27:05 rgclws ftwd[11899]: Processing: /tmp Apr 29 13:27:05 rgclws ftwd[11899]: walk complete. Apr 29 13:27:07 rgclws ftwd[11899]: timeout set to 18 Apr 29 13:27:07 rgclws ftwd[11899]: walking file tree. Apr 29 13:27:07 rgclws ftwd[11899]: Processing: /mnt/user Apr 29 13:27:07 rgclws ftwd[11899]: Processing: /tmp Apr 29 13:27:07 rgclws ftwd[11899]: walk complete. Apr 29 13:27:10 rgclws ftwd[11899]: timeout set to 23 Apr 29 13:27:10 rgclws ftwd[11899]: walking file tree. Apr 29 13:27:10 rgclws ftwd[11899]: Processing: /mnt/user Apr 29 13:27:10 rgclws ftwd[11899]: Processing: /tmp Apr 29 13:27:10 rgclws ftwd[11899]: walk complete. root@rgclws ~/src/ftwd >./ftwd -V -2 ftwd: signal(12) sent successfully, (pid=11899) root@rgclws ~/src/ftwd >./ftwd -V -2 ftwd: signal(12) sent successfully, (pid=11899) root@rgclws ~/src/ftwd >./ftwd -V -2 ftwd: signal(12) sent successfully, (pid=11899) root@rgclws ~/src/ftwd >./ftwd -V -2 ftwd: signal(12) sent successfully, (pid=11899) root@rgclws ~/src/ftwd >./ftwd -V -2 ftwd: signal(12) sent successfully, (pid=11899) root@rgclws ~/src/ftwd >./ftwd -V -2 ftwd: signal(12) sent successfully, (pid=11899) Apr 29 13:27:37 rgclws ftwd[11899]: timeout set to 18 Apr 29 13:27:37 rgclws ftwd[11899]: walking file tree. Apr 29 13:27:37 rgclws ftwd[11899]: Processing: /mnt/user Apr 29 13:27:37 rgclws ftwd[11899]: Processing: /tmp Apr 29 13:27:37 rgclws ftwd[11899]: walk complete. Apr 29 13:27:38 rgclws ftwd[11899]: timeout set to 13 Apr 29 13:27:38 rgclws ftwd[11899]: walking file tree. Apr 29 13:27:38 rgclws ftwd[11899]: Processing: /mnt/user Apr 29 13:27:38 rgclws ftwd[11899]: Processing: /tmp Apr 29 13:27:38 rgclws ftwd[11899]: walk complete. Apr 29 13:27:38 rgclws ftwd[11899]: timeout set to 8 Apr 29 13:27:38 rgclws ftwd[11899]: walking file tree. Apr 29 13:27:38 rgclws ftwd[11899]: Processing: /mnt/user Apr 29 13:27:38 rgclws ftwd[11899]: Processing: /tmp Apr 29 13:27:38 rgclws ftwd[11899]: walk complete. Apr 29 13:27:39 rgclws ftwd[11899]: timeout set to 3 Apr 29 13:27:39 rgclws ftwd[11899]: walking file tree. Apr 29 13:27:39 rgclws ftwd[11899]: Processing: /mnt/user Apr 29 13:27:39 rgclws ftwd[11899]: Processing: /tmp Apr 29 13:27:39 rgclws ftwd[11899]: walk complete. Apr 29 13:27:40 rgclws ftwd[11899]: timeout set to 0 Apr 29 13:27:40 rgclws ftwd[11899]: Paused... Apr 29 13:27:41 rgclws ftwd[11899]: timeout set to 0 Apr 29 13:27:41 rgclws ftwd[11899]: Paused... Apr 29 13:27:42 rgclws ftwd[11899]: Paused... This was an interesting exercise. Thoughts?
April 29, 200818 yr Looks interesting.... Looks like you found "getopt(3)" (or hand coded your own equivalent) Let me know when you are ready for beta-testing volunteers. Did you limit the number of open file descriptors it will use? (hint, you should) I've seen my shell script version lock up twice now. I'm not certain of the cause... it almost looked like the read of the user file-system "blocked" somewhere. I'll be curious if your daemon process will do the same. As I said earlier, I've got my shell script version looping at 30 second increments to do the "ls -R" and it seems to keep the directory inodes in memory when I'm playing a DVD.ISO from the server. Joe L. PS. Will you share the source code once you work out all the bugs?
April 29, 200818 yr > Looks like you found "getopt(3)" (or hand coded your own equivalent) I always use getopt. > Did you limit the number of open file descriptors it will use? (hint, you should) Yes, I used the model from the man page. > I'll be curious if your daemon process will do the same. If your shell script hangs, so will the daemon. I.E. Unless I fork off and do the scan. Right now I'm doing it all in one process. I can alter it to 30 seconds, tom seemed to suggest 3. Not sure what to do here. > PS. Will you share the source code once you work out all the bugs? Of course. It's only about 950 lines. I'll attach the current source for review. It's a quick hack I would compile it for slackware, only I shutdown my slackware vm machine last night, Therefore I have to wait till I get home to compile it for unraid.
May 2, 200818 yr I'm still working on this. I should be able to release a binary in a day or so. I'm still writing up the README. I got an idea this afternoon about adding functionality to monitor files and if they have been modified within a threshold run a specific job on them i.e. to synchronize them somewhere else. This could be an rsync to another drive, offsite or even be used as a triegger or replacement for the mover process. (if one wanted more immediate movement of files) I'm still thinking it through, but at the very least the first version will just scan the tree to keep it in memory. I actually have a need for the syncing function for my GIGABYTE-RAMDISK. I plan to scan the ramdisk and when files are older then 5 minutes, rsync them to a compact flash matched in size to the ramdisk. Even though the ramdisk is battery backed, I like the idea of a hard backup As the program works now, It uses it's own name to make it's pidfile so you can run more then one program if it is renamed I.E. ftwd-user 30 /mnt/user (Creates idfile /var/run/ftwd-user.pid) ftwd-cache -J rsyncjob.sh --age "00:00:30:00" -i 300 -d /mnt/cache (Creates pidfile /var/run/ftwd-cache.pid) Where age = "DD=days:HH=hours:MM=minutes:SS=Seconds" I have not fuilly thought this out, but one parameter is a -n newer parameter to insure files are not processed over and over again. I.E. a job is run when a file is newer then the newer flag file (could be pid file by default) But has to be unmodified for the -age parameter. I'm still working it all out. I have special job submission daemons like this at my job, people love it. Files come in from customers and all these jobs are triggered automagically. I have not choosen to allow per file job submission, only one submission per daemon. eh, I'll see how it goes.
May 11, 200818 yr I put the ability to run a job when new files appear past the threshold of when the daemon is started. Could be useful to run an rsync job or something like that. Plus I have to create a slackware package for it. Still testing it out.
Archived
This topic is now archived and is closed to further replies.