cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up

[Gizmotoy]

Not to derail the lively discussion here, but for anyone stumbling on this I just wanted to mention that the suggested parameters for preclearing did in fact solve the issue.  I'm on my second preclear cycle without a complaint from the server.  It does appear to be taking quite a bit longer, though.

 

As a quick follow-up, my run with the default settings took 25 hours.  I just finished one cycle with the new settings, and it took 50 hours.  So the settings Joe posted above give you exactly a 2x slowdown.  Beware.

[trurl]

basically, you ran out of free memory.

Would a swap file help with this?

 

I have 4GB memory and don't usually run with a swap file because creating it seems to be one of the longest parts of boot time.

 

But it might be worth it for those rare occasions when I need to do a preclear.

 

[Joe L.]

basically, you ran out of free memory.

Would a swap file help with this?

 

I have 4GB memory and don't usually run with a swap file because creating it seems to be one of the longest parts of boot time.

 

But it might be worth it for those rare occasions when I need to do a preclear.

no, it would not help at all.  (if you  needed one, and did not have one, processes would have been killed in an attempt to free memory and your server would have effectively "crashed"  Therefore, if you did not crash, you did not need one. )

 

Most people will never need a swap file on an unRAID server unless they have very limited RAM and are running memory intensive programs. (where the program itself needs the memory, not the disk-buffer-cache)  My older server only has 512 Meg of RAM, and unless I'm compiling a ffmpeg (a huge program) I do not need a swap file.

 

Joe L.

[Gizmotoy]

Not to derail the lively discussion here, but for anyone stumbling on this I just wanted to mention that the suggested parameters for preclearing did in fact solve the issue.  I'm on my second preclear cycle without a complaint from the server.  It does appear to be taking quite a bit longer, though.

So I recently bought 2 more 3TB drives and I'm in the process of preclearing them.  I'm using the modified (slower) command Joe posted above, but cache_dirs is still crashing.  It's happened twice now, so I thought that had fixed it but apparently it did not.  According to /usr/bin/free, I'm using 2GB of my 4GB of total system RAM, so it's not even close to running out of memory.  This seems like something else.  Any ideas?

[BRiT]

The key is how much low memory you have free. Also, how is cache_dirs crashing?

[Gizmotoy]

The key is how much low memory you have free. Also, how is cache_dirs crashing?

 

If the kernel is not crashing, why would low memory be important?  Isn't low memory that which is reserved for the kernel?  It looks like I have 8MB free, which seemed pretty low, but after some research online I can't find any other example /proc/meminfo outputs with more free low memory.  I don't know if that's unusually low for unRaid or not.

 

This is the current output of /proc/meminfo

cat /proc/meminfo
MemTotal:        4090796 kB
MemFree:         1073768 kB
Buffers:          459036 kB
Cached:          1676776 kB
SwapCached:            0 kB
Active:           753568 kB
Inactive:        1220544 kB
Active(anon):     510516 kB
Inactive(anon):     9604 kB
Active(file):     243052 kB
Inactive(file):  1210940 kB
Unevictable:      666824 kB
Mlocked:               0 kB
HighTotal:       3226632 kB
HighFree:        1064864 kB
[b]LowTotal:         864164 kB
LowFree:            8904 kB[/b]
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:             54156 kB
Writeback:         29500 kB
AnonPages:        505012 kB
Mapped:            32648 kB
Shmem:             15128 kB
Slab:             303184 kB
SReclaimable:     286392 kB
SUnreclaim:        16792 kB
KernelStack:        2536 kB
PageTables:         4420 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     2045396 kB
Committed_AS:    1552712 kB
VmallocTotal:     122880 kB
VmallocUsed:        9540 kB
VmallocChunk:     112436 kB
DirectMap4k:        6136 kB
DirectMap2M:      907264 kB

 

It's crashing the same way it was previously, which I described in this post two pages back: http://lime-technology.com/forum/index.php?topic=4500.msg207729#msg207729

 

For comparison, this is the most recent crash log excerpt:

Nov 30 20:18:01 Hyperion crond[1238]: ignoring /var/spool/cron/crontabs/root- (non-existent user) 
Nov 30 21:00:46 Hyperion kernel: mdcmd (68): spindown 0 (Routine)
Nov 30 21:07:01 Hyperion crond[1238]: ignoring /var/spool/cron/crontabs/root- (non-existent user) 
Nov 30 21:08:36 Hyperion kernel: ------------[ cut here ]------------
Nov 30 21:08:36 Hyperion kernel: WARNING: at arch/x86/kernel/apic/ipi.c:109 default_send_IPI_mask_logical+0x2f/0xb9() (Minor Issues)
Nov 30 21:08:36 Hyperion kernel: Hardware name: MS-7680
Nov 30 21:08:36 Hyperion kernel: empty IPI mask
Nov 30 21:08:36 Hyperion kernel: Modules linked in: md_mod xor i2c_i801 i2c_core ahci libahci r8169 sata_sil24 mvsas libsas scsi_transport_sas [last unloaded: md_mod] (Drive related)
Nov 30 21:08:36 Hyperion kernel: Pid: 11945, comm: cache_dirs Not tainted 3.0.30-unRAID #4 (Errors)
Nov 30 21:08:36 Hyperion kernel: Call Trace: (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c1028a84>] warn_slowpath_common+0x65/0x7a (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c101636e>] ? default_send_IPI_mask_logical+0x2f/0xb9 (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c1028afd>] warn_slowpath_fmt+0x26/0x2a (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c101636e>] default_send_IPI_mask_logical+0x2f/0xb9 (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c1014b93>] native_send_call_func_ipi+0x4c/0x4e (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c1049a53>] smp_call_function_many+0x18c/0x1a4 (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c105ee4a>] ? page_alloc_cpu_notify+0x2d/0x2d (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c105ee4a>] ? page_alloc_cpu_notify+0x2d/0x2d (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c1049a85>] smp_call_function+0x1a/0x1e (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c1049a9b>] on_each_cpu+0x12/0x27 (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c105fcbe>] drain_all_pages+0x14/0x16 (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c106031e>] __alloc_pages_nodemask+0x371/0x47f (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c1026fd4>] dup_task_struct+0x46/0x119 (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c10279ef>] copy_process+0x70/0x9f6 (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c1028443>] do_fork+0xce/0x1e6 (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c1032e31>] ? set_current_blocked+0x27/0x38 (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c1032f67>] ? sigprocmask+0x7e/0x89 (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c100819b>] sys_clone+0x1b/0x20 (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c130f59d>] ptregs_clone+0x15/0x38 (Errors)
Nov 30 21:08:36 Hyperion kernel:  [<c130ec65>] ? syscall_call+0x7/0xb (Errors)

[sambo]

Hello,

 

Wanted to have few informations, cache seems workin fine here but sometimes disk start spinning w/o any access ... I guess its because i run out of memory. Someone can confirm me?

 

free
             total       used       free     shared    buffers     cached
Mem:       8299848    8043288     256560          0     176584    7615500
-/+ buffers/cache:     251204    8048644
Swap:            0          0          0

 

Thanks

[bonienl]

Hello,

 

Wanted to have few informations, cache seems workin fine here but sometimes disk start spinning w/o any access ... I guess its because i run out of memory. Someone can confirm me?

 

free
             total       used       free     shared    buffers     cached
Mem:       8299848    8043288     256560          0     176584    7615500
-/+ buffers/cache:     251204    8048644
Swap:            0          0          0

 

Thanks

 

You have more than enough free memory...

 

It could be that something is trying to read more than just your directory content (windows explorer is notorious in that respect)...

 

[jeff.lebowski]

Hello,

 

Wanted to have few informations, cache seems workin fine here but sometimes disk start spinning w/o any access ... I guess its because i run out of memory. Someone can confirm me?

 

free
             total       used       free     shared    buffers     cached
Mem:       8299848    8043288     256560          0     176584    7615500
-/+ buffers/cache:     251204    8048644
Swap:            0          0          0

 

Thanks

 

You have more than enough free memory...

 

It could be that something is trying to read more than just your directory content (windows explorer is notorious in that respect)...

 

Possibly related: even WITH cache_dirs installed, viewing directories with Windows Explorer on an unRAID server would cause disks to spin up. Not so using Total Commander.

[Joe L.]

Hello,

 

Wanted to have few informations, cache seems workin fine here but sometimes disk start spinning w/o any access ... I guess its because i run out of memory. Someone can confirm me?

 

free
             total       used       free     shared    buffers     cached
Mem:       8299848    8043288     256560          0     176584    7615500
-/+ buffers/cache:     251204    8048644
Swap:            0          0          0

 

Thanks

 

You have more than enough free memory...

 

It could be that something is trying to read more than just your directory content (windows explorer is notorious in that respect)...

 

Possibly related: even WITH cache_dirs installed, viewing directories with Windows Explorer on an unRAID server would cause disks to spin up. Not so using Total Commander.

That is usually because Window's Explorer opens up the files themselves so it can present thumbnail images to you.  cache_dirs does not help with caching the contents of the files themselves.

[jeff.lebowski]

Stupid thumbs! 

[sambo]

I have samba disable, cant be that 

[bonienl]

I have samba disable, cant be that 

 

So what systems are accessing your server?

 

Likely something is trying to retrieve more than just the directory listing...

 

[sambo]

I disabled all access. Just run in background cache_dirs, and sometimes i saw in uumenu that some drives are spinup.

Thats why i was thinkin i run out of memory or sometihn like that.

 

 

I saw spindown event in syslog, there is not spinup?

[bonienl]

I disabled all access. Just run in background cache_dirs, and sometimes i saw in uumenu that some drives are spinup.

Thats why i was thinkin i run out of memory or sometihn like that.

 

 

I saw spindown event in syslog, there is not spinup?

 

It is a common thing for Linux to take most of your memory for caching purposes. This is dynamically released when needed by applications. No need to worry, you have plenty of free memory available.

 

[sambo]

My disk spinup this night, after like 10hours w/o spinning ... all my network was down, no access possible ...

 

Tue Jan 8 04:29:00 2013 Executed find  in 0.237657 seconds, weighted avg=0.237744 seconds, now sleeping 3 seconds

Tue Jan 8 04:29:03 2013 Executed find  in 0.239507 seconds, weighted avg=0.237898 seconds, now sleeping 2 seconds

Tue Jan 8 04:29:06 2013 Executed find  in 0.240106 seconds, weighted avg=0.238104 seconds, now sleeping 1 seconds

Tue Jan 8 04:29:07 2013 Executed find  in 0.238908 seconds, weighted avg=0.238181 seconds, now sleeping 1 seconds

Tue Jan 8 04:29:49 2013 Executed find  in 40.835681 seconds, weighted avg=4.104615 seconds, now sleeping 1 seconds

Tue Jan 8 04:29:56 2013 Executed find  in 5.818847 seconds, weighted avg=4.442791 seconds, now sleeping 1 seconds

Tue Jan 8 04:29:58 2013 Executed find  in 1.039318 seconds, weighted avg=4.299200 seconds, now sleeping 2 seconds

Tue Jan 8 04:30:00 2013 Executed find  in 0.309221 seconds, weighted avg=4.082264 seconds, now sleeping 3 seconds

Tue Jan 8 04:30:03 2013 Executed find  in 0.244694 seconds, weighted avg=3.858842 seconds, now sleeping 3 seconds

Tue Jan 8 04:30:07 2013 Executed find  in 0.512284 seconds, weighted avg=3.660871 seconds, now sleeping 3 seconds

Tue Jan 8 04:30:10 2013 Executed find  in 0.295178 seconds, weighted avg=3.440918 seconds, now sleeping 3 seconds

 

 

[pcjacobse]

I have a strange problem using this script (the latest version 1.6.6). I run unraid 5.0-rc8a on a full slackware 14 distro.

 

in my /boot/config/go i have:

/usr/bin/cache_dirs -d 5 -m 3 -M 5 -w

 

After the boot the cache_dirs script takes up 99% cpu in the foreground. Preventing my other bootscripts to run.

When I login on ssh and try to stop it using cache_dirs -q, i get the message that its not running. And indeed there is no lock file at /var/lock/cache_dirs.LCK, but the process is running.

 

I can kill the script. And than my boot scripts continue (starting sickbeard etc.)

 

When i run it in the foreground i get the following output:

# /usr/bin/cache_dirs -d 5 -m 3 -M 5 -w -F

Executed find in 0.076995 seconds, weighted avg=0.076995 seconds, now sleeping 4 seconds

rm: cannot remove '/var/lock/cache_dirs.LCK': No such file or directory

/usr/bin/cache_dirs: line 1: kill: (6493) - No such process

rm: cannot remove '/var/lock/cache_dirs.LCK': No such file or directory

 

I guess this happens because i rebooted, so there is still a line 'kernel: mdcmd (11): stop' in the syslog

 

The script does work when I use `/usr/bin/cache_dirs -d 5 -m 3 -M 5 -w -F -B`.

 

Any suggestions on how to fix this?

[Joe L.]

I have a strange problem using this script (the latest version 1.6.6). I run unraid 5.0-rc8a on a full slackware 14 distro.

 

in my /boot/config/go i have:

/usr/bin/cache_dirs -d 5 -m 3 -M 5 -w

 

After the boot the cache_dirs script takes up 99% cpu in the foreground. Preventing my other bootscripts to run.

When I login on ssh and try to stop it using cache_dirs -q, i get the message that its not running. And indeed there is no lock file at /var/lock/cache_dirs.LCK, but the process is running.

 

I can kill the script. And than my boot scripts continue (starting sickbeard etc.)

 

When i run it in the foreground i get the following output:

# /usr/bin/cache_dirs -d 5 -m 3 -M 5 -w -F

Executed find in 0.076995 seconds, weighted avg=0.076995 seconds, now sleeping 4 seconds

rm: cannot remove '/var/lock/cache_dirs.LCK': No such file or directory

/usr/bin/cache_dirs: line 1: kill: (6493) - No such process

rm: cannot remove '/var/lock/cache_dirs.LCK': No such file or directory

 

I guess this happens because i rebooted, so there is still a line 'kernel: mdcmd (11): stop' in the syslog

 

The script does work when I use `/usr/bin/cache_dirs -d 5 -m 3 -M 5 -w -F -B`.

 

Any suggestions on how to fix this?

Only one.

 

Apparently your version of slackware is using a different version of the "bash" shell (or you are using a different shell entirely) and the

disown %%

command is not recognized.  This would exhibit exactly the symptoms you described.

 

So, if not using "bash", use it.  If using a different version of the "bash" shell, change to one with a working "disown" command.

If your environment does not have the

$SHELL

variable set, set it to /bin/bash before invoking the script.

If your version of "bash" is not in /bin/bash, make a link for it there, or modify the first few lines in the script to where it lives on your system.

 

Joe L.

[Helmonder]

I start cache dirs from my GO file with the following command:

 

# Starting cache_dirs
/usr/local/sbin/cache_dirs -w -B -u -m10 -M60 -d9999 -e /mnt/user/Music

 

Syslog shows the following error:

 

Jan 11 13:21:00 Tower cache_dirs: ==============================================
Jan 11 13:21:00 Tower cache_dirs: command-args=-w -m 10 -M 60 -d 9999 -e /mnt/user/Music -B -u -a -noleaf
Jan 11 13:21:00 Tower cache_dirs: vfs_cache_pressure=10
Jan 11 13:21:00 Tower cache_dirs: max_seconds=60, min_seconds=10
Jan 11 13:21:00 Tower cache_dirs: max_depth=9999
Jan 11 13:21:00 Tower cache_dirs: command=find -noleaf
Jan 11 13:21:00 Tower cache_dirs: version=1.6.6
Jan 11 13:21:00 Tower cache_dirs: ---------- caching directories ---------------
Jan 11 13:21:00 Tower cache_dirs: 
Jan 11 13:21:00 Tower cache_dirs: ----------------------------------------------
Jan 11 13:21:00 Tower cache_dirs: ERROR: excluded directory "/mnt/user/Music" does not exist.
Jan 11 13:21:00 Tower cache_dirs: cache_dirs process ID 17895 started, To terminate it, type: cache_dirs -q

 

I would like to exclude the Music share since it consists out of an enormous amount of directories and -IF- I access I will also play the files in it, this combined with the fact that the share is on one drive is the reason for me to want to exclude it..

 

Am I making a syntax error ?

[pcjacobse]

Only one.

 

Apparently your version of slackware is using a different version of the "bash" shell (or you are using a different shell entirely) and the

disown %%

command is not recognized.  This would exhibit exactly the symptoms you described.

 

So, if not using "bash", use it.  If using a different version of the "bash" shell, change to one with a working "disown" command.

If your environment does not have the

$SHELL

variable set, set it to /bin/bash before invoking the script.

If your version of "bash" is not in /bin/bash, make a link for it there, or modify the first few lines in the script to where it lives on your system.

 

Joe L.

Thanks for the suggestion!

 

I edited the cache_dirs files and added:

SHELL=/bin/bash

as 2nd line. Now it seems to work correctly.

My ssh shell was /bin/bash, however the boot script (and the simpleFeatures invoker) used /bin/sh as shell.

[bonienl]

I would like to exclude the Music share since it consists out of an enormous amount of directories and -IF- I access I will also play the files in it, this combined with the fact that the share is on one drive is the reason for me to want to exclude it..

 

Am I making a syntax error ?

 

Yes you do

 

Simply give the name "Music", cache_dirs knows it is under /mnt/user...

[Helmonder]

I would like to exclude the Music share since it consists out of an enormous amount of directories and -IF- I access I will also play the files in it, this combined with the fact that the share is on one drive is the reason for me to want to exclude it..

 

Am I making a syntax error ?

 

Yes you do

 

Simply give the name "Music", cache_dirs knows it is under /mnt/user...

 

Thanks !

[dalben]

I run cache_dirs and it's made life a little better.  I was wondering though if there is a way for cache_dirs to trigger a spin up of the disks when I start browsing through the directories.  It's rare for me to be browsing without eventually opening a file which is when the delay kicks in.  It would be great if the disks started up in anticipation of a file being opened.

[xamindar]

I run cache_dirs and it's made life a little better.  I was wondering though if there is a way for cache_dirs to trigger a spin up of the disks when I start browsing through the directories.  It's rare for me to be browsing without eventually opening a file which is when the delay kicks in.  It would be great if the disks started up in anticipation of a file being opened.

Lol you are joking, right? The whole reason for cache_dirs is to prevent disc spinup. Disable cache_dirs and you will have your machine set how you want.

[dalben]

Joking ? No. But if please feel free to explain the humor.

 

If i disable cache_dir then moving through a directory tree will be slow.  Slow enough that my cheapo media player will timeout and I need to start from root again. It's also a matter of having the best of both world.  Fast directory searches witot a lag for disk spin p when a file is needed.

 

Its not often is I start looking for a file without actually opening a file.