cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up

[Joe L.]

I am getting the "duplicate objects" log entries whenever mover is running.  As a test I stopped cache_dirs and ran the mover manually and did not get the "duplicate object" log entries.

 

I can't say for sure that cache_dirs is causing these entries, but it looks to be the cuplrit.  Is there a way to confirm this?  If so, then what steps can I take to stop the log entries?  I have checked and I am running the latest version of cache_dirs.

 

There are a lot of these log entries at night when the mover is running.  Seems like a lot of needless entries in the log.

Any time you list the contents of the shares with the duplicate entries you will get the "duplicate object" messages in the log file if they exist.

 

Basically, all cache_dirs is doing, is repeatedly listing the directories, to keep them in the most recently used disk buffer cache (or, more accurately, to keep them from being the least recently used and freed for other use.).  To get rid of the duplicate object log entries, get rid of the duplicate objects.

 

Joe L.

[dlandon]

If I read you correctly, you are saying that I actually have duplicate objects?  As far as I know I don't.  I ran a dupe check and came up with nothing.

 

These log entries only come up when "mover" is running.

 

I stopped cache_dirs yesterday because I am running a pre_clear on a new disk.  I stopped it because I didn't want to slow down the pre_clear.  Last night I did not get any "duplicate objects" when mover was running.

 

As a side note, I have 8Gb of memory.  I don't know if this means anything.

 

 

[Joe L.]

If I read you correctly, you are saying that I actually have duplicate objects?  As far as I know I don't.  I ran a dupe check and came up with nothing.

 

These log entries only come up when "mover" is running.

 

I stopped cache_dirs yesterday because I am running a pre_clear on a new disk.  I stopped it because I didn't want to slow down the pre_clear.  Last night I did not get any "duplicate objects" when mover was running.

 

As a side note, I have 8Gb of memory.  I don't know if this means anything.

 

 

OK.... I mis-understood.  (or rather, I did not notice you said they only occurred when your "mover" was moving files to the protected array)

 

While the file is being copied, it does exist on two disks...  (the cache drive AND the parity protected data disk it is being copied to)

In my opinion, the error message in the log should not occur under that move, and is a bug if it is occurring in reporting dupes. (that type of "duplicate object" should be expected and ignored while a file is being moved.  No syslog entries should be made for them.)

 

Please send a PM to limetech, or an email to [email protected] and report the bug.

 

Joe L.

 

[Eisi2005]

@Joe L.

 

thx for your help.

 

Greets

Eisi

 

[dlandon]

While the file is being copied, it does exist on two disks...  (the cache drive AND the parity protected data disk it is being copied to)

In my opinion, the error message in the log should not occur under that move, and is a bug if it is occurring in reporting dupes. (that type of "duplicate object" should be expected and ignored while a file is being moved.  No syslog entries should be made for them.)

 

Joe,

 

I don't see this as an unraid issue.  I don't think I am making myself clear.  Let me try again.

 

You have logic in the cache_dirs script to stop caching when 'mover' is running.  I don't believe that logic is stopping the caching in my situation.  Whenever cache_dirs is running on my setup and the 'mover' script runs, I get the "duplicate object" log entries.  I get them at night when 'mover' runs, and I get them also when I execute 'mover' manually.

 

If I stop cache_dirs (cache_dirs -q), the log entries stop when 'mover' runs.

 

Is there anything I can do to troubleshoot this?

[Joe L.]

While the file is being copied, it does exist on two disks...  (the cache drive AND the parity protected data disk it is being copied to)

In my opinion, the error message in the log should not occur under that move, and is a bug if it is occurring in reporting dupes. (that type of "duplicate object" should be expected and ignored while a file is being moved.  No syslog entries should be made for them.)

 

Joe,

 

I don't see this as an unraid issue.  I don't think I am making myself clear.  Let me try again.

 

You have logic in the cache_dirs script to stop caching when 'mover' is running.  I don't believe that logic is stopping the caching in my situation.  Whenever cache_dirs is running on my setup and the 'mover' script runs, I get the "duplicate object" log entries.  I get them at night when 'mover' runs, and I get them also when I execute 'mover' manually.

 

If I stop cache_dirs (cache_dirs -q), the log entries stop when 'mover' runs.

 

Is there anything I can do to troubleshoot this?

Looking in the code.    

 

It is basically looking once per loop for the file /var/run/mover.pid

and if the process whose PID is in it is running, will wait until it is gone.

 

If the mover starts while cache_dirs is in the middle of a scan, I can see how the duplicate object messages can occur.

(They are sill a bug in unRAID... the log messages should not occur while mover is moving a file from cache)

The messages should not continue though... not once it finishes a scan and loops where it will again test for the /var/run/mover.pid file..

 

However, the real cause was reported recently... http://lime-technology.com/forum/index.php?topic=4500.msg132162#msg132162

I was a typo...

 

Corrected script now attached to the first post in this thread.

 

Joe L.

[dlandon]

Corrected script now attached to the first post in this thread.

 

Got it.

 

Thanks Joe.

[cartbaby]

When I browse my share from windows, unraid will spin up a drive.  The cache script is running.  what could cause the spin up?  I recently added 2 new drives and changed ram.

[Joe L.]

When I browse my share from windows, unraid will spin up a drive.  The cache script is running.  what could cause the spin up?  I recently added 2 new drives and changed ram.

Do you have windows set to show icons?  That would mean windows is accessing the contents of the file, not just the directory entries.

 

Joe L.

[Subscript]

Joe L.

 

I installed cache_dirs for the first time after your 1.6.6 update.  However, the link on the first page is downloading 1.6.5 and I've tried it twice (2 days apart).  Where can I get 1.6.6?

 

This is from my syslog:

 

Sep 26 23:19:19 Tower cache_dirs: ==============================================

Sep 26 23:19:19 Tower cache_dirs: command-args=

Sep 26 23:19:19 Tower cache_dirs: vfs_cache_pressure=10

Sep 26 23:19:19 Tower cache_dirs: max_seconds=10, min_seconds=1

Sep 26 23:19:19 Tower cache_dirs: max_depth=9999

Sep 26 23:19:19 Tower cache_dirs: command=find -noleaf

Sep 26 23:19:19 Tower cache_dirs: version=1.6.5

Sep 26 23:19:19 Tower cache_dirs: ---------- caching directories ---------------

Sep 26 23:19:19 Tower cache_dirs: Media

Sep 26 23:19:19 Tower cache_dirs: ----------------------------------------------

Sep 26 23:19:19 Tower cache_dirs: cache_dirs process ID 26951 started, To terminate it, type: cache_dirs -q

 

Thank you for your work,

 

Subscript

[dlandon]

I installed cache_dirs for the first time after your 1.6.6 update.  However, the link on the first page is downloading 1.6.5 and I've tried it twice (2 days apart).  Where can I get 1.6.6?

 

That is the latest version.  Joe didn't change the version message in the script.

[Fluid]

Hey Joe, I've been having some quirks on my box and I wanted to troubleshoot a few things a step at a time.  If I want to disable this script and put my unraid box back into it's default state, I can just remove it from the go script right?  There are no cached temp files I need to worry about deleting?

[Joe L.]

Hey Joe, I've been having some quirks on my box and I wanted to troubleshoot a few things a step at a time.  If I want to disable this script and put my unraid box back into it's default state, I can just remove it from the go script right?  There are no cached temp files I need to worry about deleting?

it is all done in memory... so no files to delete to disable it.

 

You can put a "#" at the start of the line in the "go" script to comment out the line from being executed.

[Lunatic]

is there a way to NOT receive an email when Cache_Dirs starts?

 

I get an email with a subject line "Output from your job 3?" and body message "cache_dirs process ID 5140 started, To terminate it, type: cache_dirs -q"

 

I don't see the value of this email. Maybe if it contained the info on what is being cached or when it completed. Thoughts?

 

Thanks

 

[Joe L.]

is there a way to NOT receive an email when Cache_Dirs starts?

 

I get an email with a subject line "Output from your job 3?" and body message "cache_dirs process ID 5140 started, To terminate it, type: cache_dirs -q"

 

I don't see the value of this email. Maybe if it contained the info on what is being cached or when it completed. Thoughts?

 

Thanks

 

don't start it using "at". 

In other words, it is queuing itself to be started later since when you are starting it the array itself is not yet up and running.

[Lunatic]

is there a way to NOT receive an email when Cache_Dirs starts?

 

I get an email with a subject line "Output from your job 3?" and body message "cache_dirs process ID 5140 started, To terminate it, type: cache_dirs -q"

 

I don't see the value of this email. Maybe if it contained the info on what is being cached or when it completed. Thoughts?

 

Thanks

 

don't start it using "at". 

In other words, it is queuing itself to be started later since when you are starting it the array itself is not yet up and running.

 

 

awesome!!

 

thanks Joe

 

[tyrindor]

Joe does this look right to you?

/boot/cache_dirs -w -d 2 -e "cache" -a '-noleaf -name Jukebox -prune -o -print'

 

What I want to happen:

Caches all my shares, two directories deep. I assume Tower/Media/Movies is 2 depth? Media is the share name. I want it to cache the contents of movies folder, but no folders inside of it.

Ignores the Jukebox subfolder in one of my shares.

Ignores my cache drive since it's always spun up, it doesn't need to be cached.

 

Thanks.

[kapperz]

Hi Joe,

 

I'm trying to track down a problem with some software I'm using where new directories are not being scanned. Anyhow, I think it might be a cache issue (browser, application files). I love your script, but I have a question.

 

Where are the cache files saved? Just killing cache_dirs.sh doesn't fix my problem and I'm not sure what your script is doing. I guess I can start the server without your script ever started and copy new directories over to eliminate cache_dirs, but I want to use cache_dir. It could be an issue of completely refreshing the cache of dirs like the server was rebooted (which doesn't seem to fix it every time).

 

thanks for your input!

[peter_sm]

Hi Joe!

 

is this normal 

 

cache_dirs -w -d 4 -i  "Movie" -F
Executed find in 0.098190 seconds, weighted avg=0.098190 seconds, now sleeping 5 seconds
Executed find in 0.089682 seconds, weighted avg=0.092518 seconds, now sleeping 6 seconds
Executed find in 0.089436 seconds, weighted avg=0.090977 seconds, now sleeping 7 seconds
Executed find in 0.089534 seconds, weighted avg=0.090400 seconds, now sleeping 8 seconds
Executed find in 0.089415 seconds, weighted avg=0.090071 seconds, now sleeping 9 seconds

[Joe L.]

Hi Joe!

 

is this normal 

 

cache_dirs -w -d 4 -i  "Movie" -F
Executed find in 0.098190 seconds, weighted avg=0.098190 seconds, now sleeping 5 seconds
Executed find in 0.089682 seconds, weighted avg=0.092518 seconds, now sleeping 6 seconds
Executed find in 0.089436 seconds, weighted avg=0.090977 seconds, now sleeping 7 seconds
Executed find in 0.089534 seconds, weighted avg=0.090400 seconds, now sleeping 8 seconds
Executed find in 0.089415 seconds, weighted avg=0.090071 seconds, now sleeping 9 seconds

If everything is in RAM, yes.  There is no need for the OS to to read the directories from the physical disk, since they are in the linux buffer cache, so each "find" command is very fast.

[tyrindor]

After installing benni-chan's tranmission plugin cache_dirs no longer starts/works. No errors that I can see. Nothing about cache_dirs in the log, even though it's still in my go script. http://lime-technology.com/forum/index.php?topic=17606.0

 

My go file:

#!/bin/bash

# Start the Management Utility

/usr/local/sbin/emhttp &

installpkg /boot/packages/simpleFeatures-0.9b-unraid-speeding_ant.tgz

echo nameserver 192.168.0.1 >/etc/resolv.conf

echo 192.168.0.102 UNRAID >>/etc/hosts

sleep 30; blockdev --setra 2048 /dev/md*

/boot/cache_dirs -w

 

Tried moving cache_dirs above simplefeature, same thing. Manually starting cache_dirs after transmission has launched (with the same "/boot/cache_dirs -w" command) starts it, and it starts to cache.. but it never finishes and it loses it's cached data very fast.

[MortenSchmidt]

The other day, I was sorting some media, and somehow ended up with 5 duplicate files. This resulted in the the logger crashing and restarting less than a day after I had 'organized' those files. Looks to be because cachedirs keeps scanning the duplicated files, and each one is reported to syslog each time cache_dirs runs. Would have liket to attach a syslog, but... :-)

 

Is it likely the cause is as I have described, and is there any way to avoid this (other than avoid creating duplicate files..) ?

 

If not, can we talk about an extension to unrad_notify that would send an email when syslog gets over a certain size (or RAM filesystem is short on space) ?

[jeff.lebowski]

It appears as though all disks are spinning up when I open a directory. This did not occur yesterday, before my internet was down for a few hours. Side note: If any piece of network hardware is rebooted, I lose connectivity to both servers. Odd.

 

I use Total Commander in Source: Only file names mode. That means no thumbnails.

 

Anyhoo, script from the go file: /boot/cache_dirs -d 5 -m 3 -M 5 -w. Looks fine to me, and from other examples I've seen.

 

What do you think is going on?

 

syslog-2012-02-15.txt

[Joe L.]

It appears as though all disks are spinning up when I open a directory. This did not occur yesterday, before my internet was down for a few hours. Side note: If any piece of network hardware is rebooted, I lose connectivity to both servers. Odd.

 

I use Total Commander in Source: Only file names mode. That means no thumbnails.

 

Anyhoo, script from the go file: /boot/cache_dirs -d 5 -m 3 -M 5 -w. Looks fine to me, and from other examples I've seen.

 

What do you think is going on?

It means the blocks needed are not in memory so it has to go to the disks.

 

You can always install "inotify tools" and then:

To track activity under /mnt/user, type:inotifywait -mr /mnt/user

To track activity on a specific disk (/mnt/disk1), type:inotifywait -mr /mnt/disk1

they will show you exactly what is being accessed.

[settings]

This is a great script.

Beforehand if I went to browse my movies on my HTPC it would spin up the drives even if I changed my mind and watched tv off another drive (even though the xmbc database was saved on the htpc). Now it is much better and doesnt have any negative effects on my setup.

I take it its just the raid that is cached not the cache drive?

 

I did use 'cache_dirs -w' to start and it spun up the drives without waiting. Great scripts to have though, thanks!