Disk Spin Down Delay.....whats a good starting point?.,.

[scottw]

Would we call this the same way WeeboTech??? Can I just add: nohup nice /boot/scripts/cache_user.sh > /dev/null 2>%1 < /dev/null & to the "GO" script?? or should I not nohup it??

 

Thanks,

Scott

 

 

Building on the script below.

 

I call it cache_user.sh

 

This has the benefit of logging to syslog.

Creating a pid file as /var/run/cache_user.pid (or whatever you name it).

 

To stop the daemon, just remove the pid file (or kill the shell, it will eventually remove the pid file and end the loop).

 

One benefit here is you just run it.

It should automatically nohup/disown itself into the background.

 

I may just create an /etc/rc.d/rc.cache_user script and create an installable package to simplify it.

 

One thing that has me curious.

This does an ls -R but I'm not sure it fully caches all the "stat" blocks.

The directory entries are read fully and cached, but the information about the file (type, time, size may not be).

 

 

It may be more effective to use

 

find /mnt/user -ls >/dev/null 2>&1

 

or

find /mnt/user -type d >/dev/null 2>&1

This has the effect of doing a stat on each name to see if it is a directory and only printing directories (which go to null anyway)

 

or do an

 

ls -lR >/dev/null 2>&1

 

correct me if I'm wrong.

 

 

Example of times on my system with various commands.

 

root@unraid /mnt/user #time ls -R /mnt/user >/dev/null 2>&1       

 

real    0m55.372s

user    0m8.390s

sys     0m11.540s

 

root@unraid /mnt/user #time ls -lR /mnt/user >/dev/null 2>&1

 

real    1m4.935s

user    0m13.310s

sys     0m15.350s

 

root@unraid /mnt/user #time find /mnt/user >/dev/null 2>&1         

 

real    0m17.524s

user    0m1.390s

sys     0m3.480s

 

root@unraid /mnt/user #time find /mnt/user -type f >/dev/null 2>&1

 

real    0m17.341s

user    0m1.610s

sys     0m3.020s

root@unraid /mnt/user #time find /mnt/user -type d >/dev/null 2>&1

 

real    0m17.060s

user    0m1.320s

sys     0m2.980s

 

 

 

#!/bin/bash

if [ ${DEBUG:=0} -gt 0 ]
   then set -x -v
fi

P=${0##*/}              # basename of program
R=${0%%$P}              # dirname of program
P=${P%.*}               # strip off after last . character


cache_loop()
{

    echo "$$" > /var/run/${P}.pid
    trap "rm -f /var/run/${P}.pid" EXIT HUP INT QUIT TERM

    logger -is -t${P} "Starting"

    while [ -f /var/run/${P}.pid ]
    do ls -R /mnt/user  >/dev/null 2>&1
       sleep 10
    done

    logger -is -t${P} "Terminating"

}

if [ -f /var/run/${P}.pid ]
   then echo "$0: already running? pidfile: /var/run/${P}.pid"
        ps -fp $(</var/run/cache_user.pid)
        exit
fi

cache_loop > /var/log/${P}.log 2>&1 &
JPID=$!

logger -is -t${P} "Spawned (Pid=$JPID)"
# ps -fp "$JPID"
disown "$JPID"

[WeeboTech]

My version does not need the nohup nice or redirection.

I'm lazy and put it all in the script with the redirection and disown command.

 

[scottw]

Ok I must be doing something wrong here 

 

I have a test Unraid machine here at work and have 2 drives with movies on it. I copied WeeboTech's script over to the root of my flash drive after changing this line:

    do ls -R /mnt/user  >/dev/null 2>&1

 

to this:

    do ls -R /mnt/user/Movies  >/dev/null 2>&1

 

Add:

/boot/ls-r.sh

to the "go" script

 

I reboot the unraid server and once up I can see the 2 drives with activity on them assuming it's the script doing it's magic. Once it's done I spin down all drives.

I pick a movie I know is on Drive2 and call it directly from the run command, not to browse any directories, \\tower\Movies\whatever.mkv. I see it spin up drive1 and start looking on it then after around a minute it spins Drive2 and starts playing  ???

 

Am I doing or assuming something wrong here??

 

Another thing I noticed last night at home is I have drives 2-7 with movies on them. I execute a movie and see it spin up one by one drives 2-7 and look thru them then go back to disk 5 and start playing the movie, why is that???

 

One thing I know is I only have 512mb of ram in the home machine and 1gb in the test box at work....do I need more ram for this to work???

 

Sorry about all of the questions but I am pulling my hair out here.

 

Thanks,

Scott

[NAS]

...

As this is a non-destructive operation and dirty objects are not freeable, the user should run "sync" first!

 

Thats why i didnt post this as its a potential data loss command based on a user remembering to actually do something. It is cool though

 

WeeboTech your solution is far superior. Whats the chances of making this an installable package with a user config stored on flash for timings and paths. We both have been running variations on ls for months now as well as many other members and whilst its not perfect IMO its safe and a fantastic hack for real world use.

[WeeboTech]

WeeboTech your solution is far superior. Whats the chances of making this an installable package with a user config stored on flash for timings and paths. We both have been running variations on ls for months now as well as many other members and whilst its not perfect IMO its safe and a fantastic hack for real world use.

 

I have some ideas.. I'll ponder it s'more and re-post. I think I want to make it an /etc/rc.d/rc.cache_user script.

This way it will have a start and stop command.

external configs, do-able with bash arrays or a simple text file such as

 

/path 10

/path2 20

 

[WeeboTech]

Another thing I noticed last night at home is I have drives 2-7 with movies on them. I execute a movie and see it spin up one by one drives 2-7 and look thru them then go back to disk 5 and start playing the movie, why is that???

 

One thing I know is I only have 512mb of ram in the home machine and 1gb in the test box at work....do I need more ram for this to work???

 

The more ram you have the more you can cache data and directory entry blocks.

it also depends on how many files you have plus how often you select different files.

 

As far as the disks 2-7 spinning up and selecting disk 5.. I have the same issue. however my machine has 8GB but is busy with torrents.. Before that I do not remember this issue cropping up.

 

 

[NAS]

WeeboTech your solution is far superior. Whats the chances of making this an installable package with a user config stored on flash for timings and paths. We both have been running variations on ls for months now as well as many other members and whilst its not perfect IMO its safe and a fantastic hack for real world use.

 

I have some ideas.. I'll ponder it s'more and re-post. I think I want to make it an /etc/rc.d/rc.cache_user script.

This way it will have a start and stop command.

external configs, do-able with bash arrays or a simple text file such as

 

/path 10

/path2 20

 

 

Sounds good.

 

One thing thats worth pondering is  the ability for the daemon to detect that the cache is shrinking and drop the ls timings. Counters in /PROC are not static in so far as they fluctuate even when nothing is happening but this fluctuation is a fractions of a percent. In theory if the relevant cache counter dropped by say 10% between normal runs the ls scipt could kick into overdrive trying to force the cache from dropping from RAM.

 

perhaps that is a step to far and its unclear if it would actually work but its worth pondering.

 

Alternatively dropping the sleep time to 1 or even 0 seconds might be viable. Once the cache is in memory the actual resources used from then on is negligible so of the top of my head i cant think of a downside to just constantly running ls. All this solution is an inelegant hack anyway until the kernel peeps give us something better.

[scottw]

Another thing I noticed last night at home is I have drives 2-7 with movies on them. I execute a movie and see it spin up one by one drives 2-7 and look thru them then go back to disk 5 and start playing the movie, why is that???

 

One thing I know is I only have 512mb of ram in the home machine and 1gb in the test box at work....do I need more ram for this to work???

 

The more ram you have the more you can cache data and directory entry blocks.

it also depends on how many files you have plus how often you select different files.

 

As far as the disks 2-7 spinning up and selecting disk 5.. I have the same issue. however my machine has 8GB but is busy with torrents.. Before that I do not remember this issue cropping up.

 

 

 

Ok I will upgrade the ram to see if that helps!!!! I would hate to have to leave the drives spinning all the time.

[WeeboTech]

Ok I will upgrade the ram to see if that helps!!!! I would hate to have to leave the drives spinning all the time.

 

The drives will eventually spin down. It's just the initial read that causes them to spin up. Once the data is found. The unused drives spin down according to the set timing.

 

[WeeboTech]

Counters in /PROC are not static in so far as they fluctuate even when nothing is happening but this fluctuation is a fractions of a percent.

 

What values in /proc are you talking about?

 

 

[scottw]

Ok I will upgrade the ram to see if that helps!!!! I would hate to have to leave the drives spinning all the time.

 

The drives will eventually spin down. It's just the initial read that causes them to spin up. Once the data is found. The unused drives spin down according to the set timing.

 

 

Yeah I know...it was the initial spin up that was bothering me. It would be alot faster if it only had to spin one drive.

[NAS]

Counters in /PROC are not static in so far as they fluctuate even when nothing is happening but this fluctuation is a fractions of a percent.

 

What values in /proc are you talking about?

 

 

 

Some more research will need done but off the top of my head:

 

/proc/slabinfo

 

dentry

and possibly one of the inode one like reiser_inode_cache

 

[nsta]

Hey Weebotech, your information is very useful....id like to try this myself, and see what sort of real-world improvements i get....

 

Could you confirm im doing everything correctly step by step....im still real linux/command line iliterate!

 

1) First create a "cache.sh" script, and have the following contained within it:

 

#!/bin/bash

 

if [ ${DEBUG:=0} -gt 0 ]

   then set -x -v

fi

 

P=${0##*/}              # basename of program

R=${0%%$P}              # dirname of program

P=${P%.*}               # strip off after last . character

 

 

cache_loop()

{

 

    echo "$$" > /var/run/${P}.pid

    trap "rm -f /var/run/${P}.pid" EXIT HUP INT QUIT TERM

 

    logger -is -t${P} "Starting"

 

    while [ -f /var/run/${P}.pid ]

    do ls -R /mnt/user  >/dev/null 2>&1

       sleep 10

    done

 

    logger -is -t${P} "Terminating"

 

}

 

if [ -f /var/run/${P}.pid ]

   then echo "$0: already running? pidfile: /var/run/${P}.pid"

        ps -fp $(</var/run/cache_user.pid)

        exit

fi

 

cache_loop > /var/log/${P}.log 2>&1 &

JPID=$!

 

logger -is -t${P} "Spawned (Pid=$JPID)"

# ps -fp "$JPID"

disown "$JPID"

 

 

3) I then place this script in /boot, i'm assuming the cached_user.pid is created automatically?.....and do i have to rename "user" in the line below, to the name of my user share (ie Movies) or not?

 

    do ls -R /mnt/user  >/dev/null 2>&1

 

4) I then add this to my go script:

 

     /boot/cache.sh

 

 

Thanks in advise?

 

 

[scottw]

Hey Weebotech, your information is very useful....id like to try this myself, and see what sort of real-world improvements i get....

 

Could you confirm im doing everything correctly step by step....im still real linux/command line iliterate!

 

1) First create a "cache.sh" script, and have the following contained within it:

 

#!/bin/bash

 

if [ ${DEBUG:=0} -gt 0 ]

   then set -x -v

fi

 

P=${0##*/}              # basename of program

R=${0%%$P}              # dirname of program

P=${P%.*}               # strip off after last . character

 

 

cache_loop()

{

 

    echo "$$" > /var/run/${P}.pid

    trap "rm -f /var/run/${P}.pid" EXIT HUP INT QUIT TERM

 

    logger -is -t${P} "Starting"

 

    while [ -f /var/run/${P}.pid ]

    do ls -R /mnt/user  >/dev/null 2>&1

       sleep 10

    done

 

    logger -is -t${P} "Terminating"

 

}

 

if [ -f /var/run/${P}.pid ]

   then echo "$0: already running? pidfile: /var/run/${P}.pid"

        ps -fp $(</var/run/cache_user.pid)

        exit

fi

 

cache_loop > /var/log/${P}.log 2>&1 &

JPID=$!

 

logger -is -t${P} "Spawned (Pid=$JPID)"

# ps -fp "$JPID"

disown "$JPID"

 

 

3) I then place this script in /boot, i'm assuming the cached_user.pid is created automatically?.....and do i have to rename "user" in the line below, to the name of my user share (ie Movies) or not?

 

    do ls -R /mnt/user  >/dev/null 2>&1

 

4) I then add this to my go script:

 

     /boot/cache.sh

 

 

Thanks in advise?

 

 

 

From what I can tell:

1. Yes

2. There is no 2 

3.Just place it in the root of your flash drive..that is /boot when you telnet in to the unraid server. Yes the .pid is created automatically.

I think you can just leave this:

    do ls -R /mnt/user  >/dev/null 2>&1

the way it is or you can add your share name after user:

    do ls -R /mnt/user/Movies  >/dev/null 2>&1 not positive on that though.

4. Yes

 

All of the above is from my experience which is low and what I have done. Please anybody correct me if I am wrong...I don't want to give out the wrong info.

 

Hope this helps.

[WeeboTech]

Could you confirm im doing everything correctly step by step....im still real linux/command line iliterate!

 

1) First create a "cache.sh" script, and have the following contained within it:

 

3) I then place this script in /boot, i'm assuming the cached_user.pid is created automatically?.....and do i have to rename "user" in the line below, to the name of my user share (ie Movies) or not?

 

    do ls -R /mnt/user  >/dev/null 2>&1

 

4) I then add this to my go script:

 

     /boot/cache.sh

 

 

 

From what I can tell:

1. Yes

2. There is no 2 

3.Just place it in the root of your flash drive..that is /boot when you telnet in to the unraid server. Yes the .pid is created automatically.

I think you can just leave this:

    do ls -R /mnt/user  >/dev/null 2>&1

the way it is or you can add your share name after user:

    do ls -R /mnt/user/Movies  >/dev/null 2>&1 not positive on that though.

4. Yes

 

All of the above is from my experience which is low and what I have done. Please anybody correct me if I am wrong...I don't want to give out the wrong info.

 

Hope this helps.

 

Thanks scott.

 

I usually create a /boot/custom/bin and put my executable scripts there.

 

mkdir -p /boot/custom/bin

 

put the cache_user.sh script there.

 

Make sure it is executable with

 

chmod a+rx /boot/custom/bin/cache_user.sh

 

put an entry in your go script

/boot/custom/bin/cache_user.sh

 

To stop it from running

kill `cat /var/run/cache_user.pid`

rm /var/run/cache_user.pid

 

wait for the sleep cycle.

 

This script will use whatever name you call it for the pid file.

So if you call it cache.sh then the /var/run file will be /var/run/cache.pid

 

I'm working on a new version which will handle things better.

 

 

[nsta]

Thanks - So much help here in these forums its awesome:)

 

Im going to give it a try tonight...

 

Few quick q's:

 

1) Before i have the script running, id like to do some quick tests, before and after....is there an easy way to do this (via command line)?

 

2) I have disk 1 and 2 that spans between one user share (movies), and i have disk3 thats solely used for data, which i transfer directly to (dont use user shares for disk3).....do i need to add another command to enable the ls-r hack for the disk3?

 

3) would there be a reason to kill PID....Thanks:)

 

[scottw]

Thanks - So much help here in these forums its awesome:)

 

Im going to give it a try tonight...

 

Few quick q's:

 

1) Before i have the script running, id like to do some quick tests, before and after....is there an easy way to do this (via command line)?

 

2) I have disk 1 and 2 that spans between one user share (movies), and i have disk3 thats solely used for data, which i transfer directly to (dont use user shares for disk3).....do i need to add another command to enable the ls-r hack for the disk3?

 

3) would there be a reason to kill PID....Thanks:)

 

 

Sorry but I will let the experts handle these questions but I have a question(s) of my own.

 

It looks like this will run every 10sec. and I assume this will not spin up disks to run the ls-r right, I guess that would defeat the purpose??

 

I understand having this run on startup since all of the disks will be spun up by default and it will be able to cache all of the entries, but how does it work once the disks are spun down and then say one disk is spun up while watching a movie, how does it refresh the directory entries when the other disks are spun down.

 

Sorry for being such a NEWBIE at this but if I understand it a little more it would be helpful for me.

 

 

Thanks!!!

Scott

[WeeboTech]

 

It looks like this will run every 10sec. and I assume this will not spin up disks to run the ls-r right, I guess that would defeat the purpose??

 

The assumption is incorrect. the purpose is to "try" to keep directory entries in the dentry cache by making them referenced as often as possible.

If the system is starved for ram, it still may free unreferenced dentry blocks, which later may cause a drive to spin up.

It all depends on how much ram you have and how often you do operations that require large amounts of ram.

 

I understand having this run on startup since all of the disks will be spun up by default and it will be able to cache all of the entries, but how does it work once the disks are spun down and then say one disk is spun up while watching a movie, how does it refresh the directory entries when the other disks are spun down.

 

If the dentry blocks are already in ram. then the dentry cache supplies the data rather then reading from disk and populating the dentry_cache.

Again, by referencing the dentry blocks often, it diminishes the chance they will be freed/removed from the cache.

 

[WeeboTech]

1) Before i have the script running, id like to do some quick tests, before and after....is there an easy way to do this (via command line)?

Not an easy way.

Fire up the script. Press the SPIN DOWN button.  Do a test on the drive or share via SAMBA.

 

2) I have disk 1 and 2 that spans between one user share (movies), and i have disk3 thats solely used for data, which i transfer directly to (dont use user shares for disk3).....do i need to add another command to enable the ls-r hack for the disk3?

 

if you want to cache the two directories, you will need to add another ls -r for disk3

 

3) would there be a reason to kill PID....Thanks:)

 

if you wanted to add a new directory.

 

[NAS]

Rather than ls the user directory it might be more sensible to ls each drive individually. Discovering what drives are in place is easy enough.

 

Also since i moved to doing just one ls and the big user share my disks "seem to" spin up more. I may be imagining it but i could swear they spin up less doing multiple ls's

 

Edit: my search foo escapes me. Since unRAID doesnt have dos2unix whats the command to strip ^M etc again?

[WeeboTech]

Edit: my search foo escapes me. Since unRAID doesnt have dos2unix whats the command to strip ^M etc again?

 

fromdos & todos

[NAS]

Excellent thanks. A quick test and it should be fairly trivial to combine wget, fromdos into a script that can grab stuff from pastebin by id.

 

This would make it easy for users to grab sample scripts direct to their unRAID server. Might be another idea that goes no where but then again

 

 

pastegrab 1245 /boot.scriptscache.sh

 

sorta thing

 

[Joe L.]

Edit: my search foo escapes me. Since unRAID doesnt have dos2unix whats the command to strip ^M etc again?

http://lime-technology.com/wiki/index.php?title=FAQ#Why_do_my_scripts_have_problems_with_end-of-lines.3F

[scottw]

 

It looks like this will run every 10sec. and I assume this will not spin up disks to run the ls-r right, I guess that would defeat the purpose??

 

The assumption is incorrect. the purpose is to "try" to keep directory entries in the dentry cache by making them referenced as often as possible.

If the system is starved for ram, it still may free unreferenced dentry blocks, which later may cause a drive to spin up.

It all depends on how much ram you have and how often you do operations that require large amounts of ram.

 

I understand having this run on startup since all of the disks will be spun up by default and it will be able to cache all of the entries, but how does it work once the disks are spun down and then say one disk is spun up while watching a movie, how does it refresh the directory entries when the other disks are spun down.

 

If the dentry blocks are already in ram. then the dentry cache supplies the data rather then reading from disk and populating the dentry_cache.

Again, by referencing the dentry blocks often, it diminishes the chance they will be freed/removed from the cache.

 

 

Thanks for taking the time to answer WeeboTech. I just ordered 4gb of ram from NewEgg yesterday. I will install that when it comes in and setup your script....thanks!!!!

[NAS]

Proof of concept only

 

#!/bin/bash

pastebinid=$S1
nametocallfile=$2

echo "Using Pastebin ID: $1"
echo "Saving File as: $2"

wget http://pastebin.com/pastebin.php?dl=$1 -O $1.tmp
fromdos  <$1.tmp  >$2
rm $1.tmp

 

Call like:

 

pbgrab.sh PastebinID FileToSAveAs

 

e.g.

 

pbgrab.sh f709ba57c cached.sh

 

 

would grab WeeboTech's script and save it locally.

 

Could be a very simple way to distribute scripts rather than all this copy and paste from forum milarky.

 

Any for comment only.

 

 

 

Where