User share and power saving

[WeeboTech]

And to my mind, any 'fragility' to something so basic as reading a folder directory just does not make sense.

 

Agreed, if frequent ls -lR can cause your system or disk to hang or go offline, there is a larger problem here.

Possibly wiring, controller or bus contention. It could also be a spin up issue or timeout. I.E. Drive taking too long to spin up.

 

I beat the hell out of my unRAID server. It begs for mercy!!!! LOL.and I've never had a disk go offline that was not a physical issue.

 

The only time a disk caused problems was when the wire became slightly loose due to the vibration (the downside of the removables I selected). This was over 4 months ago and has not happened since.

 

Before this weekend I had an uptime of over 100 days.

[yodine]

:'( :'( :'(

Ok, so i made the step and installed the script. Well first it didn't work because the ls script has to be converted to Unix.

 

After this i got no error at boot, but doing a "crontab -l" didn't show the frequent ls lines. I then tried to run ls-R manualy from telnet, it went well and i could see the entry in the crontab. So the script seems to work when run manualy, but not when run from the go script.

 

Does someone knows why at boot it didn't show ? Is the crontab build after some time by unraid, thus erasing the crontab created from the script ?

 

Here is my complete go script :

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp &

# Set cron job for ls -R task, keep directories cached
/boot/ls-R

# Increase the directory caching
echo vm.vfs_cache_pressure = 0 >>/etc/sysctl.conf
sysctl -p /etc/sysctl.conf

 

Any idea would be much apreciated

[RobJ]

One possibility, I noticed that when I created ls-R on my Windows machine (and set end-of-lines to be Unix-style), then copied it to the flash while it was installed in my unRAID server, the file name became ls-r, don't recall why just now.  So I run it as /boot/ls-r, because Linux is case-sensitive.

[Joe L.]

Does someone knows why at boot it didn't show ? Is the crontab build after some time by unraid, thus erasing the crontab created from the script ?

Exactly.

 

here are the lines in my syslog showing how crontab is established, just before starting the unRAID array.

[pre]Aug 23 19:01:47 Tower emhttp: shcmd (18): cp /var/spool/cron/crontabs/root- /var/spool/cron/crontabs/root

Aug 23 19:01:47 Tower emhttp: shcmd (19): echo '# Generated mover schedule:' >>/var/spool/cron/crontabs/root

Aug 23 19:01:47 Tower emhttp: shcmd (20): echo '40 3 * * * /usr/local/sbin/mover 2>&1 | logger' >>/var/spool/cron/crontabs/root

Aug 23 19:01:47 Tower emhttp: shcmd (21): crontab /var/spool/cron/crontabs/root

Aug 23 19:01:47 Tower emhttp: shcmd (22): killall -w smbd nmbd

Aug 23 19:01:48 Tower emhttp: shcmd (23): /usr/sbin/nmbd -D

Aug 23 19:01:48 Tower emhttp: shcmd (24): /usr/sbin/smbd -D

Aug 23 19:01:48 Tower kernel: mdcmd (3): start

[/pre]

 

You might try a

sleep 30

in your "go" script, before invoking the script you created to change the cron schedule.

 

Joe L.

[yodine]

You might try a

sleep 30

in your "go" script, before invoking the script you created to change the cron schedule.

 

Thanks a lot, i added the "sleep 30" just before the call of ls-R. I'll try to remember the next time a reboot the server to check the crontab

 

In the end, it wasn't an easy task to just do a simple ls 

[scottw]

If I may jump in here and ask a question 

 

Does this only help not spinning up all drives while browsing???

 

I have a HTPC frontend that calls the movies directly from the share (//tower/Movies/Spiderman.mpv)...there is no browsing. I can't remember if it spins up all the drives when a movie starts or just the drive it is on. I remember looking at it after I started a movie and all drives were spun but the parity but I don't remember if I browsed the file structure somewhere in there before when I was testing something.

 

I guess my questions are:

 

1) Should just calling a movie directly from a share, not thru Disk1 Disk2 etc, spin up all drives even though I did not browse the file structure?

2) If so would calling the movie thru it's appropriate disk fix that??

 

As I said I am not positive this is even happening, I will check when I get home. I just saw this post and got curious  

 

Thanks,

[WeeboTech]

I guess my questions are:

 

1) Should just calling a movie directly from a share, not thru Disk1 Disk2 etc, spin up all drives even though I did not browse the file structure?

2) If so would calling the movie thru it's appropriate disk fix that??

 

1. Should is not the proper word here.. It is more of a could.. and the answer is Yes accessing a file by direct name on a user share could cause multiple drives to spin up.  Even though the client may know the name, the machine itself has to find the location of the name. This in turn causes the user share daemon to search tables & open directories thus in turn spinning up drives.

This is why caching the directory entries is important. If the directory entries are in ram, then a search for where the file lives is satisfied by accessing structures in ram.

 

2. If you know a specific movie is on a specific disk, then accessing it via the disk share would only spin up the associated drive.

[scottw]

Thanks WeeboTech that makes sense.

 

I would prefer to just call the movie from the share so it looks like I will have to try out these scripts

 

How important is memory here?? I am running the minimum of 512 at the moment and plan to upgrade it eventually but is that not enough to use the caching until I can get some more?

 

Thanks,

[WeeboTech]

The more memory, the right kernel tuning and the right tools would keep as much dentry (directory entry) items in ram as possible.

 

I suppose you only need this if the spin up issue is making you uncomfortable.

it all depends on your usage patterns.

 

> but is that not enough to use the caching until I can get some more?

It's the minimal amount and it's based on how many files, disks and your usage pattern.

If a compressed movie is from 350MB to 700MB, you can see how your cache would be flushed.

 

you can always add the sysctl option to your go script to help the process along.

 

sysctl vm.vfs_cache_pressure=0

[scottw]

Thanks. I will do some testing to see if this is even necessary.

 

 

Thanks,

[scottw]

Three more questions  :

 

 

1) Will the addition of a cache drive change the frequency of when this should be run???

 

2) So far I hear some people running it every 10 seconds..others every 30, what would be recommended??

 

3) And finally from what I can tell is if you pick the wrong frequency to run it the worst that will happen is it will spin up all the drives, right?? I guess from my stand point is if I don't run this I will probably get the same result of most if not all drives spinning up anyway since I will be playing movies directly from the shares So I would have nothing to lose, it can only make things better...if I am right.

 

Thanks for all the help!!!!

[WeeboTech]

1) Will the addition of a cache drive change the frequency of when this should be run???

No.

 

2) So far I hear some people running it every 10 seconds..others every 30, what would be recommended??

At one time every 30 seconds was enough for me.

as I added more files, and my system got busier I changed it to every 10 seconds.

There came a point where it was counter productive (due to my system being so busy) so I shut it off.

 

3) And finally from what I can tell is if you pick the wrong frequency to run it the worst that will happen is it will spin up all the drives, right?? I guess from my stand point is if I don't run this I will probably get the same result of most if not all drives spinning up anyway since I will be playing movies directly from the shares So I would have nothing to lose, it can only make things better...if I am right.

Yes, It also depends on how busy the system is.

If the system is really busy and you run it too often then it will keep the drives spinning all the time.

 

As a general media portal server, you should be fine.

If you start to run your torrents on the server and you have allot of them, then it becomes counter productive.

 

[scottw]

Great thanks for all of your answers. Yeah this will mainly just be for watching movies. Most of the time the drives should be spun down...which is why this would be so effective for me.

 

So I will set it for 10sec and go from there.

 

 

Thanks alot!!!!!

[Joe L.]

Three more questions  :

 

 

1) Will the addition of a cache drive change the frequency of when this should be run???

No, not that I can think of.

2) So far I hear some people running it every 10 seconds..others every 30, what would be recommended??

Blocks in the disk buffer cache are released for re-use based on a "least-recently-used" rule.  If the blocks representing your directory entries accessed by your media player are in memory, and accessed, they, at that moment, will be the most-recently-used blocks.  If the directory listing can be satisfied by the disk blocks in memory, there will be no need to spin up the disk to read them (at that point, they will be in memory once more, until they become the least-recently-used blocks and the memory is needed for some other purpose.)

 

We all have different hardware (the amount of RAM) and different usage profiles of our servers.

If you had an infinite amount of RAM, and the Linux kernel did not have to release the least-recently-used block when it needed memory for something else, then it would read the disks once... ever, or at least until you powered down and the contents of RAM was lost.

 

I only have 512Meg of RAM.  My usage is that only one or two ISO images are ever streamed to media players.  During that process, over the course of 2 hours or so, I can be accessing 8Gig of data.

 

Doing some crude math, this seems to say that in 10 seconds, I will read somewhere around 11 meg of data from my disks.  Since my disk cache is much bigger than that, my directory entries should not be displaced (they should never end up as the least-recently-used blocks.  Since I perform my ls -R every 10 seconds.)  Eventually, the blocks of data I played on my media player 10 minutes or so ago will be the least recently used.  If I skipped back a chapter, they would need to be re-read from the physical disk.

 

Now, if I can simply copying from disk to disk an ISO file the transfer rate is not as slow as when I am watching a movie.  In that situation, I can still keep up with most anything.  (my IDE disks are not as fast as SATA disks)

 

If you have 4 Gig of ram, it takes a LOT longer to end up as the least-recently-used block of disk cached data.  You can then afford to do the "ls -R" less frequently.

 

3) And finally from what I can tell is if you pick the wrong frequency to run it the worst that will happen is it will spin up all the drives, right??

No, not all the drives, just those where the directory blocks are no longer in memory.  And, as long as nothing displaces them (needs memory when they are the least-recently-used) they will be there for the next ls-R.

I guess from my stand point is if I don't run this I will probably get the same result of most if not all drives spinning up anyway since I will be playing movies directly from the shares So I would have nothing to lose, it can only make things better...if I am right.

Unless you have a huge amount of RAM, and tiny movies, you will almost always be playing your movies by reading them from your physical disks.

 

You have nothing to lose... 

You must realize, the only reason I do it at all is because my media player times-out if the directory listing request does not return in a timely manner.  It thinks the network is lost.  If it waited for the disk spinup, I would not need to do anything at all.

 

If you are using your server as a media server, and keep your media organized on a few disks, you will not have any downside.  If using 10 of 12 disks in a heavy usage manner, then the 2 that are not being used might have their directory blocks end  up as least-recently-used and spin up when the ls -R is done...  Then, there is a tiny downside.

 

The end all to this is... it is your usage, and your hardware, you decide.

Thanks for all the help!!!!

 

[scottw]

Wow thanks, thats alot to take in!!!

 

At the moment I am running with 512mb of ram and 3 250gb sata HD's. I am moving everything to a different case and will have atleast 6 sata drives, eventually 9-12 (probably 250's since I have some left over). It's mainly for movies that will be scattered throughout disks. I would not be devastated if all the disks spin up from time to time but if I can keep it as efficient as possible that would be a plus  The disks will not be hammered and probably run around the same as yours...one or two movies streams (1.5gb - 2gb per stream) at a time max. I may do some backups and maybe throw some pics and shared files up there.

So since it sounds alot like your setup I will start with the 10second ls -r.

 

Does that sound ok?? 

[scottw]

Sorry after reading again I get the suspicion you may be saying I don't need this. I was under the assumption that since I am pointing my media player directly to the movie file via the share and not thru the individual disk that it would start looking thru some of the disks to find the file.

 

Is that not correct??

[RobJ]

I don't believe he is saying not to use it, but that the benefit may vary with your usage patterns.  Some users may not gain as much as other users.

[yodine]

So after running the ls script every 10s for a week, it doesn't work 100%, even when watching a simple DVD. Once i was relinking the files for my mediaceter while my kid was watching a movie (around 8Go). During that time, it had to spin up 3 drives out of the 12.

 

But the nice thing is that not the 12 spinup. Now i know why reading the latest answers.

 

Even with just streaming videos, the ls every 10s doesn't garanty the file structure will always stay in RAM, but it helps a lot, which is great.

 

I am thinking of going from 2Go to 4Go to help it even more, as i have 11To full so that makes a lot of files (i have HD titles in ISO but DVD riped as folders/files)

 

I use it because when it takes too long to get a file, the software timeout. And awaking 11 drives takes a lot of time, more than 2 minutes.

[scottw]

Good to hear...and yes 3 drives is ALOT better that 12. I am going to give it a try as soon as I get a chance.

[scottw]

Ok just tried this out and am having a problem. I am testing with just movies right now. I set everything up rebooted and I can look at the web page and see when the 4 drives (1-4) have the ls -R run on them because the "read" numbers increase on only those 4 drives, my other 2 drives have nothing on them. I click the button to spin down all of the drives. I call a movie by it's share directly (\\tower\Movies\Spiderman.mkv), no browsing, and it spins up all 4 drives and starts playing off of drive 4. As another test I spin down the drives again and pick a movie I know is on a different disk and call it directly from the share and it only spins up drive 2.

I even tried running the a ls -R /mnt/user/Movies manually and that did not help either.

 

What am I doing wrong??

 

Here is the ls-r script:

#!/bin/sh
crontab -l >/tmp/crontab
grep -q "frequent ls" /tmp/crontab 1>/dev/null 2>&1
if [ "$?" = "1" ]
then
    echo "# frequent ls to keep directory blocks in memory:" >>/tmp/crontab
    echo "* * * * * for i in 1 2 3 4 5 6;do sleep 10; ls -R /mnt/user/Movies 1>/dev/null 2>&1; done" >>/tmp/crontab
    crontab /tmp/crontab
fi

 

and my go script:

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp &
installpkg /boot/pkg/socat-1.6.0.1-i486-2bj.tgz
installpkg /boot/pkg/unraid_notify-2.20-noarch-unRAID.tgz
unraid_notify start
cp /boot/smarttest.sh /etc/cron.daily/
/boot/monthly_parity_check.sh
sleep 30
/boot/ls-r
echo vm.vfs_cache_pressure = 0 >>/etc/sysctl.conf
sysctl -p /etc/sysctl.conf
cd /boot
nohup sh -c "while true; do awk -W re-interval -f /boot/unmenu.awk;done" >/dev/null 2>&1 &

 

Thanks

[WeeboTech]

do

more /proc/sys/vm/vfs_cache_pressure

Make sure the setting is 0

 

Other then that I'm a lil stumped too. accessing my system via user shares spins the drives up too.

However I'm running a busy rtorrent client so I attribute it to that.

[scottw]

do

more /proc/sys/vm/vfs_cache_pressure

Make sure the setting is 0

 

Other then that I'm a lil stumped too. accessing my system via user shares spins the drives up too.

However I'm running a busy rtorrent client so I attribute it to that.

 

Yeah it's set to 0. I was hoping to only spin up the drive I needed but oh well...I don't want to have to set each disk manually to call the movies so I guess I will just live with all drives spinning. I just don't understand that after it finds one movie and plays it then it seems to remember.

 

If anyone else has any ideas please let me know.

[scottw]

One thing I don't understand is if I do a ls -R on //tower/Movies (the share) it lists all of the movies fine but does not show what disk they are on. So how does this help having the directory listing in ram if it does not know which disk it is on, it just seems to show what share it is one which would still spin up all the disks.

 

I think I am just not understanding it properly 

 

Am I supposed to be doing the ls -R on the disks themselves??

 

Thanks,

[Joe L.]

One thing I don't understand is if I do a ls -R on //tower/Movies (the share) it lists all of the movies fine but does not show what disk they are on. So how does this help having the directory listing in ram if it does not know which disk it is on, it just seems to show what share it is one which would still spin up all the disks.

 

I think I am just not understanding it properly 

 

Am I supposed to be doing the ls -R on the disks themselves??

 

Thanks,

You should not need to.  when you do an ls -R on //tower/Movies the underlying fuse-file-system must scan the physical disks for the files and folders to present o you.  It is at that time the disk blocks representing the directories and file names are read into memory.  To read them, they must spin if they are not already in memory.  Since disk drivers do read-ahead, it is possible some of the blocks might already be in memory from the initial boot up and mounting process.

 

I personally think you are stressing too much over all this, or have a very high cost for your electric rates.  Accept that our disks will spin up.  Accept that some will spin up when disk folders are scanned by windows.  Not much you can do about it other than what you have done already to try to keep the disk blocks for directory entries in memory.

 

Joe L.

[scottw]

Cool...thanks for putting it that way. I DO stress over things like this WAY too much. You are right....so it spins up a few drives when I want to watch a movie, after an hour they will spin back down.

 

Thanks for chiming in!!!