spinup groups of affiliated disks when accessed. A different approach

[Joe L.]

This script is a different approach to making media players happy (and wife happy as a result) by immediately spinning up all the disks affiliated with a given "user-share" whenever any file or folder within the user-share is accessed.  This is NOT a green approach as it will result in more spinning disks, so those trying to minimize power consumption can stop reading now.

 

For those with media players that will not wait for disk spinup, and time-out, and make for a less enjoyable movie watching experience, this might work.

 

This version must be used with the latest of unRAID versions that have spinup groups.  If you're not on 4.5beta13, or 4.5final, or later, stop reading now.

 

A zipped version is attached.

 

Only issue so far is when one of my media players does not "cleanly" close its session when it times out and returns to its main menu after playing a movie.  the smbd daemon shows it still having a directory listing active on the "/mnt/user/Movies" directory.  I think smbd will clean itself up when I power down the media player after a while.  (We'll see soon enough) 

Edit: the smbd process does terminate itself within roughly 30 minutes of the media player being turned off.  Cool.

 

To invoke type:

spinup_when_accessed.sh

(It will run itself in the background and detach itself from your terminal session. output will be written to the syslog)

 

To terminate the program after it detaches from your terminal, type:

spinup_when_accessed.sh -q

 

To invoke in the foreground, and not detach from your terminal, and see the output as it runs, type:

spinup_when_accessed.sh -F

 

To get help, type: spinup_when_accessed.sh -?

root@Tower:/boot/custom/bin# spinup_when_accessed.sh -?

Usage: spinup_when_accessed.sh [-m "media_player_ip_list"] [-d "disks_to_spin"] [-F] [-u "user_share_to_spin"] [-e "exclude_user_share"]

      spinup_when_accessed.sh -v      = print program version

      spinup_when_accessed.sh -q      = terminate currently running program

      spinup_when_accessed.sh -F      = do not automatically run in the background, run in the forground instead

      spinup_when_accessed.sh -m "media_player_ip_list"      =  spin up if any device matching IP in list is online.

                          (example -m "192.168.1.10 192.168.1.11 192.168.1.12")

                    -d "disk_numbers_to_spin_when_media_player_online"

                          (example -d "1 4 6")

                    -u "user_share_to_spin_when_media_player_online" (may be repeated as many times as desired)

                          (example -u "Movies" -u "Music") 

                    -e "exclude_user_share"  (may be repeated as many times as desired)

 

You do not need to specify the "-m" option at all unless you wish to.  This script, in combination with the cache_dirs script is all I've found I need, and I don't need to supply any paramaters at all when I invoke it here on my server.  I simply invoke it as

spinup_when_accessed.sh

The -d and -u options only are used in combination with the -m option.  You do not need to specify -d or -u, as the default is to spin up all the disks if a media player listed by the -m option is detected.  It just allows you to be a bit more green if only some of your drives have specific media.  You may specify the "-u" option multiple times to spin up multiple "user-shares" when an IP address given by the -m option is detected on the LAN.  (when it responds to a "ping" command)

 

The program's activity is logged in the syslog.  It uses the new "spinup" commands lime-technology has added in recent releases.  It is not usable on older releases, as they lack the spinup commands needed.  If you attempt to use it on an older release, it will exit with an error message and not run.

 

If you wish to watch it as it runs, run it in the foreground with the "-F" option.  It will print messages as it runs showing what it is doing.  To work properly, it expects the disk spin-down time to be 30 minutes or greater.  (I have mine set for 1 hour)

 

This is now version 1.5.

# spinup_when_accessed.sh

# December 10, 2009, 03:20:19 PM  version 1.1  Joe L. initial re-write... basic functions only for now.

# December 13, 2009, 03:01:52 PM  version 1.2  Joe L. Added options, poll of media players.

# December 15, 2009, 01:34:08 PM  version 1.3  Joe L. Added more details of which disks are being spun

#                                                                            and ability to exclude specific user-shares.

# version 1.4  Joe L. Fixed syntax error that created file named "0"

#                    on flash drive.

# version 1.5  Joe L. Fixed error when output of lsof heading changed format

#                    when smbd user changed from "root" to "nobody"

 

Edit: updated attachment to version 1.5.  May 30, 2013. Joe L.

 

Joe L.

spinup_when_accessed.zip

[peter_sm]

Just q question, you say something about "smbd daemon", when like me accessing my TviX using NFS is there any issue then, I have no time to test this today, will give it a try during the weekend 

[Joe L.]

I expect the two samba processes will eventually close due to inactivity... as I said, we'll see.  They each have their current-working-directory as the respective user-share of Movies A-D and Movies T-Z  (They were the last two user-share-folders I played a movie from this morning in my testing)

As I expected, within 30 minutes of shutting off my media player, the associated smbd process noticed it was no longer connected and terminated... so my disks spun down after their normal spindown delay from their last access.

 

Cool stuff.

 

Joe L.

[Joe L.]

Something unexpected, and nice.  (fewer disks spin... just those needed to make media playing work without disk spin-up visible to media player users.)

 

When using the individual "spinup" commands the spinup-groups are not used.  Initially I thought they would be used.  Now, before you say anything, this is good, because the specific disk being accessed, if in a spinup group with a second drive, will spin up others in the same group when a file on it is accessed. (if spinup groups are enabled)

 

So, Using my original example

If I have /mnt/disk1/Music, and /mnt/disk5/Music and /mnt/disk9/Music then

when I browse to /mnt/user/Music (The \\tower\Music share) this script will immediately spin up disk 1,5 and 9.

 

If I play a file that actually resides on /mnt/disk5/Music, and disk5 is part of a spinup group that includes disk6, then the spin-up logic Tom recently added will spin it (disk6) up too to minimize any possibility of a pause in the music.    The fact that disk 1 and 9 are potentially in their own spin-up groups does not matter and their affiliated group members are not spun up unless a file on disk1 or disk9 are accessed.

 

Joe L.

[peter_sm]

i have a Tvix when tvix is on-line and have not accessed the share all disk spun down, tying to access the share is not working, and tha tis becouse al ldisk are spunn down.

 

So is it some way to add/replace so instead looking at the share just ping the MP ip:s ? to avoid any disc spin down ?

 

EDIT

So below code,  add if a IP are accessed/online

 

do
        test -f "$i" && {
          #echo $i FILE ACCESSED
          dir=`dirname "$i" | sed "s/^\/mnt\/user\///"`
        }
        test -d "$i" && {
          #echo $i DIRECTORY ACCESSED
          dir=`echo "$i" | sed "s/^\/mnt\/user\///"`
        }
        disk_dirs=`ls -d /mnt/disk*"/$dir"`
        disks=`echo "$disk_dirs" | sed -n "s/\/mnt\/disk\([0-9]*\).*/\1 /p"`
        echo "$disks" >>/tmp/spin_when_access_disks$parent_pid
    done

 

BR

Peter

[Joe L.]

i have a Tvix when tvix is on-line and have not accessed the share all disk spun down, tying to access the share is not working, and tha tis becouse al ldisk are spunn down.

 

So is it some way to add/replace so instead looking at the share just ping the MP ip:s ? to avoid any disc spin down ?

 

EDIT

So below code,  add if a IP are accessed/online

 

do
        test -f "$i" && {
          #echo $i FILE ACCESSED
          dir=`dirname "$i" | sed "s/^\/mnt\/user\///"`
        }
        test -d "$i" && {
          #echo $i DIRECTORY ACCESSED
          dir=`echo "$i" | sed "s/^\/mnt\/user\///"`
        }
        disk_dirs=`ls -d /mnt/disk*"/$dir"`
        disks=`echo "$disk_dirs" | sed -n "s/\/mnt\/disk\([0-9]*\).*/\1 /p"`
        echo "$disks" >>/tmp/spin_when_access_disks$parent_pid
    done

 

BR

Peter

I use the "cache_dirs" program to keep all the directory information in memory so I can get immediate listings of directories...  For that reason I do not need to spin up anything special and the initial listing does not time out.  This program has been working perfectly for me and my family so far.  Are you using cache_dirs?

 

When your media player goes to get its initial listing of media, what do you see with the

lsof /mnt/user

command?

 

Joe L.

[peter_sm]

What I got on the TviX side when all disc are spun down is "No Disc" when trying to access the net share, before that the tvix was not access any share at all, therefore all disc was spunn down, but I can check that you telling me 

 

EDIT

 

using lsof /mnt/user it gives nothing, the Tvix is only mount the share wile I accessing the disc share from tvix, changing back to local HD on Tvix the link to unRAID are  then gone.

[peter_sm]

What I got on the TviX side when all disc are spun down is "No Disc" when trying to access the net share, before that the tvix was not access any share at all, therefore all disc was spunn down, but I can check that you telling me 

 

EDIT

 

using lsof /mnt/user it gives nothing, the Tvix is only mount the share wile I accessing the disc share from tvix, changing back to local HD on Tvix the link to unRAID are  then gone.

 

EDIT 2

Joe, I'm not using cache_dirs, do you recommended that? for me using TviX it's should be great if you could add ping IP, so perhaps user can choose between FILE/DIRECTORY ACCESSED or ping IP

 

 

Thanks

 

Peter

[Joe L.]

What I got on the TviX side when all disc are spun down is "No Disc" when trying to access the net share, before that the tvix was not access any share at all, therefore all disc was spunn down, but I can check that you telling me 

 

EDIT

 

using lsof /mnt/user it gives nothing, the Tvix is only mount the share wile I accessing the disc share from tvix, changing back to local HD on Tvix the link to unRAID are  then gone.

 

EDIT 2

Joe, I'm not using cache_dirs, do you recommended that?

Absolutely... it is critical to keeping the directory information in memory so the disks do not need to spin up for the initial listing of media.

for me using TviX it's should be great if you could add ping IP, so perhaps user can choose between FILE/DIRECTORY ACCESSED or ping IP

I'm adding it ... next version will have ability to supply a list of IP addresses and either a list of the disks to spin up, or a list of user-shares to spin up when those IPs are detected... (if no list, all disks are spun up)

Thanks

 

Peter

You are welcome

[peter_sm]

Absolutely... it is critical to keeping the directory information in memory so the disks do not need to spin up for the initial listing of media.

I have now Installed it 

 

Looking forward to your next release

[peter_sm]

I have tested the cache_dirs now, I let unraid spin down all the discs, on my tvix I navigate to the unraid share, and I could browse my files, so I don't know reason to use any more scripts, Joe, that script you are working on , should that give me something more ?

 

So just have cache_dir running looks to work great, so now I have acess to the server even all my disc are spunn down  Is that suppose to work like this ?

I have a script called s3_notHrHdTcpIp.sh to avoid the server go to sleep if my MP is online, if MP are offline it sleep the server 10 minutes after all disc have been spunn down

 

 

//Peter

[Joe L.]

I have tested the cache_dirs now, I let unraid spin down all the discs, on my tvix I navigate to the unraid share, and I could browse my files, so I don't know reason to use any more scripts, Joe, that script you are working on , should that give me something more ?

Your choice if you wish to use its feature to spin up when an IP address is detected.  You don't need to use it if you don't want to.

So just have cache_dir running looks to work great, so now I have acess to the server even all my disc are spun down  Is that suppose to work like this ?

Yes, you should be able to get the directory listings, and, as soon as a directory listing in progress is detected by the spinup_when_accessed.sh script it will spin up the disks affiliated with the user-share being browsed.  It is working perfectly for me.

I have a script called s3_notHrHdTcpIp.sh to avoid the server go to sleep if my MP is online, if MP are offline it sleep the server 10 minutes after all disc have been spunn down

 

 

//Peter

You might still need that if you want to automatically shut your server down when your media players are turned off.

 

In any case... I'll attach the new version of this script to the first post in this thread so you can try it out.

 

It will automatically put itself in the background and detach itself from the terminal session, so to invoke it you just type

spinup_when_accessed.sh

 

If you want help, type spinup_when_accessed.sh -?

It will then print

root@Tower:/boot/custom/bin# spinup_when_accessed.sh -?

 

Usage: spinup_when_accessed.sh [-m "media_player_ip_list"] [-d "disks_to_spin"] [-F] [-u "user_share_to_spin"]

      spinup_when_accessed.sh -v      = print program version

      spinup_when_accessed.sh -q      = terminate currently running program

      spinup_when_accessed.sh -F      = do not automatically run in the background, run in the forground instead

      spinup_when_accessed.sh -m "media_player_ip_list"      =  spin up if any device matching IP in list is online.

                          (example -m "192.168.1.10 192.168.1.11 192.168.1.12")

      spinup_when_accessed.sh -d "disk_numbers_to_spin_when_media_player_online"

                          (example -d "1 4 6")

      spinup_when_accessed.sh -u "user_share_to_spin_when_media_player_online"

                          (example -u "Movies" -u "Music")

 

Joe L.

[Joe L.]

Version 1.2 now attached to first post in this thread.

 

Feedback is encouraged...

 

Enjoy

 

Joe L.

[peter_sm]

It's looking good so for:-) just a Q, if I have set my disc to spunn down at 30minutes, do I need to change something in the script ? and if, what to change ?

 

This script and cache_dirs and my S3 script is what I have right know to maintain my server, It would be a DREAM if all this could be implemented in the new version 5 :-)

 

About S3 sleep, my MB need to have s2ram instead of the original command that was in my S3 script. perhaps that also could be implemented in the new version 

[Joe L.]

peter_sm

 

I've made a few more enhancements... the program now has a few more options.  I'll attach version 1.3 to the first post and you can give it a try.  It now has the ability to specify user-shares to exclude from being spun up as a group.  It also has more detailed output on what it is doing.  It now lets you know how long each disk has been idle.   You can see below that the movie I was playing was on disk6.  (Most of my disks have movies in the E-J* title range., so most are spinning.  This gives me the incentive to group them alphabetically on the disks... but moving them around so fewer disks spin will be a huge job  .

 

You probably do not need to make any changes to the script now, as I've adjusted the spin timer to re-spin every 20 minutes. 

 

Have fun.  The new 1.3 version is attached to the first post in this thread.

 

Joe L.

 

Here is a sample run in the foreground where I excluded my "data" and "Pictures" folders from being processed:

root@Tower:/boot/custom/bin# spinup_when_accessed.sh -F -e "data" -e "Pictures"
spin_disks: #============================ Spin-Disks ================================#
spin_disks: Invoked as: spinup_when_accessed.sh -F -e data -e Pictures
spin_disks: process ID 5482.
spin_disks: To terminate it, type: Control-C
spin_disks: #============================ Spin-Disks ================================#
spin_disks: =================================================================
spin_disks: Tue Dec 15 11:54:46 EST 2009
spin_disks: Spin up disks  1 2 3 4 5 6 7 8 10 11 on shares for files/directories:
spin_disks: COMMAND   PID USER   FD   TYPE DEVICE       SIZE NODE NAME
spin_disks: smbd    30038 root  cwd    DIR   0,13         48 7839 /mnt/user/Movies E-J
spin_disks: smbd    30038 root   28u   REG   0,13 4997588992 7906 /mnt/user/Movies E-J/FIREWALL.ISO
spin_disks: disk  1 = 18 minutes, 52 seconds since last I/O
spin_disks: disk  2 = 18 minutes, 52 seconds since last I/O
spin_disks: disk  3 = 18 minutes, 52 seconds since last I/O
spin_disks: disk  4 = 18 minutes, 52 seconds since last I/O
spin_disks: disk  5 = 18 minutes, 52 seconds since last I/O
spin_disks: disk  6 = 2 seconds since last I/O
spin_disks: disk  7 = 18 minutes, 52 seconds since last I/O
spin_disks: disk  8 = 18 minutes, 52 seconds since last I/O
spin_disks: disk 10 = 18 minutes, 51 seconds since last I/O
spin_disks: disk 11 = 18 minutes, 51 seconds since last I/O
spin_disks: disk(s)   1  2  3  4  5  6  7  8 10 11 currently spinning
spin_disks: =================================================================
spin_disks:
spin_disks: =================================================================
spin_disks: Tue Dec 15 12:04:47 EST 2009
spin_disks: Spin up disks  1 2 3 4 5 6 7 8 10 11 on shares for files/directories:
spin_disks: COMMAND   PID USER   FD   TYPE DEVICE       SIZE NODE NAME
spin_disks: smbd    30038 root  cwd    DIR   0,13         48 7839 /mnt/user/Movies E-J
spin_disks: smbd    30038 root   28u   REG   0,13 4997588992 7906 /mnt/user/Movies E-J/FIREWALL.ISO
spin_disks: disk  1 = 28 minutes, 53 seconds since last I/O
spin_disks: disk  2 = 28 minutes, 53 seconds since last I/O
spin_disks: disk  3 = 28 minutes, 53 seconds since last I/O
spin_disks: disk  4 = 28 minutes, 53 seconds since last I/O
spin_disks: disk  5 = 28 minutes, 53 seconds since last I/O
spin_disks: disk  6 = 1 seconds since last I/O
spin_disks: disk  7 = 28 minutes, 53 seconds since last I/O
spin_disks: disk  8 = 28 minutes, 53 seconds since last I/O
spin_disks: disk 10 = 28 minutes, 52 seconds since last I/O
spin_disks: disk 11 = 28 minutes, 52 seconds since last I/O
spin_disks: disk(s)   6 currently spinning
spin_disks: disk(s)   1  2  3  4  5  7  8 10 11 spun again ... to keep spinning
spin_disks: =================================================================
spin_disks:
spin_disks: =================================================================
spin_disks: Tue Dec 15 12:14:48 EST 2009
spin_disks: Spin up disks  1 2 3 4 5 6 7 8 10 11 on shares for files/directories:
spin_disks: COMMAND   PID USER   FD   TYPE DEVICE       SIZE NODE NAME
spin_disks: smbd    30038 root  cwd    DIR   0,13         48 7839 /mnt/user/Movies E-J
spin_disks: smbd    30038 root   28u   REG   0,13 4997588992 7906 /mnt/user/Movies E-J/FIREWALL.ISO
spin_disks: disk  1 = 10 minutes, 1 seconds since last I/O
spin_disks: disk  2 = 10 minutes, 1 seconds since last I/O
spin_disks: disk  3 = 10 minutes, 1 seconds since last I/O
spin_disks: disk  4 = 10 minutes, 1 seconds since last I/O
spin_disks: disk  5 = 10 minutes, 1 seconds since last I/O
spin_disks: disk  6 = 1 seconds since last I/O
spin_disks: disk  7 = 10 minutes, 1 seconds since last I/O
spin_disks: disk  8 = 10 minutes, 1 seconds since last I/O
spin_disks: disk 10 = 10 minutes, 1 seconds since last I/O
spin_disks: disk 11 = 10 minutes, 1 seconds since last I/O
spin_disks: disk(s)   1  2  3  4  5  6  7  8 10 11 currently spinning
spin_disks: =================================================================
spin_disks:
spin_disks: =================================================================
spin_disks: Tue Dec 15 12:24:49 EST 2009
spin_disks: Spin up disks  1 2 3 4 5 6 7 8 10 11 on shares for files/directories:
spin_disks: COMMAND   PID USER   FD   TYPE DEVICE       SIZE NODE NAME
spin_disks: smbd    30038 root  cwd    DIR   0,13         48 7839 /mnt/user/Movies E-J
spin_disks: smbd    30038 root   28u   REG   0,13 4997588992 7906 /mnt/user/Movies E-J/FIREWALL.ISO
spin_disks: disk  1 = 20 minutes, 2 seconds since last I/O
spin_disks: disk  2 = 20 minutes, 2 seconds since last I/O
spin_disks: disk  3 = 20 minutes, 2 seconds since last I/O
spin_disks: disk  4 = 20 minutes, 2 seconds since last I/O
spin_disks: disk  5 = 20 minutes, 2 seconds since last I/O
spin_disks: disk  6 = 1 seconds since last I/O
spin_disks: disk  7 = 20 minutes, 2 seconds since last I/O
spin_disks: disk  8 = 20 minutes, 2 seconds since last I/O
spin_disks: disk 10 = 20 minutes, 2 seconds since last I/O
spin_disks: disk 11 = 20 minutes, 2 seconds since last I/O
spin_disks: disk(s)   6 currently spinning
spin_disks: disk(s)   1  2  3  4  5  7  8 10 11 spun again ... to keep spinning
spin_disks: =================================================================
spin_disks:
spin_disks: =================================================================
spin_disks: Tue Dec 15 12:34:50 EST 2009

[peter_sm]

I give it a try, but perhaps my share is not so good organised , look here....

 

My share is named tvixhd1 on the unRAID, I have that folder on all my 5 disc, below that on all disc I have folders MKV, BD, Music, Photo

Let say i play a movie that belong on disc3, is it possible to spinn down the rest of the disc except disc3 ?

 

Right now is all my disc spinning except parity disc

 

 

All disc have this structure

tvxhd1
    -MKV
        - Movies_folders
    -BD
         -BD Folders
    -Music
        -Artist folder
    -Photo
       -Photo folders

[Joe L.]

I give it a try, but perhaps my share is not so good organised , look here....

 

My share is named tvixhd1 on the unRAID, I have that folder on all my 5 disc, below that on all disc I have folders MKV, BD, Music, Photo

Let say i play a movie that belong on disc3, is it possible to spinn down the rest of the disc except disc3 ?

Sorry, but no.  As I said, you trade off being green with being able to have the server NOT get in the way of your family's use of your media player client.

 

I have even more disks than you, and if I access my "Movies" share, everything spins up and stays spinning until I finish watching the movie.  For me, this is perfectly fine, as I don't have to deal with the problems when the disks are not spinning and the media-players time-out trying to get a listing, or when trying to start a given movie.

 

Most of the time we browse the "Movies_New" share, and those are the newest movies in out collection, and not yet watched, and on just a few disks.   I'll eventually write a script to re-balance the movies across all the drives...  It will take forever to run, as I have a lot of disks... and I have no idea on which disk is best for any given title until I start doing a lot of analysis so I can get them in title-ranges.

 

95% percent of the time, my server is idle.  For some number of hours per week, we watch movies...  During those hours, I don't care if the disks are spinning as I turn the lights down in the theater  and enjoy the movie.  (see, I am green.   The savings from turning off the room lights balance things out a bit.)

[peter_sm]

 

95% percent of the time, my server is idle.  For some number of hours per week, we watch movies...  During those hours, I don't care if the disks are spinning as I turn the lights down in the theater  and enjoy the movie.  (see, I am green.  The savings from turning off the room lights balance thing out a bit.)

Hi, Actually that's my case also  , so I'm happy like you are   

[cowgill]

I love the script, but I'm having a "minor" problem which makes it useless for me at the moment...

 

Even after I turn my Popcorn Hour off (into standby mode), it will return a ping. Very odd. I confirmed when I turn the hard switch off on the back of the PCH then the IP does not return a ping, but it does return a ping in standby mode. in essence, this will spin up my disks for eternity!

 

Any ideas? Right now I'm relegated to hitting the hard switch on the PCH every time.

[Joe L.]

All I can say is don't use the "-m ip-addr" option at all.  The script should still detect when you access the shares themselves and spin up the disks that comprise the share.

 

Of course, that is hoping the PCH does not keep an active SMB connection when it goes into standby.

[cowgill]

Thanks for the suggestion, Joe. I'll mess around with it more this weekend.

[nia]

Wow - this is a brilliant and all to well hidden script.  Only found it due to the request from peter_sm here: http://lime-technology.com/forum/index.php?topic=10660.msg102646#msg102646

 

This should definitely be part of the package manager!!!

 

I am using a PCH-100 in the living room, and the timeouts at first use has been one of the only gripes with unRAID, and I have so far explained the necessity of it due to power saving from spindown. But it does not rank very high on the WAF scale.

 

With this, it will be MUCH better 

[pantner]

this looks really great! Downloading now!

 

      spinup_when_accessed.sh -m "media_player_ip_list"      =  spin up if any device matching IP in list is online.

                          (example -m "192.168.1.10 192.168.1.11 192.168.1.12")

 

One question about the above mode, I assume it uses some type of ping to check if the other computer is active? How often does it 'poll' for the computer?

 

*EDIT - Also, can you use a DNS name instead of an IP Address?

[Joe L.]

this looks really great! Downloading now!

 

      spinup_when_accessed.sh -m "media_player_ip_list"      =  spin up if any device matching IP in list is online.

                          (example -m "192.168.1.10 192.168.1.11 192.168.1.12")

 

One question about the above mode, I assume it uses some type of ping to check if the other computer is active? How often does it 'poll' for the computer?

 

*EDIT - Also, can you use a DNS name instead of an IP Address?

yes, you can use a DNS name... provided it resolves on your LAN.  (my players do not, so I use IP addresses)  If you can type

ping -c 5 dns_name

on the unRAID command line, and have it resolve, you can use the dns_name.

 

When idle, it loops checking constantly.    It is just a shell script... open it in an editor and learn how it works.

 

Joe L.