New unmenu pluging: myMain


SSD

Recommended Posts

Version 1.2

 

(Bug Fix Version)

 

(Install the latest unmenu release and then unzip the "Update 1.2.zip" file over top.)

 

 

Version 1.0

 

Core features:

1.  Uses the same read and write counts as unRAID 4.4 and 4.5

2.  Internal improvements that require unmenu 1.1 to operate properly.

 

Download as part of unmenu 1.1 HERE.

 

Version 0.9 beta:

 

Core features:

1.  Easily view SMART attributes to maintain healthy drives (smart view)

2.  Performance monitoring of disk and network throughput (performance view)

3.  Multiple views of information, easily configurable

4.  Track attributes including manufacture date, purchase date, usage notes and location in case

 

bgmymenuv09jz5.jpg

 

This version adds the following features:

 

1.  "Greenbar" appearance, making viewing information easier across the wide display (optional)

2.  Visual enhancements

3.  Separate "local" config file

4.  Uses RobJ supplied syslog filtering routine

5.  Adds sort indicators to top of columns as suggested by JoeL.

6.  "Smarter" about drive state (gaps in drive list, etc.)

7.  Numerous internal improvements, better performance

8.  Adds .conf file for use with package manager to install the bwm-ng tool

9.  Ability to configure "fieldsets" at the bottom of the screen including hyperlink tablles, the syslog set, and other web pages

10.  Smart tab enhanced to show descriptions of all attributes (using new fieldset feature).

 

Bugs fixed:

- numerous

 

Installation steps:

1 - unzip the files from the attached zip file to your unmenu directory (the demo logs should go into a directory called “smartlog” in the unmenu directory (this is the default location).

2 - Go to the package manager and install the bwm-ng package (suggest you make it install on each reboot)

3 - Go to the myMain page

4 - (Optional) Edit the myMain_local.conf file.  Add your own drive specific data.  See comments at the top of the file.in myMain_local.conf.  (If you run myMain and notice certain columns are blank (e.g. slot, id, usage notes), it is because you haven't set them in the.conf file).

 

Upgraders:

1 - If you have made no changes to the configuration file, you can just follow the normal installation steps

2 - If you have made changes, rename your myMain.conf so it is not overwritten

3 - Follow the installation steps

4 - Apply your changes to the myMain_local.conf file.

5 - In the future, upgrades will not overwrite your existing myMain_local.conf file.

 

Getting started with configuration (from top of myMain_local.conf)

    1.  Before making any changes, run myMain and make sure it works

    2.  Look on the "Inventory" view.  You will need to be able to cut and paste the serial numbers

    3.  Go the the section below called "SET DISK VALUES" in the myMain_local.conf files and make a copy of the lines for each drive. 

        (Make sure you uncomment them!)  (Note:  You don't have to set all the values, only set the ones that you want to be able

         to use.  You can even add your own!).

    4.  When done, refresh the myMain "Inventory" view.  It uses many of the configured values.  Make sure your configurations are visible.

    5.  Once you have played around a bit, you might want to customize the view settings.  See the section just below to do that.

    6.  Remember, you can create new views and new PseudoCols to make myMain work the way you want it to work.

    7.  Try to make all changes in the myMain_local.conf and not in myMain.conf.  Lines can be copied from myMain.conf for modification.

 

 

Version 0.3 beta:

 

Bugs fixed:

- Numeric smart attributes were sorting lexically not numerically

- View links contained extra "&"

 

 

Version 0.2 beta:

 

mymain02smartsg5.jpg

 

This version adds the following features:

 

1.  Multiple views:

    a. Default

    b. Performance

    c. Quick

    d. Vanilla

    e. Smart

    f. Inventory

    g . Detail

 

2.  Smartctl data

    a. Intelligent color coding of items to indicate areas you may need to investigate.  Over time the heuristics can be tweaked.

    b. There is a special column that will display errors / warnings of undisplayed smart attributes (so you don't have to display every column to see if there is a problem).

    c. Configure a directory to contain smart.txt files (from prior history or from the forums).  This will let you see how myMain displays errors on really problematic drives.  (I have included some smart files I got from the forum as examples in the .zip file)

    d. Once set by the drive, smart attributes can't be changed.  Rather than continue to show items in yellow, orange and red that you already know about, you can acknowledge the smart values and cause myMain to stop complaining about them until their values exceed an authorized limit that you set in the .conf file.

 

3.  Get temperature info from spundown drives - Automatic and manual settings

 

4.  Filtered syslog of disk-related lines  (Hopefully RobJ will take a look and make sure I did this right.)

 

5.  Extensive customization using the myMain.conf file.  (The drivedb.txt file is now obsolete.  Read the myMain.conf file for lots of information about how to customize.)

 

Bugs Fixed:

    1.  1.5T drive sizes should now be displayed properly

    2.   Skipping disk slots should now be skipped on myMain

    3.  Works when array is stopped

    4.  Network Tx and Rx speeds always the same (both were the Tx speed)

 

To Dos:

    1.  Still need to package in bwm-ng package.

    2.  Still want to do a little more with smartctl mining

    3.  Add more GUI customizations

    4.  Working with Joe L. On a way to make myMain replace “Main” for those that want to use this as the default unmenu page.

    5.  Open for suggestions

 

 

<b>version 0.1</b>

I have finished a rough beta of my first unmenu plugin called “myMain”.

 

mymainrt8.jpg

 

myMain is an array management console with these features:

 

1 – Disk performance monitoring – shows the read and write data transfer rates for each drive (See Rx Speed and Tx Speed)

2 – Network performance monitoring – shows the transmit and receive data transfer rates (displays on the “Total” row under Reads and Writes)

3 – One click disk spinup / shutdown (clicking on a “*” in the Temp column spins the drive up, clicking the temperature of a spinning drive spins it down)

4 – One click smart report and hdparm reports

5 – Configurable and extensible column set.  This version allows you to create a text file of additional drive information (e.g., Usage Notes, Purchase Date, Disk Slot in Case, Custom Samba share name, etc.) which can be easily configured as columns on your display.  (There is a file called "drivedb.txt" with a sample row in it to get you started).  There are also a number of other columns like model and serial that can be easily added.

6 – Sortable columns (not just dumb text sorts– these actually work!)

7 – Highlight a row by clicking on the Disk status (“OK”)

8 – Temperature highlighting

Future

1 – Ability to configure multiple column lists and switch between them

2 – Use of config file

3 – Download package for bwm-ng-0.6-i486-2bj.tgz

 

Feel free to edit the list of columns to display.  You'll find instructions in the 07-unmeu-mymain.awk file.

 

Remember this is only a beta.  I have only tested it on IE.  Please give it a try and help me test it out.  Any and all suggestions for improvements are welcome.

 

Note that myMain uses a utility plugin included in the .zip file.  It installs but does not create a menu item.  It is used to run smart and hdparm reports.

 

Special thanks to Joe L. for creating this incredible framework and incredible examples of how to do things in awk, not to mention low level routines for getting at all sorts of technical information.  Also special thanks to WeeboTech for his useful posts on bwm-ng.

 

Link to comment
  • Replies 168
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

I have finished a rough beta of my first unmenu plugin called “myMain”.

 

Wow....  Just incredible...    That's NOT beginner code.  I love your commenting style. (open on the right to allow ease of editing)

 

To answer one of the questions in your code, "awk" has no "include" function. (Believe me, I searched for one)  The best idea offered on the web was to pre-process files to "include" other routines and write the combined output to a temp file.  Then, the "temp" file would be executed.  I really did not like it as a solution for unmenu.awk "plug-ins" as found I could just invoke them and capture their output..  Your idea of using a "utility" plug-in might be adopted for some concepts.  A common routine in the invoking "awk" process can invoke the "utility" the same way we do Linux built-ins.  With a little thought, the common routine might be able to fill an associative array, a bit like we do for the CGI parameters and unmenu.conf define variables.

 

Did I mention how impressed I am... Wow. ;D

 

Joe L.

Link to comment

Wow....  Just incredible...    That's NOT beginner code.  I love your commenting style. (open on the right to allow ease of editing)

 

Thanks so much Joe L.!  Was hoping to add something worthy to the unmenu project.

 

BTW - Joe L. reported a bug via a PM conversation we've been having.  It does not work on a stopped array.  I have fixed the bug and will include with the next update within a day or two.

 

He also reported success on both Firefox and Google Chrome (although chrome omits the grid lines).

 

Now that looks really cool.

 

I'll be a tester and report back.

 

Thanks Rob!  Let me know how it goes.

Link to comment

Works okay for me in FF 3.0.3   I only installed it and verified that it works.   Haven't had time to tweak settings, etc yet, but it's darn cool.

 

Awesome!  I am working on an update that will make the format of the drivedb.txt file obsolete.  Should be out tomorrow.

Link to comment

I Love the "smart" view.

 

Again, WOW!!!  You have set the bar VERY high for Tom at Lime-Technology...

 

This is *exactly* how I envisioned a rapid-development tool would lead to new ideas.  Nobody will ever suspect it is just "awk"

 

Joe L.

 

myMain "smart" view of my array.  (I can see some of my older drives showing some wear. I need to keep an eye on them, in case it gets worse.)

1hrmrn.jpg

 

Link to comment

WOW, that smart view makes me want to use it now! Great Job!!!

 

In addition to seeing the raw /dev/sdx having it cross referenced with the /dev/mdx is a plus

Perhaps it could be as  sdx (mdx) or something like that? or vica versa

 

That's easy to do with just a bit of configuration work ..

 

1.  Create the pseudocolumns - I created both of them ...

 

    #LoadToken(pseudocolumn_name," pseudocolumn_def_string")

 

    LoadPseudoCol(_sdxmdx, "%dev (%md)")

    LoadPseudoCol(_mdxsdx, "%md (%dev)")

 

 

2.  Load the Tokens:

 

    #LoadToken(token_name, sort, alignment, "column heading")

 

    LoadToken(_sdxmdx, dev, center, "Dev (md)")

    LoadToken(_mdxsdx, md, center, "Md (dev)")

 

 

3.  Add one of them to one of the views, for example, to change the "default" view to use this instead of the Dev column ..

 

    SetView(default, 00001001, "_status disk _sdxmdx _sname id share reads writes errors disk_size disk_pctused disk_free usage tempc _spin _info")

 

I've added these two PseudoCols to the .conf file.  Feel free to add it to one or more of your views.  (New version wil be up in a few, after I fix Joe L.'s bug).

Link to comment

Small Bug: Clicking on a column heading to sort it, on anything but the "default" view, seems broken.

 

Very tiny bug... the built URL when clicking on an item looks like this:  myMain?&sss=zzz&rrr=ttt

The initial "?&" should only be a "?"

 

Joe L.

Joe -

 

I think I have fixed the bug that you found.  It did not present itself the same in IE, but I believe that the bug I found and fixed should work for you as well.  Please let me know.

 

I also found some other bugs based on your screenshot.

 

1 - The site uses some special characters that don't seem to be working with your browser.  They are all in the .conf file.  I have created another set of symbols that should be better supported.  The original ones that I used are commented out in the .conf file now for folks that use a browser that supports them (and, of course, you have the fonts on your workstation).

 

2 - The temperature is showing that it is adjusted, when it isn't.  This is now fixed.

 

I have added some screenshots to post 1 in this thread, as well as updated the attached .ZIP file.

Link to comment

Version 0.3 posted with more minor bug fixes.

 

Keep them bug reports comin'!

Not a bug report, but a tiny patch that will allow myMain to be used instead of the "main" page in unmenu.awk

 

If you change one line in the following function, in file 07-unmenu-mymain.awk, to eliminate the "hard-coding" of the myMain URL

(delete the line in "RED", add the line in BLUE)

[pre]

#-------------------------------------------

# Function to help create myMain hyperlink.

#-------------------------------------------

function myMainLink(extrastuff, sep, link)

{

  sep="?"

  link="myMenu"

  link=MENU[2]

  if(index(extrastuff, "sort=") == 0) {

      link=link sep "sort=" activesort

      sep=amp

  }

  if(index(extrastuff, "view=") == 0) {

      link=link sep "view=" activeview

      sep=amp

  }

  if(extrastuff != "")

      link = link sep extrastuff

  #p("link=" link)

  return(link)

}

[/pre]And then change one line near the very top of the 07-unmenu-mymain.awk myMain plug-in file (delete the line in RED, add the line in BLUE)

[pre]BEGIN {

#ADD_ON_URL=myMain

#ADD_ON_URL=

#ADD_ON_MENU=myMain

[/pre]

 

The change to the "myMainLink" function to use MENU[2] instead of a hard-coded URL works, no mater if you change the top line to delete "URL" name, or leave it as #ADD_ON_URL=myMain  (If you make this change, make sure there are no spaces after the equals sign..  The last character on the #ADD_ON_URL line should be "=" )

 

I suggest that change in the function be made in the next beta. (and people can edit it if they want it to replace the default "main" page)

If it is, then simply by changing line 2, to define myMain as the URL, or blank it out, to make myMain have the same URL as the default "main" page, and overlay it in the top set of links, will allow you to choose to have both "main" and myMain", or only one.

 

This is really cool, and takes complete advantage of the feature I built into unmenu that allows built-in pages to be replaced by plug-ins.

 

Since myMain does not do everything the default unmenu "Main" page does, (at least, not yet) might want to keep both in the menu for a while, but who knows... at least you have the choice, once the myMainLink function is changed you can easily edit one line and have a brand new "Main" page.

 

Joe L.

 

 

Link to comment

Thanks Joe L.  (Glad I decided against hardcoding the page name in multiple places!)

 

BTW, I was looking for some detailed descriptions of the S.M.A.R.T. attributes (what the heck is a "high fly write"?!).  Found THIS page.  Look about 3/4 of the way down.  Very nice compilation.

 

This got me thinking, awk is so good at parsing through command outouts.  I wondered if awk could access a URL and parse it in a similar way?  Is that possible?

 

I found the bug that caused those extraneous lines to show up on your smart page (thatnks for the logs, i've added them to my collection).  It was mistakenly parsing some of your error messages as though they were attribute lines.  Working on some minor enhancements.  Will likely have something to post over the weekend.  It should be the last beta before v1.0 unless I get a brainstorm of something major to add.

Link to comment
4.  Filtered syslog of disk-related lines  (Hopefully RobJ will take a look and make sure I did this right.)

 

Hope the following helps, my little contribution to your excellent work and creativity, great ideas.  This is a replacement for the FilterSyslog() function.  I split the analysis for the 5 different cases, and added a way to include additional drive-related lines.  It is not fully tested, as I have no IDE drives, or Cache drive, or drives outside of my array.  But it does work fine with my parity, flash, and SATA drives (except for one Emask error line mentioned below).

 

It is not perfect, would require pre-processing the syslog to get more.  One of the most helpful lines in an 'exception Emask' error sequence is the one with the human-readable error indicator (like ATA bus error, device error, media error, timeout, etc), but unfortunately it is on a continuation line, without the ata# prefix.  Of course, if any one wanted help and saw the errors related to a drive, and posted about it, we would want the entire syslog, so this is a low priority.  But perhaps in the future, it would be better to process the syslog to a temp file, replacing the blanks in the continuation lines with the ata prefix from the preceding line, then grep from the temp file.

 

function FilterSyslog(syslog, dev, disk)
{
   if ( substr(dev,1,2) == "hd" ) {   # IDE devices
      if ( disk == "parity" )
         cmd = "|md0: [^p]"
      else
         cmd = "|md" substr(disk, 5) "[^0-9]"
   }
   else {   # SATA or other SCSI devices
      #Step 1 - Find the ata#s
      cmd = "cat " syslog "|grep '\\[" dev "\\]'"
      #p(cmd)
      cmd|getline f
      close(cmd)
      split(f,d);
      t=d[7];
      delete d;
      split(t, d, ":")
      atanum=d[1]
      atanum2=d[3]
      #p(dev ", " disk ", " atanum ", " atanum2);
      if ( disk == "flash" )
         cmd = "|scsi" atanum " :|scsi " atanum ":0:" atanum2 ":0"
      else if ( disk == "parity" )
         cmd = "|scsi" atanum " :|scsi " atanum ":0:" atanum2 ":0| ata" atanum " ata" atanum ".0" atanum2 ":|md0: [^p]"
      else
         cmd = "|scsi" atanum " :|scsi " atanum ":0:" atanum2 ":0| ata" atanum " ata" atanum ".0" atanum2 "" disk "[^0-9]| md" substr(disk, 5) ":"
   }
   cmd = "cat " syslog "|egrep \"(" dev cmd ")\""
   #p(cmd)
   print RunCommand(cmd, "Filtered syslog for " disk (substr(disk,1,4) != "disk" ? " drive" : "") )
}

Link to comment

Thanks Joe L.  (Glad I decided against hardcoding the page name in multiple places!)

I'm glad too.

BTW, I was looking for some detailed descriptions of the S.M.A.R.T. attributes (what the heck is a "high fly write"?!).   Found THIS page.  Look about 3/4 of the way down.  Very nice compilation.

 

This got me thinking, awk is so good at parsing through command outouts.  I wondered if awk could access a URL and parse it in a similar way?  Is that possible?

I think I can throw something together to get you started.  I'll make it a stand-alone script as a demo, you can incorporate it, improve it as desired.

It will need to incorporate a finite-state-machine in order to parse the HTML of that wiki page.  (I'm assuming you want to grab the description of the SMART attributes dynamically)  Do you want to select one by the "Name" or by the attribute ID number?   For efficiency, I'll probably load up a local file. If it exists, I'll use its contents, if not, I'll use the parsed Wiki contents (if network connectivity exists) and create it (and then subsequently use it)

I found the bug that caused those extraneous lines to show up on your smart page (thatnks for the logs, i've added them to my collection).  It was mistakenly parsing some of your error messages as though they were attribute lines.  Working on some minor enhancements.  Will likely have something to post over the weekend.  It should be the last beta before v1.0 unless I get a brainstorm of something major to add.

Might want to use the defined variable that are passed in when looking for the myMain configuration file and not assume it is in the current directory.

When invoked, the plug-in has the following variables defined for your use:

gawk -v ConfigFile=unmenu.conf -v MyHost=Tower -v ScriptDirectory=. -v LocalConfigFile=unmenu_local.conf -v MyPort=8080 -W  re-interval -f ./07-unmenu-mymain.awk  ...

You can see the full set of arguments to the plug-in, when it is invoked, by looking at the "ps info" command button output in the system-info plug-in and seeing how "gawk" is invoking the plug-in.

 

Most interesting for you would be ScriptDirectory.  It is already set for your use, and you probably should call:

LoadConfigFile(ScriptDirectory "/myMain.conf")

instead of

LoadConfigFile("myMain.conf")

 

Joe L.

Edit: fixed typo.

Link to comment
I think I can throw something together to get you started.  I'll make it a stand-alone script as a demo, you can incorporate it, improve it as desired.

It will need to incorporate a finite-state-machine in order to parse the HTML of that wiki page.  (I'm assuming you want to grab the description of the SMART attributes dynamically)  Do you want to select one by the "Name" or by the attribute ID number?  For efficiency, I'll probably load up a local file. If it exists, I'll use its contents, if not, I'll use the parsed Wiki contents (if network connectivity exists) and create it (and then subsequently use it)

 

That would be very cool.  I can see lots of utility to being able to scrape a web page and run it through awk!

 

Might want to use the defined variable that are passed in when looking for the myMain configuration file and not assume it is in the current directory.

When invoked, the plug-in has the following variables defined for your use:

gawk -v ConfigFile=unmenu.conf -v MyHost=Tower -v ScriptDirectory=. -v LocalConfigFile=unmenu_local.conf -v MyPort=8080 -W -W re-interval -f ./07-unmenu-mymain.awk  ...

You can see the full set of arguments to the plug-in, when it is invoked, by looking at the "ps info" command button output in the system-info plug-in and seeing how "gawk" is invoking the plug-in.

 

Most interesting for you would be ScriptDirectory.  It is already set for your use, and you probably should call:

LoadConfigFile(ScriptDirectory "/myMain.conf")

instead of

LoadConfigFile("myMain.conf")

 

Done.

 

Link to comment

I think I can throw something together to get you started.  I'll make it a stand-alone script as a demo, you can incorporate it, improve it as desired.

It will need to incorporate a finite-state-machine in order to parse the HTML of that wiki page.  (I'm assuming you want to grab the description of the SMART attributes dynamically)  Do you want to select one by the "Name" or by the attribute ID number?   For efficiency, I'll probably load up a local file. If it exists, I'll use its contents, if not, I'll use the parsed Wiki contents (if network connectivity exists) and create it (and then subsequently use it)

 

That would be very cool.  I can see lots of utility to being able to scrape a web page and run it through awk!

Here is a link if you want to give it a try... I won't be able to do anything till tomorrow at the earliest.

http://www.gnu.org/software/gawk/manual/gawkinet/html_node/GETURL.html#GETURL

 

It just sends the page source to stderr, and that is the easy part.

Parsing the returned HTML to grab rows of the specific table content desired is a lot more code.

 

Joe L.

Link to comment

The suppression of rows in the disk display, for those slots in the array that are not populated, can be fixed by a slight change to the following code (adding the lines in BLUE)  This tiny bug results in un-populated slots showing up as "parity" in the display:

In addition, the logic in RED is never exercised, as the variable num is not defined, nor set. I think it will always be evaluated as 0. 

[pre]

#-----------------------------------------------------------------------

# This function begins the creation of the drivedb[] associative array.

#-----------------------------------------------------------------------

function LoadDriveDb() {

  GetArrayStatus(); #call JoeL.'s function to do the heavy lifting

  nextslot=numdisks

 

  #-----------------------------------------

  # Loop though all the disks in the array.

  #-----------------------------------------

  i=0;

  for(k=0; k<numdisks; k++) {

      if ( disk_status[ i ] == "DISK_NP" && disk_device[ i ] == "" ) {

          continue;

      }

      drivedb[i, "num"] = i

      #drivedb[i, "name"] = disk_name[ i ];

      if((disk_name[ i ] == "") && (num == 0)) {

        drivedb[i, "md"] = "parity"

        drivedb[i, "role"] = "parity"

      }

      else if(disk_name[ i ] == "")

        continue;

      else {

        drivedb[i, "md"] = disk_name[ k ];

        drivedb[i, "role"] = "array disk"

      }

[/pre]

We also need to correctly handle the display when adding a new disk. After adding the lines in blue to suppress unpopulated slots, possible logic would be to replace these lines that assign the disk labels and roles:

    if((disk_name[ i ] == "") && (num == 0) {

        drivedb[i, "md"] = "parity"

        drivedb[i, "role"] = "parity"

      }

      else if(disk_name[ i ] == "")

        continue;

      else {

        drivedb[i, "md"] = disk_name[ k ];

        drivedb[i, "role"] = "array disk"

      }

with this:

    if ( disk_name[ i ] == "" ) {

        if ( disk_status[ i ] == "DISK_NEW" ) {

          drivedb[i, "md"] = "new disk"

          drivedb[i, "role"] = "array disk"

        } else {

          drivedb[i, "md"] = "parity"

          drivedb[i, "role"] = "parity"

        }

      } else {

        drivedb[i, "md"] = disk_name[k];

        drivedb[i, "role"] = "array disk"

      }

 

(I don't currently have an un-populated slot in my array, but this should work...  ;))

The edited function I'm using looks like this:

[pre]

function LoadDriveDb() {

  GetArrayStatus(); #call JoeL.'s function to do the heavy lifting

  nextslot=numdisks

 

  #-----------------------------------------

  # Loop though all the disks in the array.

  #-----------------------------------------

  i=0;

  for(k=0; k<numdisks; k++) {

      if ( disk_status == "DISK_NP" && disk_device == "" ) {

          continue;

      }

      drivedb[i, "num"] = i

      #drivedb[i, "name"] = disk_name;

      if ( disk_name == "" ) {

        if ( disk_status == "DISK_NEW" ) {

          drivedb[i, "md"] = "new disk"

          drivedb[i, "role"] = "array disk"

        } else {

          drivedb[i, "md"] = "parity"

          drivedb[i, "role"] = "parity"

        }

      }  else {

        drivedb[i, "md"] = disk_name[k];

        drivedb[i, "role"] = "array disk"

      }

[/pre]

 

 

Joe L.

Link to comment
Guest
This topic is now closed to further replies.