SMART paramater tracking database

[bubbaQ]

New version 0.0.7 is posted.  Link is in OP.

 

New in this version:

 

- new option "-days" so you can compare current SMART data to readings several days ago.

- added read from random block to spin up drive when they are spun down and you specify the -wake ON option

 

Also, with this version, the file program configuration file in the zip file has been renamed, so as to not overwrite a user's customized configuration file.  So you need to either copy "smarthistory.program.conf.sample" to "smarthistory.program.conf" or edit your own "smarthistory.program.conf" with the new parameters in "smarthistory.program.conf.sample"

 

maybe this should be "hidden" then?

 

Maybe... I'm looking at an "ignore" option for certain devices.  Unfortunately, different flashes will react differently to SMART queries, so IDing a flash from SMART results can be tricky.  I'll have to test it a different way.

 

 

[Joe L.]

the interesting part is that I DO NOT have a drive sdg!

 

Yes you do... it is your Flash drive!

 

aaaaaah

 

so it is normal that we don't get SMART data right?

maybe this should be "hidden" then?

 

 

I have three drives that do not return SMART data, the flash drive I boot from, a drive in an external USB enclosure, and an old 8Gig Quantum Fireball that existed before SMART was in use.

 

Perhaps this is where the list of drives in the .conf file might be used.  I do have one idea though to make it easier over time...

 

Can we have an "exclusion" list rather (or in addition to) the inclusion list of disks to be processed?   I expect these three devices to never return SMART data. It would be nice to not have to constantly remember to add a drive to the inclusive list as I add drives in the future. That would be possible with an exclusion list, but not with an inclusion list.   Perhaps both could be combined  in one list /dev/hdb would be included, but !/dev/hdc would not.

 

Since drives change devices over time, perhaps inclusion/exclusion should also be able to be performed by serial number.

 

Joe L.

[dhy8386]

Bubbaq,

 

The drives are still not waking up for me. I am using your latest version and when i run it from command line, it runs fine (actually updated one drive that was up) and prints:

 

No alerts from smarthistory

2 devices active (i have one that i outside array always up), 6 sleeping, 1 did not report SMART data

 

Also, for some reason, i cannot get HTML to work. I have 2 or more reads in the flat files. HTML is on. Tried Joes script, it also does not work.

 

Im attaching my smarthistory results and conf file in the hopes that maybe you can help me.

 

Much thanks.

 

[bubbaQ]

I always planned to add an "exclusion" option.

 

I don't want to do it be /dev/device since those can change.

 

By serial number would be best, but the devices that return no SMART info don't give you a serial number!

 

I can do an option to ignore devices that don't return a serial number, but I'm a little leary of that, since I don't have any "bad" or "failing" drives to test the code on to see what a failing drive does.  I don't want to ignore a drive and report nothing, when it really is a drive that is dying!

[bubbaQ]

@dhy8386:

 

try this commandline:

 

php smarthistory.php -wake ON -report ALL -output HTML -graph LINK

[bubbaQ]

Joe, can you post the output from "smartctl -i" and "hdparm -i"  for each of the 3 devices that don't return SMART info?

[NLS]

Great.

I guess this new version no longer works with current bubbaRAID, right?

 

(time for a .21 now that can call the new standalone smarthistory, LLink and a few more links in first page?)

 

 

[bubbaQ]

No version 0.0.21 for BubbaRaid till I get some feedback (in that thread) as to it working properly!

 

The previous version of smarthistory (PHP) will continue to work on BubbaRaid.  Installing the new standalone version will have no effect.

[NLS]

I know that.

 

[dhy8386]

Bubbaq

 

the zip will not extract properly. Im getting an error on the pc saying enspecified error: 0x80004005. On the mac side i get error 1: operation not permitted

 

 

[bubbaQ]

Version 0.1.03 is now posted.  Link is in OP.

 

Fixed a couple of small bugs in graphs, and Fixed bug in saving ATA_Error_Count when it was 0.

 

Also added "(no data)" for periods when -days option pushes back before first data point.

 

@dhy8386  - I think you must have tried to download it while I was uploading it.  Try again.

[SSD]

Version 0.1.03 is now posted.  Link is in OP.

 

Fixed a couple of small bugs in graphs, and Fixed bug in saving ATA_Error_Count when it was 0.

 

Also added "(no data)" for periods when -days option pushes back before first data point.

 

@dhy8386  - I think you must have tried to download it while I was uploading it.  Try again.

 

The zip file says that it was compressed with an unknown compression method (98).  Winrar also doesn't know how to uncompress the files.  Are you using a new version of zip? Can you reupload in a standard format?

[dhy8386]

Yeh still same issue. Although all your other ones worked fine.

[bubbaQ]

OK... I'll take care of it.

 

In the mean time, I found out some libraries are not being statically linked... so the new compiled version is hosed.

 

Going back to the PHP version in the meantime.

[RobJ]

This project has been added to the UnRAID Add Ons page, here.  It is just a first stab at it, and I welcome BubbaQ's editing or critiquing it, as well as maintaining it later, if he chooses.  I personally think it lacks good operational instructions for new users, in one easy-to-find place, such as how to set up scheduling with cron, how to run it manually, and with what options, and how to produce reports.  Perhaps full operational instructions could be put in a single post, linked from the Wiki page.

 

I haven't had time yet to install and run this myself, wish all of the development would slow down a bit!  But I've been watching it closely, hoping for time to provide some comments/ideas, and time to actually get it monitoring my own drives, sooner rather than later.  I think it is an absolutely terrific idea.

[bubbaQ]

It is still in the prototype/RAD stage.... you are right that it is not really ready yet for novice users -- making it more friendly comes later

 

I wanted to get it out t here so some early testers can start collecting data, and give me some feedback on features to develop. 

 

I like RAD but I don't like developing in a vacuum ... I may go so far down ore road that I get committed to certain things that would have been much easier to change if they had been commented on early on.

[bubbaQ]

I now have smarthistory working as a standalone program, that does not need PHP.  The standalone is much larger, however, about 10MB, because of the libraries that are statically linked.

 

Which is preferable?  PHP version (small), or the standalone executable (large)?

[WeeboTech]

Which is preferable?  PHP version (small), or the standalone executable (large)?

 

My preference would be standalone (I.E. requiring as few packages as possible.)

 

[Joe L.]

I now have smarthistory working as a standalone program, that does not need PHP.  The standalone is much larger, however, about 10MB, because of the libraries that are statically linked.

 

Which is preferable?  PHP version (small), or the standalone executable (large)?

I prefer the smaller "php" version.  First, because I don't have a huge amount of ram, and second, and more important to me,  because I can learn from how you coded in php.

 

Clearly, both have their advantages.

 

Joe L.

 

 

[WeeboTech]

First, because I don't have a huge amount of ram

A standalone version can probably run off the flash drive thereby not using up any ram.

Although having access to the code in PHP clearly does have it's advantages.

[bubbaQ]

how you coded in php.

 

Boy.. now I am *really* embarrassed... someone else looking at my "quick-n-dirty" code! 

 

I suppose I will just include both the executable and php source in the zip... so you can run whatever you want.

 

In my test system, I re-installed my "bad" drive which has about 14000 ATA errors and reports SMART health check as FAILED ... and have been able to clean up some things in the program output.

 

I also changed the logic when using "-report ALL" so that all thresholds that are exceeded will be flagged, and not just ones that have had a change since the last measurement.

 

I also implemented a change for graphs when the parameter being measured has NEVER changed (i.e. is 0 for every date in the history)

 

New version will be posted in the morning.

 

Please continue to post any feature requests you can come up with, as well as defaults for warning/error thresholds for individual SMART parameters you want to suggest.

[Joe L.]

First, because I don't have a huge amount of ram

A standalone version can probably run off the flash drive thereby not using up any ram.

Although having access to the code in PHP clearly does have it's advantages.

Sorry, it may be able to reside on the flash drive, but it is most certainly read into memory when executing.  It will use at least 10 Meg plus whatever it mallocs (allocates) for itself.

 

Having access to the code allows me to learn, and perhaps even to find/fix a bug if one should show itself. (not that any remaining exist)  In any case, I figure a few dozen people have looked under the hood at my code in unMENU, and more than once I had to listen/respond as somebody looked at my code and asked a question or two... It is equally cool that your code enjoy the same scrutiny. 

 

Having "php" available as a installable package is also a huge advantage to other developers. (and smarthistory users will have it installed, as I can't imaging too many people who will not want to run smarthistory... once they learn of it)

 

In any case, having the code available allows others to build on your work... that is perhaps the best reason that php is better than just an executable.

 

Joe L.

[NLS]

I'd say to have both.

Exe for "modders" around and definitely PHP for example your bubbaRAID - both because of the size AND because the additional packages you need COULD be useful for extra services in the future anyway.

 

 

[WeeboTech]

First, because I don't have a huge amount of ram

A standalone version can probably run off the flash drive thereby not using up any ram.

Although having access to the code in PHP clearly does have it's advantages.

Sorry, it may be able to reside on the flash drive, but it is most certainly read into memory when executing.  It will use at least 10 Meg plus whatever it mallocs (allocates) for itself.

This is a given. Any program that runs uses up ram. Programs that install into the root fs use up rootfs ram Plus running ram.

Chances are if it takes 10MB to run a program, it's still going to take close to that amount after it loads all dependent shared libs that are not already available.

 

I would love to know how the EXE version differs from the php interpreted version.

Is the php version compiled into an exe? Perhaps there is a procedure to show people how to compile the source into an exe.

 

[bubbaQ]

I'm using the Roadsend PHP compiler, pcc.

 

The problem is the static linking... which no matter what language you are using, if you static link any high-level language, you get tons of baggage.  Including PCRE adds 4MB of bloat all by itself.

 

I'd love to be able to strip unused library code, but that's not an option with pcc.  Use regexps in your PHP, and wham -- 4 more MB of library code.

 

I'm not sure the loader will load the whole program into memory.  Because of the way static linking works, the code can reside in the executable, but not be loaded from disk until (and unless) needed.  That was the old ".com" format under Dos/Windoes where the whole .com was loaded and shared a 64K code and data segment.  EXEs only load the loader segment, then execute the loader which then loads whatever else is needed.