A framework for notifications a.k.a sending emails, boxcar, notifo, pushover etc

[NAS]

By request of Tom a thread to discuss "A framework for notifications".

 

One area that needs work in unRAID native is the ability to generate alarms and send these as notifications. This thread is specifically only about the later, the ability to send notifications and not the former, generating alarms in the first place.

 

My ultimate hope is that we could pull from someone else's work on this and perhaps feed back fixes rather than redesign the wheel.

 

For instance I believe the notification options in SickBeard are excellent. Ok so it is python and possibly not any use for that reason alone but already it supports:

 

Growl - A cross-platform unobtrusive global notification system.
Prowl - A Growl client for iOS.
Notifo - A platform for push-notifications to either mobile or desktop clients
Libnotify - The standard desktop notification API for Linux/*nix systems. This notifier will only function if the pynotify module is installed (Ubuntu/Debian package python-notify).
Pushover - Pushover makes it easy to send real-time notifications to your Android and iOS devices.
Boxcar - Read your messages where and when you want them! A subscription will be send if needed.
Notify My Android - Notify My Android is a Prowl-like Android App and API that offers an easy way to send notifications from your application directly to your Android device.
Twitter - A social networking and microblogging service, enabling its users to send and read other users' messages called tweets.

 

Noticeably email is missing form this list however a number services give you that as a second alert option i.e. send alarm to pushover it can also send a mail for you. Regardless email is a must.

 

 

This is how I imagine it.  A set of notification modules/accounts that users tick and ALL notifications go to ALL enabled accounts.

 

Thoughts and please lets stay on track to just the notification part not the generation of alarms (that's a whole different discussion)

 

[mr-hexen]

Couch Potato has good notifications as well.

[NAS]

Couch Potato has good notifications as well.

 

isnt that python as well?

[mr-hexen]

indeed.

[NAS]

I suppose there is an arguement to be made that adding python whilst large if it gave us a ready made solution might be worth it.

[WeeboTech]

I looked a while back at some growl applications.

 

For my purpose it was a way to receive and manage messages at the client site easily.

http://www.growlforwindows.com/gfw/default.aspx

 

 

What I had difficulty with was a simple compile-able application that could have been put on unRAID to send the notices out.

...

There seem to be some out there, still requires more research though.

https://github.com/mattn/gntp-send

 

[WeeboTech]

Here's the thing though.  email should be the first and number one addition.

 

unRAID has been around for years without pre-installed email support.

Even if one of the simpler SMTP clients was installed with a configurable front end, that would suffice.

 

I've done so much using email as a simple message system in the past.

 

Send an email and let the client work out where it grabs it from and how it presents the message.

With android clients today, a gmail delivery can provide a phone or desktop client notice.

 

In my example. I had a linux workstation monitoring an alerts mailbox.

It would forward the message to my ROKU soundbridge(s), then it would literally speak the message with a linux text to speech translator from the other room.

This worked so well, I got the same notifications at work, home and I heard it if I was in the other room.

 

http://www.cotrone.com/rob/archives/2006/04/roku_soundbridg_1.html

[WeeboTech]

 

SWEETNESS.

 

14.0 > Desktop > gntp-send (0.3.2)

 

gntp-send is:

* a command line binary for sending notifications to Growl

* a library for integrating Growl into your C or C++ based applications

 

gntp-send and libraries distributed under BSD license.

 

Maintained by: Lionel Young

Keywords: gntp,growl,notification,notify,gntp-send

 

http://slackbuilds.org/repository/14.0/desktop/gntp-send/

 

 

And for those that want to play.

 

 

https://code.google.com/p/unraid-weebotech/downloads/detail?name=gntp-send-0.3.2-i486-1_SBo.tgz

 

 

https://unraid-weebotech.googlecode.com/files/gntp-send-0.3.2-i486-1_SBo.tgz

 

[NAS]

this is why I am a fan of pushover

 

https://pushover.net/

 

right there on their homepage are 4 trivial and easy to use examples. I use the one line curl a lot since it is SO SO easy to script anywhere.

 

I agree most people will want email. It is not longer my personal preference but it is a must have.

 

I think we need some sort of input api which then trigger user defined output modules that way we can bolt on notification services without redesigning every time.

 

I also don't think this will be particularly hard to achieve as a POC.

[WeeboTech]

Is there a desktop app?

 

 

If we could get Tom to include the curl tools/libraries and compile it within PHP, this could be an effective framework.

[NAS]

I believe there are only android and ios clients to receive alerts, obviously anything can push them.

 

The main this here is we build a framework that allows lots of flavour to be added without impacting on any others. There is one thing I kow about this kidn of stuff is if you ask 100 sysadmins you will get 100 different preferences.

 

I am with you though an MTA, curl etc and we are half way there. It also should not reply on emHTTP.

[WeeboTech]

 

Here are my thoughts as I spastically get them out of my head.

 

The downsides of the pushover API are.

 

1. You have to purchase specific clients.

2. It relies on an external website and a custom API/service.

 

It seems simple enough to implement and use, however, I'm not fond of the dependency.

 

If curl is supplied and compiled correctly it can do outbound SMTP.

It can also read mailboxes with POP and IMAP.

PHP can also access POP and IMAP.  CURL libraries can also be compiled within PHP.

 

Most phones come with mail access.

 

Using mail with a few basic tools and compile options there can be an emhttp page to access the same mailbox and notifications sent out via some command line interface.

 

In any case, if mail is a basic denominator, any app can be placed on top of a pop/imap interface to resend the message.  That's how I did it with the soundbridge interface.

 

I had fetchmail running on a mailbox in multiple apps.

One at home, one at work and a text to speech linux box.

 

Mail can be off net, or even local to the machine if courier-imap is installed.

I had courier-imap on my machine before the flood.

It was cool because I would use imap and set up automated filters in thunderbird which would archive my mail from remote mailboxes to the local unraid server's courier-imap store.

 

This is a good potential use for unRAID.

i.e.  A method to archive/store all of your mail on an unRAID box.

 

Almost every NAS I've used has had the ability to send mail.

 

I think a mail interface added to unRAID provides access to great 'free' services.

i.e. smtp client, curl, possibly courier-imap.

 

Once you have storage of these messages in a standard access api, they can be forwarded to whatever service that works best for the end user.

 

At one time I had EXIM installed. I would send out the email alerts.

Using an alias'es file I could install any application which could interpret the mail message and forward it to a program and/or other interface.

 

I would put special keywords in the mail text that would drive the clients reading the messages.

 

Maybe this seems overly complex, but before we go with any of these notification systems/services, the most basic one, email should be implemented.

[NAS]

I do not disagree other than to say mail is possibly the most complex solution to the problem. It is however an absolute requirement but we should consider it be one of many notifiers that can be added in time.

 

I did not know you could use curl to send mail, this is interesting in itself.

 

You also raise an interesting point I did not think to mention... I think it best we should concern ourselves only with pushing notifications out. In terms of email this would be teaching unRAID to send mail and not how to read mail. I am not saying that it should never be done just that it would be out of scope for a unRAID notification service?

[WeeboTech]

You also raise an interesting point I did not think to mention... I think it best we should concern ourselves only with pushing notifications out. In terms of email this would be teaching unRAID to send mail and not how to read mail. I am not saying that it should never be done just that it would be out of scope for a unRAID notification service?

 

Yes. As far as push'ed out notifications, you are correct.

 

As far as teaching unRAID to access mail, that is another layer. 

I was suggesting the possibility that pushed out mail notifications could be accessed by an unRAID browser interface.

It is out of scope as far as pushing out notifications, but not out of scope as part of a 'framework' for notifications.

 

I.E.

1.  an event occurs, a notification is triggered.

2.  The notification gets delivered somewhere.

3.  The notification is accessible by some automated interface. (filter of some sort)

4.  The notification is accessible by a human interface.

5.  History of the notification. 

 

3, 4, 5 is where the IMAP/POP mailbox comes into play.

you can set filters in an email client.

you can be notified by the client or other downstream apps.

you have history.

 

This can be off net (gmail  or some other internet accessible mailbox)

or local net (imap server on local unRAID server).

 

3, 4, 5. is where I liked the idea of the windows growl interface.

I.E. you have a central notification and you can elect what messages you see, plus I think there is history.

 

For the purpose of the push notification, the ability for unRAID to access a mailbox is not a priority and is out of scope with this conversation. It can be added later on if a person chooses it. I.E. If mail were the common denominator and repository for notifications.

[NAS]

Sounds reasonable for a later expansion.

 

SO I suppose one of the first things to talk about is what we would want to pas to the notifications systems. I think we should be fairly regimented on this in the first instance. Later we could have an enhanced field to passing pretty things. I suppose using email as the example again this would be the difference between a line of text in the body and pretty html system.

 

Off the top of my head (and we should ensure this is extensible)

 

Alarm level. Suggest the typical NOTICE, INFO, DEBUG etc

Server name, to cater for multi server houses

Service name e.g. disk subsystem, sabnzb

Short notification e.g. Service started

Long notification e.g. Service started with pid on xx jan 2000 etc etc

 

I would suggest that we would define the charset, min and max lengths of each of these. Some process would validate the data before passing onto the actual notifiers. This way if it is lacking or in breach of the spec the sanity check could pass on a notification saying it is receiving failed messages. This should in theory make the system les prone to user addon developer error. The last thing we need is some random addon breaking the notifier.

 

Fundamentally though we should resist the temptation to try and be all things to all people. It would be better to have robust basic native notifications using something elegant than something convoluted.

 

We should also probably agree on a couple of initial notifers to test with. I would suggest email and pushover likely tick the boxes of the kind of things we should support. More can added later.

 

Apart for the basic field spec we need to deicde if we do this from scratch or use some other donor project such as Sickbeard. I am not a fan of Python and prefer Perl but since neither are there by default we have a choice to make. PHP could help here but we must avoid situations like having to code in awk

[WeeboTech]

Apart for the basic field spec we need to deicde if we do this from scratch or use some other donor project such as Sickbeard. I am not a fan of Python and prefer Perl but since neither are there by default we have a choice to make. PHP could help here but we must avoid situations like having to code in awk

 

I'm not fond of python due to the footprint and it's interpreted nature.

 

I'm fond of perl, but even that will have a footprint and may require external packages.

In some respect, perl provides the same abilities of shell/sed/awk and I've written some major production apps using it. Still it will require footprint to do things with small user code.

 

PHP seems to provide a common interface albeit if other libraries are included.

I.E. Curl.  When doing it via command line it can be pretty ugly to look at, but it will work. Personally I still prefer perl.

 

If curl and the libraries were to be a standard part of unRAID.

PHP and/or C programs or even the curl command line can be written to do the interface.

 

There are also simple mail clients such as SSMTP and ESMTP(I think).

 

While I know that CURL and the libraries will take up space. by including curl, wget would not be required anymore. In addition the abilities of automated network accessible interface program would grow by leaps and bounds.

 

I.E. ftp in/out, sftp in/out, http get,post(post syslogs automatically somewhere), pop/imap/smtp/telnet  many things could be scripted for automated transfer.

[WeeboTech]

 

Alarm level. Suggest the typical NOTICE, INFO, DEBUG etc

Server name, to cater for multi server houses

Service name e.g. disk subsystem, sabnzb

Short notification e.g. Service started

Long notification e.g. Service started with pid on xx jan 2000 etc etc

 

This is pretty much what NAGIOS notification does.

[madburg]

but before we go with any of these notification systems/services, the most basic one, email should be implemented.

First and foremost

 

Many including unMenu I believe have been using SSMTP succesfully on unRAID (supports TLS, and TLS certificates) and requires openssl to be complete. Its flexible to connect to both internal and external mail servers/providers. I have gotten it to work with MS Exchange (and relaying), gmail, aol, etc... based on what support a user selects to use in the GUI config for SSMTP, the configuration file is setup. A Test button would also enable 'Debug=Yes' so you can see the communication logged to know why a connection failed (e.x. TLS not supported)

 

P.S. the only key is getting the config files(s) written properly, ssmtp_config and ssmtp_revaliases (the latter not always required, but again based on users selection in the GUI, the logic (someone scripts, in this case Tom) knows whether to generate the alias file or not and clearly what parameters each of the config files need set. It is not to simple to get all that logic right, speeding_ant got a few wrong, but I didn't have the time to show him).

 

[garycase]

+1 on e-mail.    This seems to be a very basic capability that for some reason has never been integrated in UnRAID.    As WeeboTech noted, once you have an automated e-mail, the user can decide just how to receive/respond to it -- whether on a PC, SmartPhone, tablet, etc.

 

Then all you need is a "config" page to choose which events will generate the e-mails.

 

[NAS]

So in general I think we agree.

 

As far as I am concerned curl makes a case for itself full stop.

If we have a good history with SSMTP then we should continue with that.

 

I think the field definitions are pretty self explanatory and as WeeboTech they pretty much match most notification type things.

 

I propose we consider php as the language of choice. It is certainly there on my installs (although it may have come in with the new webgui). Regardless I cannot see a future where most people unRAID wont have php.

 

Stright from the pushoever website here is an example of how simple this could be

 

curl_setopt_array($ch = curl_init(), array(
  CURLOPT_URL => "https://api.pushover.net/1/messages.json",
  CURLOPT_POSTFIELDS => array(
    "token" => "abc123",
    "user" => "user123",
    "message" => "hello world",
  )));
curl_exec($ch);
curl_close($ch);

 

so in psudo code a proof of concept could almost be as simple as:

 

echo "INFO" "TEST ALERT HEADER" ""MESSAGE BODY" | notification_checker | php pushover.php

 

 

[mr-hexen]

would it make sense to code this such that new notification agents could easily work.

 

thinking out loud here, i'd envision that various services (md drvier, ups, etc.) simply echo messages to a relay / listening demon which in turn sends the basic message to the various notifiers that are installed.

 

in this way, any notifier can be added so long as the relay demon could execute commands with basic switches to send the message.

 

example:

 

UPSd echo's "Power Lost" to relay.demon

relay.demon sends the message to the list of scripts in a pre-determined folder: notifyme.sh [importance] [message] [anything else here]

 

from there on out the scripts or whatever they are know what to do with the arguments and are self sufficient.

 

[WeeboTech]

would it make sense to code this such that new notification agents could easily work.

 

thinking out loud here, i'd envision that various services (md drvier, ups, etc.) simply echo messages to a relay / listening demon which in turn sends the basic message to the various notifiers that are installed.

 

in this way, any notifier can be added so long as the relay demon could execute commands with basic switches to send the message.

 

example:

 

UPSd echo's "Power Lost" to relay.demon

relay.demon sends the message to the list of scripts in a pre-determined folder: notifyme.sh [importance] [message] [anything else here]

 

from there on out the scripts or whatever they are know what to do with the arguments and are self sufficient.

 

I'm not sure we can guarantee all of the apps to calling external programs in a specific manner.

 

I think the NAGIOS toolkit plugin approach works well.

 

There is a monitor layer (plugins) that monitors specific resources.

Upon a condition, OK, WARNING , CRITICAL, UNKNOWN an event is scheduled.

The event then calls a configurable notification layer.

The notification layer then calls the configurable message plugins.

I've done this on many platforms and with many different notification plugins.

 

There is a check plugin, an event, a notificatio plugin.

Each is configurable.

 

We do this in my organization with our own central monitoring station in the operations control room.

 

Other ideas.

 

As far as a listening daemon, there is one already in place It is syslog. all events usually get reported there.

 

Monitor, this can be a log scraper or an advanced syslog daemon such as syslog-ng and rsyslog.

 

I've worked with syslog-ng and could trigger events/apps based on specific piped messages to a perl co-process.

[NAS]

this looks simple enough

 

http://www.johnandcailin.com/blog/john/how-setup-real-time-email-notification-critical-syslog-events

 

we could extend that simply

 

Edit: the more i look at this the more elegant it looks. What I am not sure yet is how to make a bunch of notifiers work reliably in parallel when an event is triggered

[mr-hexen]

i think this should be part of the plugin framework as well, providing a method for plugin authors to have their apps send alerts when it is needed.

 

Right now, transmission for example, has its own log. Some plugins send info the the standard syslog (i.e. UPSd) while others do not. Granted transmission doesnt need to send alerts, but crashplan would, right?

 

a syslog scrapper will no doubt pick up false errors though, unMenu is a perfect example of how false "errors" are flagged. I'd imagine a scrapper will do the same but instead of adding colour , it would send alerts, emails, sms, etc.

[NAS]

Addons would have to changed to use the notification system and not the other way around. We shouldn't end up in a situation where the core notification system needs some change (however minor) to cater for an addon.

 

Now that is not to say the addon could not add its own alerting rules to notification system but it would still need to send messages to syslog.

 

We should avoid at all costs trying to build a system that monitors any number of log files.