Just noticed that my telnetd had been killed off by the Out Of Memory Killer (OOM Killer).

 

Issue is that I always only notice this when I need something that isn't there anymore, so I have made a small hourly cron script that will check for errors and tells the SimpleFeatures interface to show it, the script is easy to modify for other messages also and might be valuable for more users which is why I place it here.

 

Another thing that happens to me sometimes is duplicate files, this is listed in the syslog but since this is not a thing I watch regularly they might exist for a long time (not a real issue btw), I have also made a monitor to see this using the simplefeatures system.

 

I am using very rudimentary linux commands and it quite possibly can be done a lot nicer. One of the things I had to solve is that I do an hourly syslog check but an existing error condition that is in the syslog will remain there and will get noticed every time untill a reboot occurs. That is not usefull so I implemented a way to have an hourly syslog on the system and a seperate combined one that is located on the array, this way I have a syslog max 60 minutes old on the array (quite easy, also in case of crashes) and a syslog of max. 1 hour on the system.

 

The pgrep command checks for the PID of the syslogdaemon and restarts it, this will make sure the syslog will continue to get filled.

 

Create a file called CheckOOM and place it in the config dir of your flashdrive, contents of the file should be as follows (dont make it with notepad, use EditPad or VI on console):

 

#!/bin/bash

# Rotate syslog
mv /var/log/syslog /var/log/syslog_hour
cat /var/log/syslog_hour >> /mnt/disk8/Incomming/syslog_all
pgrep -f "/usr/sbin/syslogd" | while read PID; do kill -hup $PID; done


# check for OOM errors in the log
if grep -m1 oom /var/log/syslog>/dev/null
then
/usr/local/emhttp/plugins/simpleFeatures/include/Notify.sh -plugin "Logevent" -subject "OOM Error" -description "Out Of Memory message found in syslog, check syslog for specifics" -importance "warning"
else
echo Not there.
fi

# check for duplicate files in the log
if grep -m1 duplicate /var/log/syslog>/dev/null
then
/usr/local/emhttp/plugins/simpleFeatures/include/Notify.sh -plugin "Logevent" -subject "Duplicate detected" -description "Duplicate file message found in syslog, check syslog for specifics" -importance "warning"
else
echo Not there.
fi

 

Now add the following lines to your GO file:

 

# Setting up OOM monitor
cp /boot/config/CheckOOM /etc/cron.hourly/CheckOOM
chmod +x /etc/cron.hourly/CheckOOM

 

The "CheckOOM" file script structure is real easy, so if you need to monitor something else just copy the block and change the texts, it will work.

Memory usage can be tracked with the "system stats" plugin of SimpleFeatures.

 

This is how it looks like on my system. I have been doing simultaneous disk preclear actions and a parity sync, which is causing the cached part of the memory usage to jump up and down, but the used memory part stays fairly stable...

 

Note: this is the latest SF version, coming to you soon!

 

 

 

 

Jup, I use that (though I guess this is a preview of the new version, yours looks more flashy then mine), but it does not show OOM errors, especially when there is a low memory OOM event then there will still be memory "left"..

 

And without you knowing your shares, emhttp or telnet might be down.. This makes that visible, at least for me this is nice.

 

I am doing a same kind of thing for checking if there are messages of duplicate files in the log, can be part of the same hourly check ofcourse.

 

Problem now is that the procedure listed in the thread tells that I need to use the following syntax:

 

usr/local/emhttp/plugins/Notifications/notify.sh -plugin "Logevent" -subject "OOM Error" -description "Out Of Memory message found in syslog, check syslog for specifics" -importance "warning"

 

That script however does not appear to exist.. There is no /Notifications under plugins, there is a simpleFeatures dir, but I cannot find a "notify.sh", which script should I send this syntax to ?

 

Folders have been rearranged in the later versions of SF.

 

You need to use /usr/local/emhttp/plugins/simpleFeatures/include/Notify.sh

 

(note the upper/lower casing)

Folders have been rearranged in the later versions of SF.

 

You need to use /usr/local/emhttp/plugins/simpleFeatures/include/Notify.sh

 

(note the upper/lower casing)

 

Thanks, that works, code in the first post is corrected so it will work.

You should change the second search string to "duplicate" instead of "oom"...

 

Otherwise a nifty addition

Thanks for the pointer, I also added something to make sure the same stuff does not get reported every hour again.. Only thing I am not sure of is if a syslog file will be automatically made when it is not present..

Just found out that the syslog is not automatically restarted... I have changed the base code in the original post to make it work.

 

So the "good" version will allways be in the Original Post (OP)

A different approach (better not touching the original syslog file)...

 

#!/bin/bash

# Create rotating syslog
if [ ! -f /tmp/syslog ]
then
cp /var/log/syslog /tmp/syslog
fi

# Check out-of-memory messages
if diff -e /tmp/syslog /var/log/syslog | grep -q -m 1 'oom'
then
/usr/local/emhttp/plugins/simpleFeatures/include/Notify.sh -plugin "Logevent" -subject "OOM Error" -description "Out Of Memory message found in syslog, check syslog for details" -importance "warning"
fi

# Check duplicate files
if diff -e /tmp/syslog /var/log/syslog | grep -q -m 1 'duplicate'
then
/usr/local/emhttp/plugins/simpleFeatures/include/Notify.sh -plugin "Logevent" -subject "Duplicate Error" -description "Duplicate file message found in syslog, check syslog for details" -importance "warning"
fi

# Create rotating syslog
cp /var/log/syslog /tmp/syslog

Thats crowd development for you !!

 

A lot better, thanks !  (Changed my version immediately on my system)

 

Now would it not be great if all this was a simplefeatures function ?  The ability to add Notifications on syslog (and possibly other logs) via the interface ?  Possibly even the option to "import" a seperate list of possible notifications from a central storage somewhere ?

 

Version 2 could then even take into account what plugins you are running and add the most appropriate watches  automatically while retaining the possibility to turn them off and on as well as add new ones...

 

I still like my side-effect of retaining a syslog in-array by the way so I will add that in for myself (btw: also a nice simplefeatures idea ;-)

Instead of looking for the word "oom" in the log its better to look for "oom-killer", should have the same effect but avoids getting notifies because the mover has moved "here comes the bOOM"  ;-)