This article will present my software reed-alert, it checks
user-defined states and send user-defined notification. I made it
really easy to use but still configurable and extensible.
Description
reed-alert is not a monitoring tool producing graph or storing
values. It does a job sysadmins are looking for because there are no
alternative product (the alternatives comes from a very huge
infrastructure like Zabbix so it’s not comparable).
From its configuration file, reed-alert will check various states
and then, if it fails, will trigger a command to send a notification
(totally user-defined).
Fetch it
This is a open-source and free software released under MIT license,
you can install it with the following command:
# git clone git://bitreich.org/reed-alert
# cd reed-alert
# make
# doas make install
This will install a script reed-alert in /usr/local/bin/ with the
default Makefile variables. It will try to use ecl and then sbcl if
ecl is not installed.
A README file is available as documentation to describe how to use
it, but we will see here how to get started quickly.
You will find a few files there, reed-alert is a Common LISP
software and it has been chose for (I hope) good reasons that the
configuration file is plain Common LISP.
There is a configuration file looking like a real world example named
config.lisp.sample and another configuration file I use for testing
named example.lisp containing lot of cases.
Let’s start
In order to use reed-alert we only need to create a new
configuration file and then add a cron job.
Configuration
We are going to see how to configure reed-alert. You can find more
explanations or details in the README file.
Alerts
We have to configure two kind of parameters, first we need to set-up a
way to receive alerts, easiest way to do so is by sending a mail with
“mail” command. Alerts are declared with the function alert and as
parameters the alert name and the command to be executed. Some
variables are replaced with values from the probe, in the README
file you can find the list of probes, it looks like %date% or
%params%.
In Common LISP functions are called by using a parenthesis before its
name and until the parenthesis is closed, we are giving its
parameters.
Example:
(alert mail "echo 'problem on %hostname%' | mail me@example.com")
One should take care about nesting quotes here.
reed-alert will fork a shell to start the command, so pipes and
redirection works. You can be creative when writing alerts that:
- use a SMS service
- write a script to post on a forum
- publishing a file on a server
- send text to IRC with ii client
Checks
Now we have some alerts, we will configure some checks in order to
make reed-alert useful. It uses probes which are pre-defined
checks with parameters, a probe could be “has this file not been
updated since N minutes ?” or “Is the disk space usage of partition X
more than Y ?”
I chose to name the function “=>” to make a check, it isn’t a name
and reminds an item or something going forward. Both previous example
using our previous mail notifier would look like:
(=> mail file-updated :path "/program/file.generated" :limit "10")
(=> mail disk-usage :limit 90)
It’s also possible to use shell commands and check the return code
using the command probe, allowing the user to define useful
checks.
(=> mail command :command "echo '/is-this-gopher-server-up?' | nc -w 3 dataswamp.org 70"
:desc "dataswamp.org gopher server")
We use echo + netcat to check if a connection to a socket works. The
:desc keyword will give a nicer name in the output instead of just
“COMMAND”.
Garniture
We wrote the minimum required to configure reed-alert, now the
configuration file so your my-config.lisp file should looks like
this:
(alert mail "echo 'problem on %hostname%' | mail me@example.com")
(=> mail file-updated :path "/program/file.generated" :limit "10")
(=> mail disk-usage :limit 90)
Now, you can start it every 5 minutes from a crontab with this:
*/5 * * * * ( reed-alert /path/to/my-config.lisp )
If you prefer to use ecl:
*/5 * * * * ( reed-alert /path/to/my-config.lisp )
The time between each run is up to you, depending on what you monitor.
Important
By default, when a check returns a failure, reed-alert will only
trigger the notifier associated once it reach the 3rd failure. And
then, will notify again when the service is back (the variable %state%
is replaced by start or end to know if it starts or stops.)
This is to prevent reed-alert to send a notification each time it
checks, there is absolutely no need for this for most users.
The number of failures before triggering can be modified by using the
keyword “:try” as in the following example:
(=> mail disk-usage :limit 90 :try 1)
In this case, you will get notified at the first failure of it.
The number of failures of failed checks is stored in files (1 per
check) in the “states/” directory of reed-alert working directory.