About the author

My name is Solène Rapenne. I like to learn and share my knowledge with other. With this blog I can share my experiences and issues. Some of my interests : '(BSD Lisp Emacs cli-tool Web-infrastructure Gaming Crossbow). I love % and lambda characters.

Contact : solene on Freenode or solene+www at dataswamp dot org

This website is generated using cl-yag. A gopher version is available here

Monitor your systems with reed-alert

Written by Solène, on 17 January 2018.
Tags: #unix #monitoring #lisp

This article will present my software reed-alert, it checks user-defined states and send user-defined notification. I made it really eay to use but still configurable and extensible.

Description

reed-alert is not a monitoring tool producing graph or storing values. It does a job sysadmins are looking for because there are no alternative product (the alternatives comes from a very huge infrastructure like Zabbix so it’s not comparable).

From its configuration file, reed-alert will check various states and then, if it fails, will trigger a command to send a notification (totally user-defined).

Fetch it

This is a open-source and free software released under MIT license, you can get it with the following command :

git clone git://bitreich.org/reed-alert
cd reed-alert

A README file is available as documentation to describe how to use it, but we will see here how to get started quickly.

You will find a few files there, reed-alert is a Common LISP software and it has been choosed for (I hope) good reasons that the configuration file is plain Common LISP.

There is a config file looking like a real world example named config.lisp.sample and another configuration file I use for testing named example.lisp containing lot of cases.

Let’s start

In order to use reed-alert we only need to create a new configuration file and then add a cron job.

Configuration

We are going to see how to configure reed-alert. You can find more explanations or details in the README file.

Alerts

We have to configure two kind of parameters, first we need to set-up a way to receive alerts, easiest way to do so is by sending a mail with “mail” command. Alerts are declared with the function alert and as parameters the alert name and the command to be executed. Some variables are replaced with values from the probe, in the README file you can find the list of probes, it looks like %date% or %params%.

In Common LISP functions are called by using a parenthesis before its name and until the parenthesis is closed, we are giving its parameters.

Example :

(alert mail "echo 'problem on %hostname%' | mail me@example.com")

One should take care about nesting quotes here.

reed-alert will fork a shell to start the command, so pipes and redirections works. You can be creative when writing alerts that :

  • use a SMS service
  • write a script to post on a forum
  • publishing a file on a server
  • send text to irc with ii client

Checks

Now we have some alerts, we will configure some checks in order to make reed-alert useful. It uses probes which are pre-defined checks with parameters, a probe could be “has this file not been updated since N minutes ?” or “Is the disk space usage of partition X more than Y ?”

I choosed to name the function “=>” to make a check, it isn’t a name and reminds an item or something going forward. Both previous example using our previous mail notifier would look like :

(=> mail file-updated :path "/program/file.generated" :limit "10")
(=> mail disk-usage   :limit 90)

It’s also possible to use shell commands and check the return code using the command probe, allowing the user to define useful checks.

(=> mail command :command "echo '/is-this-gopher-server-up?' | nc -w 3 dataswamp.org 70"
                 :desc "dataswamp.org gopher server")

We use echo + netcat to check if a connection to a socket works. The :desc keyword will give a nicer name in the output instead of just “COMMAND”.

Garniture

We wrote the minimum required to configure reed-alert, now the configuration file requires only two more lines, so your my-config.lisp file should looks like the following :

(load "functions.lisp")
(alert mail "echo 'problem on %hostname%' | mail me@example.com")
(=> mail file-updated :path "/program/file.generated" :limit "10")
(=> mail disk-usage   :limit 90)
(quit)

The first line is mandatory to make things working, the last line will prevent the lisp interpreter to stay in REPL (user interactive input).

Now, you can start it every 5 minutes from a crontab with this :

*/5 * * * * ( cd /home/user/projects/reed-alert/ ; sbcl --load my-config.lisp )

If you prefer to use ecl :

*/5 * * * * ( cd /home/user/projects/reed-alert/ ; ecl --load my-config.lisp )

The time between each run is up to you, depending on what you monitor.

Important

By default, when a check returns a failure, reed-alert will only trigger the notifier associated once it reach the 3rd failure. And then, will notify again when the service is back (the variable %state% is replaced by start or end to know if it starts or stops.)

This is to prevent reed-alert to send a notification each time it checks, there is absolutely no need for this for most users.

The number of failures before triggering can be modified by using the keyword “:try” as in the following example :

(=> mail disk-usage :limit 90 :try 1)

In this case, you will get notified at the first failure of it.

The number of failures of failed checks is stored in files (1 per check) in the “states/” directory of reed-alert working directory.

Showing some Common Lisp features

Written by Solène, on 05 December 2017.
Tags: #lisp

Introduction : comparing LISP to Perl and Python

We will refer to Common LISP as CL in the following article.

I wrote it to share what I like about CL. I’m using Perl to compare CL features. I am using real world cases for the average programmer. If you are a CL or perl expert, you may say that some example could be rewritten with very specific syntax to make it smaller or faster, but the point here is to show usual and readable examples for usual programmers.

This article is aimed at people with programming interest, some basis of programming knowledge are needed to understand the following. If you know how to read C, Php, Python or Perl it should be enough. Examples have been choosed to be easy.

I thank my friend killruana for his contribution as he wrote the python code.

Variables

Scope : global

Common Lisp code

(defparameter *variable* "value")

Defining a variable with defparameter on top-level (= outside of a function) will make it global. It is common to surround the name of global variables with * character in CL code. This is only for readability for the programmer, the use of * has no incidence.

Perl code

my $variable = "value";

Python code

variable = "value";

Scope : local

This is where it begins interesting in CL. Declaring a local variable with let create a new scope with parenthesis where the variable isn’t known outside of it. This prevent doing bad things with variables not set or already freed. let can define multiple variables at once, or even variables depending on previously declared variables using let*

Common Lisp code

(let ((value (http-request)))
  (when value
    (let* ((page-title (get-title value))
           (title-size (length page-title)))
      (when page-title
        (let ((first-char (subseq page-title 0 1)))
          (format t "First char of page title is ~a~%" first-char))))))

Perl code

{
    local $value = http_request;
    if($value) {
        local $page_title = get_title $value;
        local $title_size = get_size $page_title;
        if($page_title) {
            local $first_char = substr $page_title, 0, 1;
            printf "First char of page title is %s\n", $first_char;
        }
    }
}

The scope of a local value is limited to the parent curly brakets, of a if/while/for/foreach or plain brakets.

Python code

if True:
    hello = 'World'
print(hello) # displays World

There is no way to define a local variable in python, the scope of the variable is limited to the parent function.

Printing and format text

CL has a VERY powerful function to print and format text, it’s even named format. It can even manage plurals of words (in english only) !

Common Lisp code

(let ((words (list "hello" "Dave" "How are you" "today ?")))
  (format t "~{~a ~}~%" words))

format can loop over lists using ~{ as start and ~} as end.

Perl code

my @words = @{["hello", "Dave", "How are you", "today ?"]};
foreach my $element (@words) {
    printf "%s ", $element;
}
print "\n";

Python code

# Printing and format text
# Loop version
words = ["hello", "Dave", "How are you", "today ?"]
for word in words:
    print(word, end=' ')
print()

# list expansion version
words = ["hello", "Dave", "How are you", "today ?"]
print(*words)

Functions

function parameters : rest

Sometimes we need to pass to a function a not known number of arguments. CL supports it with &rest keyword in the function declaration, while perl supports it using the @_ sigil.

Common Lisp code

(defun my-function(parameter1 parameter2 &rest rest)
  (format t "My first and second parameters are ~a and ~a.~%Others parameters are~%~{    - ~a~%~}~%"
          parameter1 parameter2 rest))

(my-function "hello" "world" 1 2 3)

Perl code

sub my_function {
    my $parameter1 = shift;
    my $parameter2 = shift;
    my @rest = @_;

    printf "My first and second parameters are %s and %s.\nOthers parameters are\n",
        $parameter1, $parameter2;

    foreach my $element (@rest) {
        printf "    - %s\n", $element;
    }
}

my_function "hello", "world", 0, 1, 2, 3;

Python code

def my_function(parameter1, parameter2, *rest):
    print("My first and second parameters are {} and {}".format(parameter1, parameter2))
    print("Others parameters are")
    for parameter in rest:
        print(" - {}".format(parameter))

my_function("hello", "world", 0, 1, 2, 3)

The trick in python to handle rests arguments is the wildcard character in the function definition.

function parameters : named parameters

CL supports named parameters using a keyword to specify its name. While it’s not at all possible on perl. Using a hash has parameter can do the job in perl.

CL allow to choose a default value if a parameter isn’t set, it’s harder to do it in perl, we must check if the key is already set in the hash and give it a value in the function.

Common Lisp code

(defun my-function(&key (key1 "default") (key2 0))
  (format t "Key1 is ~a and key2 (~a) has a default of 0.~%"
          key1 key2))

(my-function :key1 "nice" :key2 ".Y.")

There is no way to pass named parameter to a perl function. The best way it to pass a hash variable, check the keys needed and assign a default value if they are undefined.

Perl code

sub my_function {
    my $hash = shift;

    if(! exists $hash->{key1}) {
        $hash->{key1} = "default";
    }

    if(! exists $hash->{key2}) {
        $hash->{key2} = 0;
    }

    printf "My key1 is %s and key2 (%s) default to 0.\n",
        $hash->{key1}, $hash->{key2};
}

my_function { key1 => "nice", key2 => ".Y." };

Python code

def my_function(key1="default", key2=0):
    print("My key1 is {} and key2 ({}) default to 0.".format(key1, key2))

my_function(key1="nice", key2=".Y.")

Loop

CL has only one loop operator, named loop, which could be seen as an entire language itself. Perl has do while, while, for and foreach.

loop : for

Common Lisp code

(loop for i from 1 to 100
   do
     (format t "Hello ~a~%" i))

Perl code

for(my $i=1; $i <= 100; $i++) {
    printf "Hello %i\n";
}

Python code

for i in range(1, 101):
   print("Hello {}".format(i))

loop : foreach

Common Lisp code

(let ((elements '(a b c d e f)))
  (loop for element in elements
     counting element into count
     do
       (format t "Element number ~s : ~s~%"
               count element)))

Perl code

# verbose and readable version
my @elements = @{['a', 'b', 'c', 'd', 'e', 'f']};
my $count = 0;
foreach my $element (@elements) {
    $count++;
    printf "Element number %i : %s\n", $count, $element;
}

# compact version
for(my $i=0; $i<$#elements+1;$i++) {
    printf "Element number %i : %s\n", $i+1, $elements[$i];
}

Python code

# Loop foreach
elements = ['a', 'b', 'c', 'd', 'e', 'f']
count = 0
for element in elements:
    count += 1
    print("Element number {} : {}".format(count, element))

# Pythonic version
elements = ['a', 'b', 'c', 'd', 'e', 'f']
for index, element in enumerate(elements):
    print("Element number {} : {}".format(index, element))

LISP only tricks

Store/restore data on disk

The simplest way to store data in LISP is to write a data structure into a file, using print function. The code output with print can be evaluated later with read.

Common Lisp code

(defun restore-data(file)
  (when (probe-file file)
    (with-open-file (x file :direction :input)
      (read x))))

(defun save-data(file data)
  (with-open-file (x file
                     :direction :output
                     :if-does-not-exist :create
                     :if-exists :supersede)
    (print data x)))

;; using the functions
(save-data "books.lisp" *books*)
(defparameter *books* (restore-data "books.lisp"))

This permit to skip the use of a data storage format like XML or JSON. Common LISP can read Common LISP, this is all it needs. It can store objets like arrays, lists or structures using plain text format. It can’t dump hash tables directly.

Creating a new syntax with a simple macro

Sometimes we have cases where we need to repeat code and there is no way to reduce it because it’s too much specific or because it’s due to the language itself. Here is an example where we can use a simple macro to reduce the written code in a succession of conditions doing the same check.

We will start from this

Common Lisp code

(when value
  (when (string= line-type "3")
    (progn
      (print-with-color "error" 'red line-number)
      (log-to-file "error")))
  (when (string= line-type "4")
    (print-with-color text))
  (when (string= line-type "5")
    (print-with-color "nothing")))

to this, using a macro

Common Lisp code

(defmacro check(identifier &body code)
  `(progn
     (when (string= line-type ,identifier)
     ,@code)))

(when value
  (check "3"
         (print-with-color "error" 'red line-number)
         (log-to-file "error"))
  (check "4"
         (print-with-color text))
  (check "5"
         (print-with-color "nothing")))

The code is much more readable and the macro is easy to understand. One could argue that in another language a switch/case could work here, I choosed a simple example to illustrate the use of a macro, but they can achieve more.

Create powerful wrappers with macros

I’m using macros when I need to repeat code that affect variables. A lot of CL modules offers a structure like with-something, it’s a wrapper macro that will do some logic like opening a database, checking it’s opened, closing it at the end and executing your code inside.

Here I will write a tiny http request wrapper, allowing me to write http request very easily, my code being able to use variables from the macro.

Common Lisp code

(defmacro with-http(url)
  `(progn
     (multiple-value-bind (content status head)
         (drakma:http-request ,url :connection-timeout 3)
       (when content
         ,@code))))

(with-http "https://dataswamp.org/"
  (format t "We fetched headers ~a with status ~a. Content size is ~d bytes.~%"
          status head (length content)))

In Perl, the following would be written like this

Perl code

sub get_http {
    my $url = $1;
    my %http = magic_http_get $url;
    if($http{content}) {
        return %http;
    } else {
        return undef;
    }
}

{
    local %data = get_http "https://dataswamp.org/";
    if(%data) {
        printf "We fetched headers %s with status %d. Content size is %d bytes.\n",
            $http{headers}, $http{status}, length($http{content});
    }
}

The curly brackets are important there, I want to emphase that the local %data variable is only available inside the curly brackets. Lisp is written in a successive of local scope and this is something I really like.

Python code

import requests
with requests.get("https://dataswamp.org/") as fd:
    print("We fetched headers %s with status %d. Content size is %s bytes." \
                % (list(fd.headers.keys()), fd.status_code, len(fd.content)))

New cl-yag version

Written by Solène, on 21 January 2017.
Tags: #lisp #cl-yag

Today I just updated my tool cl-yag that implies a slightly change on my website. Now, on the top of this blog, you can see a link “Index of articles”. This page only display articles titles, without any text from the article.

Cl-yag is a tool to generate static website like this one. It’s written in Common LISP. For reminder, it’s also capable of producing both html and gopher output now.

If you don’t know what Gopher is, you will learn a lot reading the following links Wikipedia : Gopher (Protocol) and Why is gopher still relevant

Common LISP : How to open an SSL / TLS stream

Written by Solène, on 26 September 2016.
Tags: #lisp #network

Here is a tiny code to get a connection to an SSL/TLS server. I am writing an IRC client and an IRC bot too and it’s better to connect through a secure channel.

This requires usocket and cl+ssl :

(usocket:with-client-socket (socket stream *server* *port*)
  (let ((ssl-stream (cl+ssl:make-ssl-client-stream stream
                               :external-format '(:iso-8859-1 :eol-style :lf)
                               :unwrap-stream-p t
                               :hostname *server*)))
    (format ssl-stream "hello there !~%")
    (force-output ssl-stream)))

Port of the week : stumpwm

Written by Solène, on 21 September 2016.
Tags: #stumpwm #portoftheweek #wm #lisp

When I started port of the week articles I was planning to write an article every week but now I don’t have much ports too speak about.

Today is about x11/stumpwm ! I already wrote about this window manager but since this month, it’s now available in ports on -current and later in 6.1 release.

Website now compatible gopher !

Written by Solène, on 11 August 2016.
Tags: #gopher #network #lisp

My website is now available with Gopher protocol ! I really like this protocol. If you don’t know it, I encourage you reading this page : Why is Gopher still relevant ?.

This has been made possible by modifying the tool generating the website pages to make it generating gopher compatible pages. This was a bit of work but I am now proud to have it working.

I have also made a “big” change into the generator, it now rely on a “markdown-to-html” tool which sadden me a bit. Before that, I was using ham-mode in emacs which was converting html on the fly to markdown so I can edit in markdown, and was exporting into html on save. This had pros and cons. Nothing more than a lisp interpreter was needed on the system generating the files, but I was sometimes struggling with ham-mode because the conversion was destructive. Multiple editing in a row of the same file was breaking code blocks, because it wasn’t exported the same way each time until it wasn’t a code block anymore. There are some articles that I update sometimes to keep it up-to-date or fix an error in it, and it was boring to fix the code everytime. Having the original markdown text was mandatory for gopher export, and is now easier to edit with any tool.

There is a link to my gopher site on the right of this page. You will need a gopher client to connect to it. There is an android client working, also Firefox can have an extension to become compatible (gopher support was native before it have been dropped). You can find a list of clients on Wikipedia.

Gopher is nice, don’t let it die.

My Stumpwm config on OpenBSD

Written by Solène, on 06 June 2016.
Tags: #openbsd #freebsd #wm #lisp #stumpwm

I want to talk about stumpwm, a window manager written in Common LISP. I think one must at least like emacs to like stumpwm. Stumpwm is a tiling window manager one which you create “panes” on the screen like windows on Emacs. A single pane takes 100% of the screen, then you can split it into 2 panes vertically or horizontally and resize it, and you can split again and again. There is no “automatic” tiling. By default, if you have ONE pane, you will only have ONE window displayed, this is a bit different that others tiling wm I had tried. Also, virtual desktops are named groups, nothing special here, you can create/delete groups and rename it. Finally, stumpwm is not minimalistic.

To install it, you need to get the sources of stumpwm, install a common lisp interpreter (sbcl, clisp, ecl etc…), install quicklisp (which is not in packages), install the quicklisp packages cl-ppcre and clx and then you can compile stumpwm, that will produce a huge binary which embedded a common lisp interpreter (that’s a way to share common lisp executables, the interpreter can create an executable from itself and include the files you want to execute). I would like to make a package for OpenBSD but packaging quicklisp and its packages seems too difficult for me at the moment.

Here is my config file in ~/.stumpwmrc.

Updated : 23th january 2018

(defun chomp(text) (subseq text 0 (- (length text) 1)))
(defmacro cmd(command) `(progn `(:eval (chomp (stumpwm:run-shell-command ,,command t)))))

(defun get-latence()
  (let ((now (get-universal-time)))
    (when (> (- now *latence-last-update* ) 30)
      (setf *latence-last-update* now)
      (when (probe-file "/tmp/latenceresult")
        (with-open-file (x "/tmp/latenceresult"
                           :direction :input)
          (setf *latence* (read-line x))))))
  *latence*)

(defvar *latence-last-update* (get-universal-time))
(defvar *latence* "nil")


(set-module-dir "~/dev/stumpwm-contrib/")
(stumpwm:run-shell-command "setxkbmap fr")
(stumpwm:run-shell-command "feh --bg-fill red_damask-wallpaper-1920x1080.jpg")

(defvar color1 "#886666")
(defvar color2 "#222222")

(setf
 stumpwm:*mode-line-background-color* color2 
 stumpwm:*mode-line-foreground-color* color1
 stumpwm:*mode-line-border-color* "#555555"
 stumpwm:*screen-mode-line-format* (list "%g | %v ^>^7 %B | " '(:eval (get-latence)) "ms %d    ")
 stumpwm:*mode-line-border-width* 1
 stumpwm:*mode-line-pad-x* 6
 stumpwm:*mode-line-pad-y* 1
 stumpwm:*mode-line-timeout* 5
 stumpwm:*mouse-focus-policy* :click
 ;;stumpwm:*group-format* "%n·%t
 stumpwm:*group-format* "%n"
 stumpwm:*time-modeline-string* "%H:%M"
 stumpwm:*window-format* "^b^(:fg \"#7799AA\")<%25t>"
 stumpwm:*window-border-style* :tight
 stumpwm:*normal-border-width* 1
 )


(stumpwm:set-focus-color "#7799CC")
(stumpwm:grename "Alpha")
(stumpwm:gnewbg "Beta")
(stumpwm:gnewbg "Tau")
(stumpwm:gnewbg "Pi")
(stumpwm:gnewbg "Zeta")
(stumpwm:gnewbg "Teta")
(stumpwm:gnewbg "Phi")
(stumpwm:gnewbg "Rho")

(stumpwm:toggle-mode-line (stumpwm:current-screen) (stumpwm:current-head))

(set-prefix-key (kbd "M-a"))

(define-key *root-map* (kbd "c")            "exec urxvtc")
(define-key *root-map* (kbd "RET")          "move-window down")
(define-key *root-map* (kbd "z")            "fullscreen")

(define-key *top-map* (kbd "M-&")           "gselect 1")
(define-key *top-map* (kbd "M-eacute")      "gselect 2")
(define-key *top-map* (kbd "M-\"")          "gselect 3")
(define-key *top-map* (kbd "M-quoteright")  "gselect 4")
(define-key *top-map* (kbd "M-(")           "gselect 5")
(define-key *top-map* (kbd "M--")           "gselect 6")
(define-key *top-map* (kbd "M-egrave")      "gselect 7")
(define-key *top-map* (kbd "M-underscore")  "gselect 8")

(define-key *top-map* (kbd "s-l")           "exec slock")
(define-key *top-map* (kbd "s-t")           "exec urxvtc")
(define-key *top-map* (kbd "M-S-RET")       "exec urxvtc")
(define-key *top-map* (kbd "M-C")           "exec urxvtc")

(define-key *top-map* (kbd "s-s")           "exec /home/solene/dev/screen_up.sh")

(define-key *top-map* (kbd "s-Left")        "gprev")
(define-key *top-map* (kbd "s-Right")       "gnext")

(define-key *top-map* (kbd "M-ISO_Left_Tab")"other")
(define-key *top-map* (kbd "M-TAB")         "fnext")
(define-key *top-map* (kbd "M-twosuperior") "next-in-frame")

(load-module "battery-portable")
(load-module "stumptray")

I use a function to get latency from a script that is started every 20 seconds to display the network latency or nil if I don’t have internet access.

I use rxvt-unicode daemon (urxvtd) as a terminal emulator, so the terminal command is urxvtc (for client), it’s lighter and faster to load.

I also use a weird “alt+tab” combination :

  • Alt+tab switch between panes
  • Alt+² (the key above tab) circles windows in the current pane
  • Alt+Shift+Tab switch to the previous windows selected

StumpWM website

What it this blog ?

Written by Solène, on 29 April 2016.
Tags: #lisp #cl-yag

This website is generated using cl-yag, which I am the author.

I have some work to do to cl-yag but I want to keep it relatively easy to use and clean

Todo

  • less code : that should be possible and an interesting task
  • tests to check if modifying don’t break anything (but I have no idea how to do that)

Done

  • Host it on github with default settings
  • Permit to type A
  • Right panel
  • Make the code easier with macros
  • Find a way to use markdown
  • Make RSS
  • Tag list
  • Tags on articles
  • Gopher export