About me: My name is Solène Rapenne. I like learning and sharing experiences about IT stuff. Hobbies: '(BSD OpenBSD h+ Lisp cmdline gaming internet-stuff Crossbow). I love percent and lambda characters. OpenBSD developer solene@.

Contact me: solene on Freenode, solene+www at dataswamp dot org or solene@bsd.network (mastodon)

Common LISP awk macro for easy text file operations

Written by Solène, on 04 February 2020.
Tags: #awk #lisp

I like Common LISP and I also like awk. Dealing with text files in Common LISP is often painful. So I wrote a small awk like common lisp macro, which helps a lot dealing with text files.

Here is the implementation, I used the uiop package for split-string function, it comes with sbcl. But it's possible to write your own split-string or reused the infamous split-str function shared on the Internet.

(defmacro awk(file separator &body code)
  "allow running code for each line of a text file,
   giving access to NF and NR variables, and also to
   fields list containing fields, and line containing $0"
    `(progn
       (let ((stream (open ,file :if-does-not-exist nil)))
         (when stream
           (loop for line = (read-line stream nil)
              counting t into NR
              while line do
                (let* ((fields (uiop:split-string line :separator ,separator))
                       (NF (length fields)))
                  ,@code))))))

It's interesting that the "do" in the loop could be replaced with a "collect", allowing to reuse awk output as a list into another function, a quick example I have in mind is this:

;; equivalent of awk '{ print NF }' file | sort | uniq
;; for counting how many differents fields long line we have
(uniq (sort (awk "file" " " NF)))

Now, here are a few examples of usage of this macro, I've written the original awk command in the comments in comparison:

;; numbering lines of a text file with NR
;; awk '{ print NR": "$0 }' file.txt
;;
(awk "file.txt" " "
     (format t "~a: ~a~%" NR line))

;; display NF-1 field (yes it's -2 in the example because -1 is last field in the list)
;; awk -F ';' '{ print NF-1 }' file.csv
;;
(awk "file.csv" ";"
     (print (nth (- NF 2) fields)))

;; filtering lines (like grep)
;; awk '/unbound/ { print }' /var/log/messages
;;
(awk "/var/log/messages" " "
     (when (search "unbound" line)
       (print line)))

;; printing 4nth field
;; awk -F ';' '{ print $4 }' data.csv
;;
(awk "data.csv" ";"
     (print (nth 4 fields)))

BitreichCON 2019 talks available

Written by Solène, on 27 August 2019.
Tags: #unix #drist #awk

Earlier in August 2019 happened the BitreichCON 2019. There was awesome talks there during two days but there are two I would like to share. You can find all the informations about this event at the following address with the Gopher protocol gopher://bitreich.org/1/con/2019

BrCON talks are happening through an audio stream, a ssh session for viewing the current slide and IRC for questions. I have the markdown files producing the slides (1 title = 1 slide) and the audio recording.

Simple solutions

This is a talk I have made for this conference. It as about using simple solutions for most problems. Simple solutions come with simple tools, unix tools. I explain with real life examples like how to retrieve my blog articles titles from the website using curl, grep, tr or awk.

Link to the audio

Link to the slides

Experiences with drist

Another talk from Parazyd about my deployment tool Drist so I feel obligated to share it with you.

In his talk he makes a comparison with slack (debian package, not the online community), explains his workflow with Drist and how it saves his precious time.

Link to the audio

Link to the slides

About the bitreich community

If you want to know more about the bitreich community, check gopher://bitreich.org or IRC #bitreich-en on Freenode servers.

There is also the bitreich website which is a website parody of the worse of what you can daily see.

Minimalistic markdown subset to html converter using awk

Written by Solène, on 26 August 2019.
Tags: #unix #awk

Hello

As on my blog I use different markup languages I would like to use a simpler markup language not requiring an extra package. To do so, I wrote an awk script handling titles, paragraphs and code blocks the same way markdown does.

16 December 2019 UPDATE: adc sent me a patch to add ordered and unordered list. Code below contain the addition.

It is very easy to use, like: awk -f mmd file.mmd > output.html

The script is the following:

BEGIN {
    in_code=0
    in_list_unordered=0
    in_list_ordered=0
    in_paragraph=0
}

{
    # escape < > characters
    gsub(/</,"\<",$0);
    gsub(/>/,"\>",$0);

    # close code blocks
    if(! match($0,/^    /)) {
        if(in_code) {
            in_code=0
            printf "</code></pre>\n"
        }
    }

    # close unordered list
    if(! match($0,/^- /)) {
        if(in_list_unordered) {
            in_list_unordered=0
            printf "</ul>\n"
        }
    }

    # close ordered list
    if(! match($0,/^[0-9]+\. /)) {
        if(in_list_ordered) {
            in_list_ordered=0
            printf "</ol>\n"
        }
    }

    # display titles
    if(match($0,/^#/)) {
        if(match($0,/^(#+)/)) {
            printf "<h%i>%s</h%i>\n", RLENGTH, substr($0,index($0,$2)), RLENGTH
        }

    # display code blocks
    } else if(match($0,/^    /)) {
        if(in_code==0) {
            in_code=1
            printf "<pre><code>"
            print substr($0,5)
        } else {
            print substr($0,5)
        }

    # display unordered lists
    } else if(match($0,/^- /)) {
        if(in_list_unordered==0) {
            in_list_unordered=1
            printf "<ul>\n"
            printf "<li>%s</li>\n", substr($0,3)
        } else {
            printf "<li>%s</li>\n", substr($0,3)
        }

    # display ordered lists
    } else if(match($0,/^[0-9]+\. /)) {
        n=index($0," ")+1
        if(in_list_ordered==0) {
            in_list_ordered=1
            printf "<ol>\n"
            printf "<li>%s</li>\n", substr($0,n)
        } else {
            printf "<li>%s</li>\n", substr($0,n)
        }

    # close p if current line is empty
    } else {
        if(length($0) == 0 && in_paragraph == 1 && in_code == 0) {
            in_paragraph=0
            printf "</p>"
        } # we are still in a paragraph
        if(length($0) != 0 && in_paragraph == 1) {
            print
        } # open a p tag if previous line is empty
        if(length(previous_line)==0 && in_paragraph==0) {
            in_paragraph=1
            printf "<p>%s\n", $0
        }
    }
    previous_line = $0
}

END {
    if(in_code==1) {
        printf "</code></pre>\n"
    }
    if(in_list_unordered==1) {
        printf "</ul>\n"
    }
    if(in_list_ordered==1) {
        printf "</ol>\n"
    }
    if(in_paragraph==1) {
        printf "</p>\n"
    }
}