My name is Solène Rapenne. I like to learn and share my knowledge with other. With this blog I can share my experiences and issues. Some of my interests : '(BSD OpenBSD h+ Lisp Emacs cli-tool gaming internet-infrastructure Crossbow). I love % and lambda characters. OpenBSD Developer solene@.

Contact : solene on Freenode or solene+www at dataswamp dot org

This website is generated using cl-yag. A gopher version is available here

If you like what I write, you can give me a tip at donate at perso dot pw through Paypal.

# Generate qrcode using command line

Written by Solène, on 14 July 2018.
Tags: #unix

If you need to generate a QR picture using command line tool. I would recommend libqrencode.

qrencode -o file.png 'some text'


It’s also possible to display the QR code inside the terminal with the following command.

qrencode -t ANSI256 'some text'


Official qrencode website

# Tmux mastery

Written by Solène, on 05 July 2018.
Tags: #unix

Tips for using Tmux more efficiently

### Enter in copy mode

By default Tmux uses the emacs key-bindings, to make a selection you need to enter in copy-mode by pressing Ctrl+b and then [ with Ctrl+b being the tmux prefix key, if you changed it then do the replacement while reading.

If you need to quit the copy-mode, type Ctrl+C.

### Make a selection

While in copy-mode, selects your start or ending position for your selection and then press Ctrl+Space to start the selection. Now, move your cursor to select the text and press Ctrl+w to validate.

### Paste a selection

When you want to paste your selection, press Ctrl+b ] (you should not be in copy-mode for this!).

### Make a rectangle selection

If you want to make a rectangular selection, press Ctrl+space to start and immediately, press R (capitalized R), then move your cursor and validate with Ctrl+w.

### Output the buffer to X buffer

Make a selection to put the content in tmux buffer, then type

tmux save-buffer - | xclip


You may want to look at xclip (it’s a package) man page.

### Output the buffer to a file

tmux save-buffer file


### Load a file into buffer

It’s possible to load the content of a file inside the buffer for pasting it somewhere.

tmux load-buffer file


You can also load into the buffer the output of a command, using a pipe and - as a file like in this example:

echo 'something very interesting' | tmux load-buffer -


### Display the battery percentage in the status bar

If you want to display your battery percentage and update it every 40 seconds, you can add two following lines in ~/.tmux.conf:

set status-interval 40
set -g status-right "#[fg=colour155]#(apm -l)%% | #[fg=colour45]%d %b %R"


This example works on OpenBSD using apm command. You can reuse this example to display others informations.

# Writing an article using mdoc format

Written by Solène, on 03 July 2018.
Tags: #unix

I never wrote a man page. I already had to read at the source of a man page, but I was barely understand what happened there. As I like having fun and discovering new things (people call me a Hipster since last days days ;-) ).

I modified cl-yag (the website generator used for this website) to be only produced by mdoc files. The output was not very cool as it has too many html items (classes, attributes, tags etc…). The result wasn’t that bad but it looked like concatenated man pages.

I actually enjoyed playing with mdoc format (the man page format on OpenBSD, I don’t know if it’s used somewhere else). While it’s pretty verbose, it allows to separate the formatting from the paragraphs. As I’m playing with ed editor last days, it is easier to have an article written with small pieces of lines rather than a big paragraph including the formatting.

Finally I succeded at writing a command line which produced an usable html output to use it as a converter in cl-yag. Now, I’ll be able to write my articles in the mdoc format if I want :D (which is fun). The convert command is really ugly but it actually works, as you can see if you read this.

cat data/%IN  | mandoc -T markdown | sed -e '1,2d' -e '$d' | multimarkdown -t html -o %OUT  The trick here was to use markdown as an convert format between mdoc to html. As markdown is very weak compared to html (in possibilities), it will only use simple tags for formatting the html output. The sed command is needed to delete the mandoc output with the man page title at the top, and the operating system at the bottom. By having played with this, writing a man page is less obscure to me and I have a new unusual format to use for writing my articles. Maybe unusual for this use case, but still very powerful! # Trying to move away from emacs Written by Solène, on 03 July 2018. Tags: #unix #emacs Hello Today I will write about my current process of trying to get rid of emacs. I use it extensively with org-mode for taking notes and making them into a agenda/todo-list, this helped me a lot to remember tasks to do and what people told to me. I also use it for editing of course, any kind of text or source code. This is usually the editor I use for writing the blog articles that you can read here. This one is written using ed. I also read my emails in emacs with mu4e (which last version doesn’t work anymore on powerpc due to a c++14 feature used and no compiler available on powerpc to compile it…). While I like Emacs, I never liked to use one big tool for everything. My current quest is to look for a portable and efficient way to replace differents emacs parts. I will not stop using Emacs if the replacements are not good enough to do the job. So, I identified my Emacs uses: • todo-list / agenda / taking notes • writing code (perl, C, php, Common LISP) • IRC • mails • writing texts • playing chess by mail • jabber client I will try for each topic to identify alternatives and challenge them to Emacs. ## Todo-list / Agenda / Notes taking This is the most important part of my emacs use and it is the one I would really like to get out of Emacs. What I need is: writing quickly a task, add a deadline to it, add explanations or a description to it, be able to add sub-tasks for a task and be able to display it correctly (like in order of deadline with days / hours before deadline). I am trying to convert my current todo-list to taskwarrior, the learning curve is not easy but after spending one hour playing with it while reading the man page, I have understood enough to replace org-mode with it. I do not know if it will be as good as org-mode but only time will let us know. By the way, I found vit, a ncurses front-end for taskwarrior. ## Writing code Actually Emacs is a good editor. It supports syntax coloring, can evaluates regions of code (depend of the language), the editor is nice etc… I discovered jed which is a emacs-like editor written in C+libslang, it’s stable and light while providing more features than mg editor (available in OpenBSD base installation). While I am currently playing with ed for some reasons (I will certainly write about it), I am not sure I could use it for writing a software from scratch. ## IRC There are lots of differents IRC clients around, I just need to pick up one ## Mails I really enjoy using mu4e, I can find my mails easily with it, the query system is very powerful and interesting. I don’t know what I could use to replace it. I have been using alpine some times ago, and I tried mutt before mu4e and I did not like it. I have heard about some tools to manage a maildir folder using unix commands, maybe I should try this one. I did not any searches on this topic at the moment. ## Writing text For writing plain text like my articles or for using$EDITOR for differents tasks, I think that ed will do the job perfectly :-) There is ONE feature I really like in Emacs but I think it’s really easy to recreate with a script, the function bind on M-q to wrap a text to the correct column numbers!

Update: meanwhile I wrote a little perl script using Text::Wrap module available in base Perl. It wraps to 70 columns. It could be extended to fill blanks or add a character for the first line of a paragraph.

#!/usr/bin/env perl
use strict;use warnings;
use Text::Wrap qw(wrap $columns); open IN, '<'.$ARGV[0];
$columns = 70; my @file = <IN>; print wrap("","",@file);  This script does not modify the file itself though. Some people pointed me that Perl was too much for this task. I have been told about Groff or Par to format my files. Finally, I found a very BARE way to handle this. As I write my text with ed, I added an new alias named “ruled” with spawn ed with a prompt of 70 characters #, so I have a rule each time ed displays its prompt!!! :D It looks like this for the last paragraph: ###################################################################### c been told about Groff or Par to format my files. Finally, I found a very **BARE** way to handle this. As I write my text with ed, I added an new alias named "ruled" with spawn ed with a prompt of 70 characters #, so I have a rule each time ed displays its prompt!!! :D . ###################################################################### w  Obviously, this way to proceed only works when writing the content at first. If I need to edit a paragraph, I will need a tool to format correctly my document again. ## Jabber client Using jabber inside Emacs is not a very good experience. I switched to profanity (featured some times ago on this blog). ## Playing Chess Well, I stopped playing chess by mails, I am still waiting for my recipient to play his turn since two years now. We were exchanging the notation of the whole play in each mail, by adding our turn each time, I was doing the rendering in Emacs, but I do not remember exactly why but I had problems with this (replaying the string). # Easy encrypted backups on OpenBSD with base tools Written by Solène, on 26 June 2018. Tags: #unix #openbsd # Old article Hello, it turned out that this article is obsolete. I will rewrite it soon (soon for 2 August 2018), the security used in is not safe at all so the goal of this backup system isn’t achievable, thus it should not be used and I need another backup system. One of the most important feature for me for dump was to keep track of the inodes numbers. Another solution is to save the list of the inodes numbers and their path in a file before doing a backup. This can be achieved with the following command. doas ncheck -f "\I \P\n" /var  I planned to write an article about backup software as they are many of them in the OpenBSD ports tree, like borg, restic, duplicity, bacula, rsnapshot and others I did not mention. # Beginning of article Today’s topic is “Encrypted backups” using only OpenBSD base tools. I am planning to write a bigger article later about backups but it’s a wide topic with a lot of software to cover and a lot of explanations about the differents uses cases, needs, issues an solutions. Here I will stick on explaining how to make reliable backups for an OpenBSD system (my laptop). What we need is the dump command (see man 8 dump for its man page). It’s an utility to make a backup for a filesystem, it can only make a backup of one filesystem at a time. On my laptop I only backup /home partition so this solution is suitable for me while still being easy. Dump can do incremental backups, it means that it will only save what changed since the last backup of lower level. If you do not understand this, please refer to the dump man page. What is very interesting with dump is that it honors nodump flag which is an extended attribute of a FFS filesystem. One can use the command chflags nodump /home/solene/Downloads to tells dump not do save that folder (under some circumstances). By default, dump will not save thoses files, EXCEPT for a level 0 backup. Important features of this backup solution: • save files with attributes, permissions and flags • can recreate a partition from a dump, restore files interactively, from a list or from its inode number (useful when you have files in lost+found) • one dump = one file My process is to make a huge dump of level 0 and keep it on a remote server, then, once a week I make a level 1 backup which will contain everything changed since the last dump of level 0, and everyday I do a level 2 backup of my files. The level 2 will contain latest files and the files changing a lot, which are often the most interesting. The level 1 backup is important because it will offload a lot of changes for the level 2. Let me explain: let says my full backup is 60 GB, full of pictures, sources files, GUI applications data files etc… A level 1 backup will contain every new picture, new projects, new GUI files etc.. since the full backup, which will produce bigger and bigger dump over time, usually it is only 100 MB to 1GB. As I don’t add new pictures everyday or use new software everyday, the level 2 will take care of most littles changes to my data, like source code edited, little works on files etc… The level 2 backup is really small, I try to keep it under 50 MB so I can easily send it on my remote server everyday. One could you more dump level, up to level 9, but keep in mind that those are incremental. In my case, if I need to restore all my partition, I will need to use level 0, 1 and 2 to get up to latest backup state. If you want to restore a file deleted a few days ago, you need to remember in which level its latest version is. History note: dump was designed to be used with magnetic tapes. Now, the interesting part: how to use it? ## Commands to make a backup The process is the following: dump | compression | openssl > file To make a level 0 dump ignoring files having nodump flag: dump -0 -h0 -a -u -f - /home  This will output the dump to stdout and ignore nodump files below level 0, which will ignore it whatever the current level is. ## WARNING (POST PUBLICATION) SOME PEOPLE REPORTED ME THAT USING THIS OPENSSL FEATURE IS NOT SAFE AT ALL, PLEASE DO NOT USE THIS ON REMOTE BACKUPS YOU DO NOT TRUST AS IT SHOULD BE EASY TO BRUTEFORCE THE PASSWORD. I NEED TO WRITE A NEW WAY TO DO BACKUPS BUT I STILL NEED TO FIND A CORRECT WAY TO MANAGE ENCRYPTION.  Now we will encrypt (with very low security) it with openssl to store securely the files on any media we could use (usb media, remote server, local server, another computer). We will use openssl command for this, with the password in the command line (this is not a problem for me as my computer is trustable). LEVEL=0 dump -${LEVEL} -h0 -a -u -f - /home | \
openssl enc -k "7H3_P@$$W0RD" -aes-256-cbc -salt -out dump-level{LEVEL}.enc  I choosed .enc extension file for encoded. You will need the password to read the file. Now, we will see that using a compression tool before openssl can save a lot of space (depending of your data though). It is really easy to add compress in this pipe command. LEVEL=0 dump -{LEVEL} -h0 -a -u -f - /home | \ gzip -f -c | \ openssl enc -k "7H3_P@$$W0RD" -aes-256-cbc -salt -out dump-level${LEVEL}.gz.enc  In this case, gzip will save a lot of space if like me, most of your disk usage are mails and text files. We can push the compression a little further as we want to reduce backup size for sending it to a remote server. We will use xz command with 2 threads to make it faster and disable checksum because anyway, openssl will allow to verify the integrity of the file. LEVEL=0 dump -${LEVEL} -h0 -a -u -f - /home | \
xz -C none -T 2 -f -c | \
openssl enc -k "7H3_P@$$W0RD" -aes-256-cbc -salt -out dump-level{LEVEL}.xz.enc  ## How to restore from a dump Now that we have a shiniy backup that we hope we will never have to use, it is important to understand to use it if needed. The process is the following: cat file | openssl | decompression | restore Before someone scream about the cat command, I know that I could use openssl file instead of but it feels more pleasant to read like this. cat dump-level0.gz.enc | \ openssl enc -d -k "7H3_P@$$W0RD" -aes-256-cbc -salt - | \
xz -d -T2 -f -c - | \
restore -i -f -


One could write a short script like the following to give the file as a parameter and allow to choose restore’s parameters.

FILE=$1 ; shift test -f "${FILE}" && cat "$1" | \ openssl enc -d -k "7H3_P@$$W0RD" -aes-256-cbc -salt - | \ xz -d -T2 -f -c - | \ restore$@ -f -


I have faced situations where restore should be called with differents flags, like -m to show inodes.

I hope you found this article interesting, I wanted to share a daily usage of simple tools which can give interesting features when combined together.

# Change default application for xdg-open

Written by Solène, on 25 June 2018.
Tags: #unix

I write it as a note for me and if it can helps some other people, it’s fine.

To change the program used by xdg-open for opening some kind of file, it’s not that hard.

First, check the type of the file:

$xdg-mime query filetype file.pdf application/pdf  Then, choose the right tool for handling this type: $ xdg-mime default mupdf.desktop application/pdf


Honestly, having firefox opening PDF files with GIMP IS NOT FUN.

# Share a tmux session with someone with tmate

Written by Solène, on 01 June 2018.
Tags: #unix

New port of the week, and it’s about tmate.

If you ever wanted to share a terminal with someone without opening a remote access to your computer, tmate is the right tool for this.

Once started, tmate will create a new tmux instance connected through the tmate public server, by typing tmate show-messages you will get url for read-only or read-write links to share with someone, by ssh or web browser. Don’t forget to type clear to hide url after typing show-messages, otherwise viewing people will have access to the write url (and it’s not something you want).

If you don’t like the need of a third party, you can setup your own server, but we won’t cover this in this article.

When you want to end the share, you just need to exit the tmux opened by tmate.

If you want to install it on OpenBSD, just type pkg_add tmate and you are done. I think it’s available on most unix systems.

There is no much more to say about it, it’s great, simple, work out-of-the-box with no configuration needed.

# Deploying cron programmaticaly the unix way

Written by Solène, on 31 May 2018.
Tags: #unix

Here is a little script to automatize in some way your crontab deployment when you don’t want to use a configuration tool like ansible/salt/puppet etc…

The script works this way:

./install_cron crontab_solene


with crontab_solene file being an actual crontab correct, which could looks like this:

## TAG ##
MAILTO=""
*/5 * * * *  ( cd ~/dev/reed-alert && ecl --load check.lisp )
*/10 * * * * /usr/local/bin/r2e run
1 * * * * vacuumdb -azf -U postgres
## END_TAG ##


Then it will include the file into my current user crontab, the TAG in the file is here to be able to remove it and replace it later with the new version. The script could be easily modified to support the tag name as parameter, if you have multiple deploymenets using the same user on the same machine.

#!/bin/sh

if [ -z "$1" ]; then echo "Usage:$0 user_crontab_file"
exit 1
fi

VALIDATION=0
grep "^## TAG ##$" "$1" >/dev/null
VALIDATION=$? grep "^## END_TAG ##$" "$1" >/dev/null VALIDATION=$(( VALIDATION + $? )) if [ "$VALIDATION" -ne 0 ]
then
echo "file ./${1} needs \"## TAG ##\" and \"## END_TAG ##\" to be used" exit 2 fi crontab -l | \ awk '{ if($0=="## TAG ##") { hide=1 };  if(hide==0) { print } ; if($0=="## END_TAG ##") { hide=0 }; }' | \ cat - "${1}" | \
crontab -


# Faster SSH with multiplexing

Written by Solène, on 22 May 2018.
Tags: #unix #ssh

I discovered today an OpenSSH feature which doesn’t seem to be widely known. The feature is called multiplexing and consists of reusing an opened ssh connection to a server when you want to open another one. This leads to faster connection establishment and less processes running.

To reuse an opened connection, we need to use the ControlMaster option, which requires ControlPath to be set. We will also set ControlPersist for convenience.

• ControlMaster defines if we create, or use or nothing about multiplexing
• ControlPath defines where to store the socket to reuse an opened connection, this should be a path only available to your user.
• ControlPersist defines how much time to wait before closing a ssh connection multiplexer after all connection using it are closed. By default it’s “no” and once you drop all connections the multiplexer stops.

I choosed to use the following parameters into my ~/.ssh/config file:

Host *
ControlMaster auto
ControlPath ~/.ssh/sessions/%h%p%r.sock
ControlPersist 60


This requires to have ~/.ssh/sessions/ folder restricted to my user only. You can create it with the following command:

install -d -m 700 ~/.ssh/sessions


(you can also do mkdir ~/.ssh/sessions && chmod 700 ~/.ssh/sessions but this requires two commands)

The ControlPath variable will creates sessions with the name “${hostname}${port}${user}.sock”, so it will be unique per remote server. Finally, I choose to use ControlPersist to 60 seconds, so if I logout from a remote server, I still have 60 seconds to reconnect to it instantly. Don’t forget that if for some reason the ssh channel handling the multiplexing dies, all the ssh connections using it will die with it. ## Benefits with ProxyJump Another ssh feature that is very useful is ProxyJump, it’s really useful to access ssh hosts which are not directly available from your current place. Like servers with no public ssh server available. For my job, I have a lot of servers not facing the internet, and I can still connect to them using one of my public facing server which will relay my ssh connection to the destination. Using the ControlMaster feature, the ssh relay server doesn’t have to handle lot of connections anymore, but only one. In my ~/.ssh/config file: Host *.private.lan ProxyJump public-server.com  Those two lines allow me to connect to every servers with .private.lan domains (which is known by my local DNS server) by typing ssh some-machine.private.lan. This will establish a connection to public-server.com and then connects to the next server. # Sending mail with mu4e Written by Solène, on 22 May 2018. Tags: #unix #emacs In my article about mu4e I said that I would write about sending mails with it. This will be the topic covered in this article. There are a lot of ways to send mails with a lot of differents use cases. I will only cover a few of them, the documentation of mu4e and emacs are both very good, I will only give hints about some interestings setups. I would thank Raphael who made me curious about differents ways of sending mails from mu4e and who pointed out some mu4e features I wasn’t aware of. ## Send mails through your local server The easiest way is to send mails through your local mail server (which should be OpenSMTPD by default if you are running OpenBSD). This only requires the following line to works in your ~/.emacs file: (setq message-send-mail-function 'sendmail-send-it)  Basically, it would be only relayed to the recipient if your local mail is well configured, which is not the case for most servers. This requires a reverse DNS address correctly configured (assuming a static IP address), a SPF record in your DNS and a DKIM signing for outgoing mail. This is the minimum to be accepted to others SMTP servers. Usually people send mails from their personal computer and not from the mail server. ### Configure OpenSMTPD to relay to another smtp server We can bypass this problem by configuring our local SMTP server to relay our mails sent locally to another SMTP server using credentials for authentication. This is pretty easy to set-up, by using the following /etc/mail/smtpd.conf configuration, just replace remoteserver by your server. table aliases file:/etc/mail/aliases table secrets file:/etc/mail/secrets listen on lo0 accept for local alias <aliases> deliver to mbox accept for any relay via secure+auth://label@remoteserver:465 auth <secrets>  You will have to create the file /etc/mail/secrets and add your credentials for authentication on the SMTP server. From smtpd.conf(5) man page, as root: # touch /etc/mail/secrets # chmod 640 /etc/mail/secrets # chown root:_smtpd /etc/mail/secrets # echo "label username:password" > /etc/mail/secrets  Then, all mail sent from your computer will be relayed through your mail server. With ’sendmail-send-it, emacs will delivered the mail to your local server which will relay it to the outgoing SMTP server. ## SMTP through SSH One setup I like and I use is to relay the mails directly to the outgoing SMTP server, this requires no authentication except a SSH access to the remote server. It requires the following emacs configuration in ~/.emacs: (setq message-send-mail-function 'smtpmail-send-it smtpmail-smtp-server "localhost" smtpmail-smtp-service 2525)  The configuration tells emacs to connect to the SMTP server on localhost port 2525 to send the mails. Of course, no mail daemon runs on this port on the local machine, it requires the following ssh command to be able to send mails. $ ssh -N -L 127.0.0.1:2525:127.0.0.1:25 remoteserver


This will bind the port 127.0.0.1:25 from the remote server point of view on your address 127.0.0.1:2525 from your computer point of view.

Your mail server should accept deliveries from local users of course.

## SMTP authentication from emacs

It’s also possible to send mails from emacs using a regular smtp authentication directly from emacs. It is boring to setup, it requires putting credentials into a file named ~/.authinfo that it’s possible to encrypt using GPG but then it requires a wrapper to load it. It also requires to setup correctly the SMTP authentication. There are plenty of examples for this on the Internet, I don’t want to cover it.

## Queuing mails for sending it later

Mu4e supports a very nice feature which is mail queueing from smtpmail emacs client. To enable it, it requires two easy steps:

In ~/.emacs:

(setq
smtpmail-queue-mail t
smtpmail-queue-dir "~/Mail/queue/cur")


$mu mkdir ~/Mail/queue$ touch ~/Mail/queue/.noindex


Then, mu4e will be aware of the queueing, in the home screen of mu4e, you will be able to switch from queuing to direct sending by pressing m and flushing the queue by pressing f.

Note: there is a bug (not sure it’s really a bug). When sending a mail into the queue, if your mail contains special characters, you will be asked to send it raw or to add a header containing the encoding.

# Autoscrolling text for lazy reading

Written by Solène, on 17 May 2018.
Tags: #unix

Today I found a software named Lazyread which can read and display file an autoscroll at a chosen speed. I had to read its source code to make it work, the documentation isn’t very helpful, it doesn’t read ebooks (as in epub or mobi format) and doesn’t support stdin… This software requires some C code + a shell wrapper to works, it’s complicated for only scrolling.

So, after thinking a few minutes, the autoscroll can be reproduced easily with a very simple awk command. Of course, it will not have the interactive keys like lazyread to increase/decrease speed or some others options, but the most important part is there: autoscrolling.

If you want to read a file with a rate of 1 line per 700 millisecond, just type the following command:

awk '{system("sleep 0.7");print}' file


Do you want to read an html file (documentation file on the disk or from the web), you can use lynx or w3m to convert the html file on the fly to a readable text and pass it to awk stdin.

w3m -dump doc/slsh/slshfun-2.html | awk '{system("sleep 0.7");print}'
lynx -dump doc/slsh/slshfun-2.html | awk '{system("sleep 0.7");print}'
w3m -dump https://dataswamp.org/~solene/ | awk '{system("sleep 0.7");print}'


Maybe you want to read a man page?

man awk | awk '{system("sleep 0.7");print}'


If you want to pause the reading, you can use the true unix way, Ctrl+Z to send a signal which will stop the command and let it paused in background. You can resume the reading by typing fg.

One could easily write a little script parsing parameters for setting the speed or handling files or url with the correct command.

Notes: If for some reasons you try to use lazyread, fix the shebang in the file lesspipe.sh and you will need to call lazyread binary with the environment variable LESSOPEN="|./lesspipe.sh %s" (the path of the script if needed). Without this variable, you will have a very helpful error “file not found”.

# Port of the week: Sent

Written by Solène, on 15 May 2018.
Tags: #unix

As the new port of the week, We will discover Sent. While we could think it is mail related, it is not. Sent is a nice software to make presentations from a simple text file. It has been developped by Suckless, a hacker community enjoying writing good software while keeping a small and sane source code, they also made software like st, dwm, slock, surf…

Sent is about simplicity. I will reuse a part of the example file which is also the documentation of the tool.

usage:
$sent FILE1 [FILE2 …] ▸ one slide per paragraph ▸ lines starting with # are ignored ▸ image slide: paragraph containing @FILENAME ▸ empty slide: just use a \ as a paragraph @nyan.png this text will not be displayed, since the @ at the start of the first line makes this paragraph an image slide.  The previous text, saved into a file and used with sent will open a fullscreen window containg three “slides”. Each slide will resize the text to maximize the display usage, this mean the font size will change on each slide. It is really easy to use. To display next slide, you have the choice between pressing space, right arrow, return or clicking any button. Pressing left arrow will go back. If you want to install it on OpenBSD: pkg_add sent, the package comes from the port misc/sent. Be careful, Sent does not produce any file, you will need it for the presentation! Suckless sent website # Mounting remote samba share through SSH tunnel Written by Solène, on 04 May 2018. Tags: #unix If for some reason you need to access a Samba share outside of the network, it is possible to access it through ssh and mount the share on your local computer. Using the ssh command as root is required because you will bind local port 139 which is reserved for root: # ssh -L 139:127.0.0.1:139 user@remote-server -N  Then you can mount the share as usual but using localhost instead of remote-server. Example of a mount element for usmb <mount id="public" credentials="me"> <server>127.0.0.1</server> <!--server>192.168.12.4</server--> <share>public</share> <mountpoint>/mnt/share</mountpoint> <options>allow_other,uid=1000</options> </mount>  As a reminder, <!--tag>foobar</tag--> is a XML comment. # Extract files from winmail.dat Written by Solène, on 02 May 2018. Tags: #unix #email If you ever receive a mail with an attachment named “winmail.dat” then may be disappointed. It is a special format used by Microsoft Exchange, it contains the files attached to the mail and need some software to extract them. Hopefully, there is a little and effecient utility named “tnef” to extract the files. Install it: pkg_add tnef List files: tnef -t winmail.dat Extract files: tnef winmail.dat That’s all ! # Port of the week : ledger Written by Solène, on 02 May 2018. Tags: #unix In this post I will do a short presentation of the port productivity/ledger, an very powerful command line accounting software, using plain text as back-end. Writing on it is not an easy task, I will use a real life workflow of my usage as material, even if my use is special. As I said before, Ledger is very powerful. It can helps you manage your bank accounts, bills, rents, shares and others things. It uses a double entry system which means each time you add an operation (withdraw, paycheck, …) , this entry will also have to contain the current state of the account after the operation. This will be checked by ledger by recalculating every operations made since it has been initialized with a custom amount as a start. Ledger can also tracks categories where you spend money or statistics about your payment method (check, credit card, bank transfer, money…). As I am not an english native speaker and that I don’t work in banks or related, I am not very familiar with accounting words in english, it makes me very hard to understand all ledger keywords, but I found a special use case for accounting things and not money which is really practical. My special use case is that I work from home for a company working in a remote location. From time to time, I take the train to the to the office, the full travel is [home] → [underground A] → [train] → [underground B] → [office] [office] → [underground B] → [train] → [underground A] → [home]  It means I need to buy tickets for both underground A and underground B system, and I want to track tickets I use for going to work. I buy the tickets 10 by 10 but sometimes I use it for my personal use or sometimes I give a ticket to someone. So I need to keep track of my tickets to know when I can give a bill to my work for being refunded. Practical example: I buy 10 tickets of A, I use 2 tickets at day 1. On day 2, I give 1 ticket to someone and I use 2 tickets in the day for personal use. It means I still have 5 tickets in my bag but, from my work office point of view, I should still have 8 tickets. This is what I am tracking with ledger. 2018/02/01 * tickets stock Initialization + go to work Tickets:inv 10 City_A Tickets:inv 10 City_B Tickets:inv -2 City_A Tickets:inv -2 City_B Tickets 2018/02/08 * Work Tickets:inv -2 City_A Tickets:inv -2 City_B Tickets 2018/02/15 * Work + Datacenter access through underground Tickets:inv -4 City_B Tickets:inv -2 City_A Tickets  At the point, running ledger -f tickets.dat balance Tickets shows my tickets remaining: 4 City_A 2 City_B Tickets:inv  Will add another entry which requires me to buy tickets: 2018/02/22 * Work + Datacenter access through underground Tickets:inv -4 City_B Tickets:inv -2 City_A Tickets:inv 10 City_B Tickets  Now, running ledger -f tickets.dat balance Tickets shows my tickets remaining: 2 City_A 8 City_B Tickets:inv  I hope that the example was clear enought and interesting. There is a big tutorial document available on the ledger homepage, I recommend to read it before using ledger, it contains real world examples with accounting. Homepage link # Port of the week : dnstop Written by Solène, on 18 April 2018. Tags: #unix Dnstop is an interactive console application to watch in realtime the DNS queries going through a network interface. It currently only supports UDP DNS requests, the man page says that TCP isn’t supported. It has a lot of parameters and keybinding for the interactive use To install it on OpenBSD: doas pkg_add dnstop We will start dnstop on the wifi interface using a depth of 4 for the domain names: as root type dnstop -l 4 iwm0 and then press ‘3’ to display up to 3 sublevel, the -l 4 parameter means we want to know domains with a depth of 4, it means that if a request for the domain my.very.little.fqdn.com. happens, it will be truncated as very.little.fqdn.com. If you press ‘2’ in the interactive display, the earlier name will be counted in the line fqdn.com’. Example of output: Queries: 0 new, 6 total Tue Apr 17 07:17:25 2018 Query Name Count % cum% --------------- --------- ------ ------ perso.pw 3 50.0 50.0 foo.bar 1 16.7 66.7 hello.mydns.com 1 16.7 83.3 mydns.com.lan 1 16.7 100.0  If you want to use it, read the man page first, it has a lot of parameters and can filters using specific expressions. # How to read a epub book in a terminal Written by Solène, on 17 April 2018. Tags: #unix If you ever had to read an ebook in a epub format, you may have find yourself stumbling on Calibre software. Personally, I don’t enjoy reading a book in Calibre at all. Choice is important and it seems that Calibre is the only choice for this task. But, as the epub format is very simple, it’s possible to easily read it with any web browser even w3m or lynx. With a few commands, you can easily find xhtml files that can be opened with a web browser, an epub file is a zip containing mostly xhtml, css and images files. The xhtml files have links to CSS and images contained in others folders unzipped. In the following commands, I prefer to copy the file in a new directory because when you will unzip it, it will create folder in your current working directory. $ mkdir /tmp/myebook/
$cd /tmp/myebook$ cp ~/book.epub .
$unzip book.epub$ cd OPS/xhtml
$ls *xhtml  I tried with differents epub files, in most case you should find a lot of files named chapters-XX.xhtml with XX being 01, 02, 03 and so forth. Just open the files in the correct order with a web browser aka “html viewer”. # Port of the week : tig Written by Solène, on 10 April 2018. Tags: #unix #git Today we will discover the software named tig whose name stands for Text-mode Interface for Git. To install it on OpenBSD: pkg_add tig Tig is a light and easy to use terminal application to browse a git repository in an interactive manner. To use it, just ‘cd’ into a git repository on your filesystem and type tig. You will get the list of all the commits, with the author and the date. By pressing “Enter” key on a commit, you will get the diff. Tig also displays branching and merging in a graphical way. Tig has some parameters, one I like a lot if blame which is used like this: tig blame afile. Tig will show the file content and will display for each line to date of last commit, it’s author and the small identifier of the commit. With this function, it gets really easy to find who modified a line or when it was modified. Tig has a lot of others possibilities, you can discover them in its man pages. # Monitor your systems with reed-alert Written by Solène, on 17 January 2018. Tags: #unix #monitoring #lisp This article will present my software reed-alert, it checks user-defined states and send user-defined notification. I made it really eay to use but still configurable and extensible. ## Description reed-alert is not a monitoring tool producing graph or storing values. It does a job sysadmins are looking for because there are no alternative product (the alternatives comes from a very huge infrastructure like Zabbix so it’s not comparable). From its configuration file, reed-alert will check various states and then, if it fails, will trigger a command to send a notification (totally user-defined). ## Fetch it This is a open-source and free software released under MIT license, you can get it with the following command: git clone git://bitreich.org/reed-alert cd reed-alert  A README file is available as documentation to describe how to use it, but we will see here how to get started quickly. You will find a few files there, reed-alert is a Common LISP software and it has been choosed for (I hope) good reasons that the configuration file is plain Common LISP. There is a config file looking like a real world example named config.lisp.sample and another configuration file I use for testing named example.lisp containing lot of cases. ## Let’s start In order to use reed-alert we only need to create a new configuration file and then add a cron job. ### Configuration We are going to see how to configure reed-alert. You can find more explanations or details in the README file. #### Alerts We have to configure two kind of parameters, first we need to set-up a way to receive alerts, easiest way to do so is by sending a mail with “mail” command. Alerts are declared with the function alert and as parameters the alert name and the command to be executed. Some variables are replaced with values from the probe, in the README file you can find the list of probes, it looks like %date% or %params%. In Common LISP functions are called by using a parenthesis before its name and until the parenthesis is closed, we are giving its parameters. Example: (alert mail "echo 'problem on %hostname%' | mail me@example.com")  One should take care about nesting quotes here. reed-alert will fork a shell to start the command, so pipes and redirections works. You can be creative when writing alerts that: • use a SMS service • write a script to post on a forum • publishing a file on a server • send text to irc with ii client #### Checks Now we have some alerts, we will configure some checks in order to make reed-alert useful. It uses probes which are pre-defined checks with parameters, a probe could be “has this file not been updated since N minutes ?” or “Is the disk space usage of partition X more than Y ?” I choosed to name the function “=>” to make a check, it isn’t a name and reminds an item or something going forward. Both previous example using our previous mail notifier would look like: (=> mail file-updated :path "/program/file.generated" :limit "10") (=> mail disk-usage :limit 90)  It’s also possible to use shell commands and check the return code using the command probe, allowing the user to define useful checks. (=> mail command :command "echo '/is-this-gopher-server-up?' | nc -w 3 dataswamp.org 70" :desc "dataswamp.org gopher server")  We use echo + netcat to check if a connection to a socket works. The :desc keyword will give a nicer name in the output instead of just “COMMAND”. #### Garniture We wrote the minimum required to configure reed-alert, now the configuration file requires only two more lines, so your my-config.lisp file should looks like the following: (load "functions.lisp") (alert mail "echo 'problem on %hostname%' | mail me@example.com") (=> mail file-updated :path "/program/file.generated" :limit "10") (=> mail disk-usage :limit 90) (quit)  The first line is mandatory to make things working, the last line will prevent the lisp interpreter to stay in REPL (user interactive input). Now, you can start it every 5 minutes from a crontab with this: */5 * * * * ( cd /home/user/projects/reed-alert/ ; sbcl --load my-config.lisp )  If you prefer to use ecl: */5 * * * * ( cd /home/user/projects/reed-alert/ ; ecl --load my-config.lisp )  The time between each run is up to you, depending on what you monitor. #### Important By default, when a check returns a failure, reed-alert will only trigger the notifier associated once it reach the 3rd failure. And then, will notify again when the service is back (the variable %state% is replaced by start or end to know if it starts or stops.) This is to prevent reed-alert to send a notification each time it checks, there is absolutely no need for this for most users. The number of failures before triggering can be modified by using the keyword “:try” as in the following example: (=> mail disk-usage :limit 90 :try 1)  In this case, you will get notified at the first failure of it. The number of failures of failed checks is stored in files (1 per check) in the “states/” directory of reed-alert working directory. # New cl-yag version Written by Solène, on 16 December 2017. Tags: #unix ## Introduction cl-yag is a static website generator. It's a software used to publish a website and/or a gopher hole from a list of articles. As the developer of cl-yag I'm happy to announce that a new version has been released. ## New features The new version, with its number 0.6, bring lot of new features : • supporting different markup language per article • date format configurable • gopher output format configurable • ship with the default theme "clyma", minimalist but responsive (the one used on this website) • easier to use • full user documentation The code is available at git://bitreich.org/cl-yag, the program requires sbcl or ecl to work. ### Per article markup language The best feature I'm proud of is allowing to use a different language per article. While on my blog I choosed to use markdown, it's sometimes not adapted for more elaborated articles like the one about LISP containing code which was written in org-mode then converted to markdown manually to fit to cl-yag. Now, the user can declare a named "converter" which is a command line with pattern replacement, to produce the html file. We can imagine a lot of things with this, even producing a gallery with find + awk command. Now, I can use markdown by default and specify if I want to use org-mode or something else. This is the way to declare a converter, taking org-mode as example, which is not very simple, because of emacs not being script friendly : (converter :name :org-mode :extension ".org" :command (concatenate 'string "emacs data/%IN --batch --eval '(with-temp-buffer (org-mode) " "(insert-file \"%IN\") (org-html-export-as-html nil nil nil t)" "(princ (buffer-string)))' --kill | tee %OUT"))  And an easy way to produce a gallery with awk from a .txt file containing a list of images path. (converter :name :gallery :extension ".txt" :command (concatenate 'string "awk 'BEGIN { print \"<div class=\\\"gallery\\\">\"} " "{ print \"<img src=\\\"static/images/\"$1\"\\\" />\" } "
" END { print  \"</div>\"} data/%IN | tee %OUT"))


The concatenate function is only used to improve the presentation, to split the command in multiples lines and make it easier to read. It's possible to write all the command in one line.

The patterns %IN and %OUT are replaced by the input file name and the output file name when the command is executed.

For an easier example, the default markdown converter looks like this, calling multimarkdown command :

(converter :name :markdown :extension ".md"
:command "multimarkdown -t html -o %OUT data/%IN")


It's really easy (I hope !) to add new converters you need with this feature.

### Date format configurable

One problem I had with cl-yag is that it's plain vanilla Common LISP without libraries, so it's easier to fetch and use but it lacks some elaborated libraries like one to parse date and format a date. Before this release, I was writing in plain text "14 December 2017" in the date field of a blog post. It was easy to use, but not really usable in the RSS feed in the pubDate attribute, and if I wanted to change the display of the date for some reason, I would have to rewrite everything.

Now, the date is simply in the format "YYYYMMDD" like "20171231" for the 31rd December 2017. And in the configuration variable, there is a :date-format keyword to define the date display. This variable is a string allowing pattern replacement of the following variables :

%DayNumber
day of the month in number, from 1 to 31
%DayName
day of the week, from Monday to Sunday, names are written in english in the source code and can be translated
%MonthNumber
month in number, from 1 to 12
%MonthName
month name, from January to December, names are written in english in the source code and can be translated
%Year
year

Currently, as the time of writing, I use the value "%DayNumber %MonthName %Year"

A :gopher-format keyword exist in the configuration file to configure the date format in the gopher export. It can be different from the html one.

### More Gopher configuration

There are cases where the gopher server use an unusual syntax compared to most of the servers. I wanted to make it configurable, so the user could easily use cl-yag without having to mess with the code. I provide the default for geomyidae and in comments another syntax is available. There is also a configurable value to indicates where to store the gopher page menu, it's not always gophermap, it could be index.gph or whatever you need.

### Easier to use

A comparison of code will make it easier to understand. There was a little change the way blog posts are declared :

From

(defparameter *articles*
(list
(list :id "third-article"  :title "My third article" :tag "me" :date "20171205")
(list :id "second-article" :title "Another article"  :tag "me" :date "20171204")
(list :id "first-article"  :title "My first article" :tag "me" :date "20171201")
))


to

(post :id "third-article"  :title "My third article" :tag "me" :date "20171205")
(post :id "second-article" :title "Another article"  :tag "me" :date "20171204")
(post :id "first-article"  :title "My first article" :tag "me" :date "20171201")


Each post are independtly declared and I plan to add a "page" function to create static pages, but this is going to be for the next version !

## Future work

I am very happy to hack on cl-yag, I want to continue improving it but I should really think about each feature I want to add. I want to keep it really simple even if it limits the features.

I want to allow the creation of static pages like "About me", "Legal" or "websites I liked" that integrates well in the template. The user may not want all the static pages links to go at the same place in the template, or use the same template. I'm thinking about this.

Also, I think the gopher generation could be improved, but I still have no idea how.

Others themes may come in the default configuration, allowing the user to have a choice between themes. But as for now, I don't plan to bring a theme using javascript.

# How to type using only one hand : keyboard mirroring

Written by Solène, on 12 December 2017.
Tags: #unix

Hello

Today is a bit special because I’m writing with a mirror keyboard layout. I use only half my keyboard to type all characters. To make things harder, the layout is qwerty while I use azerty usually (I’m used to qwerty but it doesn’t help).

Here, “caps lock” is a modifier key that must be pressed to obtain characters of the other side. As a mirror, one will find ‘p’ instead of ‘q’ or ‘h’ instead of ‘g’ while pressing caps lock.

It’s even possible to type backspace to delete characters or to achieve a newline. All the punctuation isn’t available throught this, only ‘.<|¦>’",’.

While I type this I get a bit faster and it become more and more easier. It’s definitely worth if you can’t use hands two.

This a been made possible by Randall Munroe. To enable it just download the file Here and type

xkbcomp mirrorlayout.kbd $DISPLAY  backspace is use with tilde and return with space, using the modifier of course. I’ve spent approximately 15 minutes writing this, but the time spent hasn’t been linear, it’s much more fluent now ! Mirrorboard: A one-handed keyboard layout for the lazy by Randall Munroe # Bandwidth limit / queue on OpenBSD 6.1 Written by Solène, on 25 April 2017. Tags: #openbsd #unix #network Today I will explain how to do traffic limit with OpenBSD and PF. This is not hard at all if you want something easy, the man page pf.conf(5) in QUEUEING section is pretty good but it may disturbing when you don’t understand how it works. This is not something I master, I’m not sure of the behaviour in some cases but the following example works as I tested it ! :) ### Use case Internet is down at home, I want to use my phone as 4G router trough my OpenBSD laptop which will act as router. I don’t want the quota (some Gb) to be eaten in a few seconds, this connection allow to download up to 10 Mb/s so it can go quickly ! We will limit the total bandwidth to 1M (~ 110 kb/s) for people behind the NAT. It will be slow, but we will be sure that nothing behind the NAT like a program updating, cloud stuff synchronizing or videos in auto play won’t consume our quota. Edit /etc/pf.conf accordigly to your network internet="urndis0" lan="em0" # we define our available bandwidth queue main on$lan bandwidth 100M

# we will let 1M but we will allow
# 3M during 200 ms when initiating connection to keep the web a bit interactive
queue limited parent main bandwidth 1M min 0K max 1M burst 3M for 200ms default

set skip on lo

# we do NAT here
match out on egress inet from !(egress:network) to any nat-to (egress:0)

block all
pass out quick inet

# we apply the queue here on EVERYTHING coming from the internet
pass in on $lan proto tcp port www set queue web  ### Per host ? As before, you can apply queues on IP host/range rather than protocols, or you can even mix both if you want. ### Warning The limit function changed in OpenBSD 5.5, everything you can read on the internet about ALTQ isn’t working anymore. # Markup languages comparison Written by Solène, on 13 April 2017. Tags: #unix For the fun, here is a few examples of the same output in differents markup languages. The list isn’t exhaustive of course. This is org-mode: * This is a title level 1 + first item + second item + third item with a [[http://dataswamp.org][link]] ** title level 2 Blah blah blah blah blah blah blah blah *bold* here #+BEGIN_SRC lisp (let ((hello (init-string))) (format t "~A~%" (+ 1 hello)) (print hello)) #+END_SRC  This is markdown : # this is title level 1 + first item + second item + third item with a [Link](http://dataswamp.org) ## Title level 2 Blah blah blah blah blah blah blah blah **bold** here (let ((hello (init-string))) (format t "~A~%" (+ 1 hello)) (print hello)) or  (let ((hello (init-string))) (format t "~A~%" (+ 1 hello)) (print hello))   This is HTML : <h1>This is title level 1</h1> <ul> <li>first item></li> <li>second item</li> <li>third item with a <a href="http://dataswamp.org">link</a></li> </ul> <h2>Title level 2</h2> <p>Blah blah blah blah blah blah blah blah <strong>bold</strong> here <code><pre>(let ((hello (init-string))) (format t "~A~%" (+ 1 hello)) (print hello))</pre></code>  This is LaTeX : \begin{document} \section{This is title level 1} \begin{itemize} \item First item \item Second item \item Third item \end{itemize} \subsection{Title level 2} Blah blah blah blah blah blah blah blah \textbf{bold} here \begin{verbatim} (let ((hello (init-string))) (format t "~A~%" (+ 1 hello)) (print hello)) \end{verbatim} \end{document}  # OpenBSD 6.1 released Written by Solène, on 11 April 2017. Tags: #openbsd #unix Today OpenBSD 6.1 has been released, I won’t copy & paste the change list but, in a few words, it gets better. Link to the official announce I already upgraded a few servers, with both methods. One with bsd.rd upgrade but that requires physical access to the server and the other method well explained in the upgrade guide which requires to untar the files and do move some files. I recommend using bsd.rd if possible. # Connect to pfsense box console by usb Written by Solène, on 10 April 2017. Tags: #unix #network #openbsd Hello, I have a pfsense appliance (Netgate 2440) with a usb console port, while it used to be a serial port, now devices seems to have a usb one. If you plug an usb wire from an openbsd box to it, you woull see this in your dmesg uslcom0 at uhub0 port 5 configuration 1 interface 0 "Silicon Labs CP2104 USB to UART Bridge Controller" rev 2.00/1.00 addr 7 ucom0 at uslcom0 portno 0  To connect to it from OpenBSD, use the following command: # cu -l /dev/cuaU0 -s 115200  And you’re done # List of useful tools Written by Solène, on 22 March 2017. Tags: #unix Here is a list of software that I find useful, I will update this list everytime I find a new tool. This is not an exhaustive list, theses are only software I enjoy using: ## Backup Tool • duplicity • borg • restore/dump ## File synchronization tool • unison • rsync • lsyncd ## File sharing tool / “Cloud” • boar • nextcloud / owncloud • seafile • pydio • syncthing (works as peer-to-peer without a master) • sparkleshare (uses a git repository so I would recommend storing only text files) ## Editors • emacs • vim • jed ## Web browsers using keyboard • qutebrowser • firefox with vimperator extension ## Todo list / Personal Agenda… • org-mode (within emacs) • ledger (accounting) ## Mail client • mu4e (inside emacs, requires the use of offlineimap or mbsync to fetch mails) ## Network • curl • bwm-ng (to see bandwith usage in real time) • mtr (traceroute with a gui that updates every n seconds) ## Files integrity • bitrot • par2cmdline • aide ## Image viewer • sxiv • feh ## Stuff • entr (run command when a file change) • rdesktop (RDP client to connect to Windows VM) • xclip (read/set your X clipboard from a script) • autossh (to create tunnels that stays up) • mosh (connects to your ssh server with local input and better resilience) • ncdu (watch file system usage interactively in cmdline) • mupdf (PDF viewer) • pdftk (PDF manipulation tool) • x2x (share your mouse/keyboard between multiple computers through ssh) • profanity (XMPP cmdline client) • prosody (XMPP server) • pgmodeler (PostgreSQL database visualization tool) # How to check your data integrity? Written by Solène, on 17 March 2017. Tags: #unix #security #data Today, the topic is data degradation, bit rot, birotting, damaged files or whatever you call it. It’s when your data get corrupted over the time, due to disk fault or some unknown reason. # What is data degradation ? I shamelessy paste one line from wikipedia: “Data degradation is the gradual corruption of computer data due to an accumulation of non-critical failures in a data storage device. The phenomenon is also known as data decay or data rot.”. Data degradation on Wikipedia So, how do we know we encounter a bit rot ? bit rot = (checksum changed) && NOT (modification time changed)  While updating a file could be mistaken as bit rot, there is a difference update = (checksum changed) && (modification time changed)  # How to check if we encounter bitrot ? There is no way you can prevent bitrot. But there are some ways to detect it, so you can restore a corrupted file from a backup, or repair it with the right tool (you can’t repair a file with a hammer, except if it’s some kind of HammerFS ! :D ) In the following I will describe software I found to check (or even repair) bitrot. If you know others tools which are not in this list, I would be happy to hear about it, please mail me. In the following examples, I will use this method to generate bitrot on a file: % touch -d "2017-03-16T21:04:00" my_data/some_file_that_will_be_corrupted % generate_checksum_database_with_tool % echo "a" >> my_data/some_file_that_will_be_corrupted % touch -d "2017-03-16T21:04:00" my_data/some_file_that_will_be_corrupted % start_tool_for_checking  We generate the checksum database, then we alter a file by adding a “a” at the end of the file and we restore the modification and acess time of the file. Then, we start the tool to check for data corruption. The first touch is only for convenience, we could get the modification time with stat command and pass the same value to touch after modification of the file. ## bitrot This is a python script, it’s very easy to use. I will scan a directory and create a database with the checksum of the files and their modification date. Initialization usage: % cd /home/my_data/ % bitrot Finished. 199.41 MiB of data read. 0 errors found. 189 entries in the database, 189 new, 0 updated, 0 renamed, 0 missing. Updating bitrot.sha512... done. % echo$?
0


Verify usage (case OK):

% cd /home/my_data/
% bitrot
Checking bitrot.db integrity... ok.
Finished. 199.41 MiB of data read. 0 errors found.
189 entries in the database, 0 new, 0 updated, 0 renamed, 0 missing.
% echo $? 0  Exit status is 0, so our data are not damaged. Verify usage (case Error): % cd /home/my_data/ % bitrot Checking bitrot.db integrity... ok. error: SHA1 mismatch for ./sometextfile.txt: expected 17b4d7bf382057dc3344ea230a595064b579396f, got db4a8d7e27bb9ad02982c0686cab327b146ba80d. Last good hash checked on 2017-03-16 21:04:39. Finished. 199.41 MiB of data read. 1 errors found. 189 entries in the database, 0 new, 0 updated, 0 renamed, 0 missing. error: There were 1 errors found. % echo$?
1


When something is wrong. As the exit status of bitrot isn’t 0 when it fails, it’s easy to write a script running every day/week/month.

Github page

bitrot is available in OpenBSD ports in sysutils/bitrot since 6.1 release.

## par2cmdline

This tool works with PAR2 archives (see below for more informations about what PAR ) and from them, it will be able to check your data integrity AND repair it.

While it has some pros like being able to repair data, the cons is that it’s not very easy to use. I would use this one for checking integrity of long term archives that won’t changes. The main drawback comes from PAR specifications, the archives are created from a filelist, if you have a directory with your files and you add new files, you will need to recompute ALL the PAR archives because the filelist changed, or create new PAR archives only for the new files, but that will make the verify process more complicated. That doesn’t seems suitable to create new archives for every bunchs of files added in the directory.

PAR2 let you choose the percent of a file you will be able to repair, by default it will create the archives to be able to repair up to 5% of each file. That means you don’t need a whole backup for the files (while it’s would be a bad idea) and only an approximately extra of 5% of your data to store.

Create usage:

% cd /home/
% par2 create -a integrity_archive -R my_data
Skipping 0 byte file: /home/my_data/empty_file

Block size: 3812
Source file count: 17
Source block count: 2000
Redundancy: 5%
Recovery block count: 100
Recovery file count: 7

Opening: my_data/[....]
[text cut here]
Opening: my_data/[....]

Computing Reed Solomon matrix.
Constructing: done.
Wrote 381200 bytes to disk
Writing recovery packets
Writing verification packets
Done

% echo $? 0 % ls -1 integrity_archive.par2 integrity_archive.vol000+01.par2 integrity_archive.vol001+02.par2 integrity_archive.vol003+04.par2 integrity_archive.vol007+08.par2 integrity_archive.vol015+16.par2 integrity_archive.vol031+32.par2 integrity_archive.vol063+37.par2 my_data  Verify usage (OK): % par2 verify integrity_archive.par2 Loading "integrity_archive.par2". Loaded 36 new packets Loading "integrity_archive.vol000+01.par2". Loaded 1 new packets including 1 recovery blocks Loading "integrity_archive.vol001+02.par2". Loaded 2 new packets including 2 recovery blocks Loading "integrity_archive.vol003+04.par2". Loaded 4 new packets including 4 recovery blocks Loading "integrity_archive.vol007+08.par2". Loaded 8 new packets including 8 recovery blocks Loading "integrity_archive.vol015+16.par2". Loaded 16 new packets including 16 recovery blocks Loading "integrity_archive.vol031+32.par2". Loaded 32 new packets including 32 recovery blocks Loading "integrity_archive.vol063+37.par2". Loaded 37 new packets including 37 recovery blocks Loading "integrity_archive.par2". No new packets found There are 17 recoverable files and 0 other files. The block size used was 3812 bytes. There are a total of 2000 data blocks. The total size of the data files is 7595275 bytes. Verifying source files: Target: "my_data/....." - found. [...cut here...] Target: "my_data/....." - found. All files are correct, repair is not required. % echo$?
0


Verify usage (with error):

par2 verify integrity_archive.par.par2
Loaded 1 new packets including 1 recovery blocks
Loaded 2 new packets including 2 recovery blocks
Loaded 4 new packets including 4 recovery blocks
Loaded 8 new packets including 8 recovery blocks
Loaded 16 new packets including 16 recovery blocks
Loaded 32 new packets including 32 recovery blocks
Loaded 37 new packets including 37 recovery blocks
No new packets found

There are 17 recoverable files and 0 other files.
The block size used was 3812 bytes.
There are a total of 2000 data blocks.
The total size of the data files is 7595275 bytes.

Verifying source files:

Target: "my_data/....." - found.
[...cut here...]
Target: "my_data/....." - found.
Target: "my_data/Ebooks/Lovecraft/Quete Onirique de Kadath l'Inconnue.epub" - damaged. Found 95 of 95 data blocks.

Scanning extra files:

Repair is required.
1 file(s) exist but are damaged.
16 file(s) are ok.
You have 2000 out of 2000 data blocks available.
You have 100 recovery blocks available.
Repair is possible.
You have an excess of 100 recovery blocks.
None of the recovery blocks will be used for the repair.

% echo $? 1  Repair usage: % par2 repair integrity_archive.par.par2 Loading "integrity_archive.par.par2". Loaded 36 new packets Loading "integrity_archive.par.vol000+01.par2". Loaded 1 new packets including 1 recovery blocks Loading "integrity_archive.par.vol001+02.par2". Loaded 2 new packets including 2 recovery blocks Loading "integrity_archive.par.vol003+04.par2". Loaded 4 new packets including 4 recovery blocks Loading "integrity_archive.par.vol007+08.par2". Loaded 8 new packets including 8 recovery blocks Loading "integrity_archive.par.vol015+16.par2". Loaded 16 new packets including 16 recovery blocks Loading "integrity_archive.par.vol031+32.par2". Loaded 32 new packets including 32 recovery blocks Loading "integrity_archive.par.vol063+37.par2". Loaded 37 new packets including 37 recovery blocks Loading "integrity_archive.par.par2". No new packets found There are 17 recoverable files and 0 other files. The block size used was 3812 bytes. There are a total of 2000 data blocks. The total size of the data files is 7595275 bytes. Verifying source files: Target: "my_data/....." - found. [...cut here...] Target: "my_data/....." - found. Target: "my_data/Ebooks/Lovecraft/Quete Onirique de Kadath l'Inconnue.epub" - damaged. Found 95 of 95 data blocks. Scanning extra files: Repair is required. 1 file(s) exist but are damaged. 16 file(s) are ok. You have 2000 out of 2000 data blocks available. You have 100 recovery blocks available. Repair is possible. You have an excess of 100 recovery blocks. None of the recovery blocks will be used for the repair. Wrote 361069 bytes to disk Verifying repaired files: Target: "my_data/Ebooks/Lovecraft/Quete Onirique de Kadath l'Inconnue.epub" - found. Repair complete. % echo$?
0


par2cmdline is only one implementation doing the job, others tools working with PAR archives exists. They should be able to all works with the same PAR files.

Parchive on Wikipedia

Github page

par2cmdline is available in OpenBSD ports in archivers/par2cmdline.

If you find a way to add new files to existing archives, please mail me.

## mtree

One can write a little script using mtree (in base system on OpenBSD and FreeBSD) which will create a file with the checksum of every files in the specified directories. If mtree output is different since last time, we can send a mail with the difference. This is a process done in base install of OpenBSD for /etc and some others files to warn you if it changed.

While it’s suited for directories like /etc, in my opinion, this is not the best tool for doing integrity check.

## ZFS

I would like to talk about ZFS and data integrity because this is where ZFS is very good. If you are using ZFS, you may not need any other software to take care about your data. When you write a file, ZFS will also store its checksum as metadata. By default, the option “checksum” is activated on dataset, but you may want to disable it for better performance.

There is a command to ask ZFS to check the integrity of the files. Warning: scrub is very I/O intensive and can takes from hours to days or even weeks to complete depending on your CPU, disks and the amount of data to scrub:

# zpool scrub zpool


The scrub command will recompute the checksum of every file on the ZFS pool, if something is wrong, it will try to repair it if possible. A repair is possible in the following cases:

If you have multiple disks like raid-Z or raid–1 (mirror), ZFS will be look on the differents disks if the non corrupted version of the file exists, if it finds it, it will restore it on the disk(s) where it’s corrupted.

If you have set the ZFS option “copies” to 2 or 3 (1 = default), that means that the file is written 2 or 3 time on the disk. Each file of the dataset will be allocated 2 or 3 time on the disk, so take care if you want to use it on a dataset containing heavy files ! If ZFS find thats a version of a file is corrupted, it will check the others copies of it and tries to restore the corrupted file is possible.

You can see the percentage of filesystem already scrubbed with

zfs status zpool


and the scrub can be stopped with

zfs scrub -s zpool


### AIDE

Its name is an acronym for “Advanced Intrusion Detection Environment”, it’s an complicated software which can be used to check for bitrot. I would not recommend using it if you only need bitrot detection.

Here is a few hints if you want to use it for checking your file integrity:

/etc/aide.conf

/home/my_data/ R
# Rule definition
All=m+s+i+sha256
summarize_changes=yes


The config file will create a database of all files in /home/my_data/ (R for recursive). “All” line list the checks we do on each file. For bitrot checking, we want to check modification time, size, checksum and inode of the files. The summarize_change line permit to have a list of changes if something is wrong.

This is the most basic config file you can have. Then you will have to run aide to create the database and then run aide to create a new database and compare the two databases. It doesn’t update its database itself, you will have to move the old database and tell it where to found the older database.

# My use case

I have different kind of data. On a side, I have static data like pictures, clips, music or things that won’t change over time and the other side I have my mails, documents and folders where the content changes regularly (creation, deletetion, modification). I am able to afford a backup for 100% of my data with some history of the backup on a few days, so I won’t be interested about file repairing.

I want to be warned quickly if a file get corrupted, so I can still get the backup in my history but I don’t keep every versions of my files for too long. I choose to go with the python tool bitrot, it’s very easy to use and it doesn’t become a mess with my folders getting updated often.

I would go with par2cmdline if I could not be able to backup all my data. Having 5% or 10% of redundancy of my files should be enough to restore it in case of corruption without taking too much space.

# Port of the week : rss2email

Written by Solène, on 24 January 2017.
Tags: #portoftheweek #unix

This is the kind of Port of the week I like. This is a software I just discovered and fall in love to. The tool r2e which is the port mail/rss2email on OpenBSD is a small python utility that solves a problem: how to deal with RSS feeds?

Until last week, I was using a “web app” named selfoss which was aggregating my RSS feeds and displaying it on a web page, I was able to filter by read/unread/marked and also filter by source. It is a good tool that does the job well but I wanted something that doesn’t rely on a web browser. Here comes r2e !

This simple software will send you a mail for each new entry in your RSS feeds. It’s really easy to configure and set-up. Just look at how I configured mine:

$r2e new my-address+rss@my-domain.com$ r2e add "http://undeadly.org/cgi?action=rss"
$r2e add "https://dataswamp.org/~solene/rss.xml"$ r2e add "https://www.dragonflydigest.com/feed"
$r2e add "http://phoronix.com/rss.php"  Add this in your crontab to check new RSS items every 10 minutes: */10 * * * * /usr/local/bin/r2e run  Add a rule for my-address+rss to store mails in a separate folder, and you’re done ! NOTE: you can use r2e run –no-send for the first time, it will create the database and won’t send you mails for current items in feeds. # Convert mailbox to maildir with dovecot Written by Solène, on 17 January 2017. Tags: #unix I have been using mbox format for a few years on my personal mail server. For those who don’t know what mbox is, it consists of only one file per folder you have on your mail client, each file containing all the mails of the corresponding folder. It’s extremely ineficient when you backup the mail directory because it must copy everything each time. Also, it reduces the system cache possibility of the server because if you have folders with lots of mails with attachments, it may not be cached. Instead, I switched to maildir, which is a format where every mail is a regular file on the file system. This takes a lot of inodes but at least, it’s easier to backup or to deal with it for analysis. Here how to switch from mbox to maildir with a dovecot tool. # dsync -u solene mirror mbox:~/mail/:INBOX=~/mail/inbox  That’s all ! In this case, my mbox folder was ~/mail/ and my INBOX file was ~/mail/inbox. It tooks me some time to find where my INBOX really was, at first I tried a few thing that didn’t work and tried a perl convert tool named mb2md.pl which has been able to extract some stuff but a lot of mails were broken. So I have been going back getting dsync working. If you want to migrate, the whole process looks like: # service smtpd stop modify dovecot/conf.d/10-mail.conf, replace the first line mail_location = mbox:~/mail:INBOX=/var/mail/%u # BEFORE mail_location = maildir:~/maildir # AFTER # service dovecot restart # dsync -u solene mirror mbox:~/mail/:INBOX=~/mail/inbox # service smtpd start  # Port of the week : entr Written by Solène, on 07 January 2017. Tags: #unix entr is a command line tool that let you run arbitrary command on file change. This is useful when you are doing something that requires some processing when you modify it. Recently, I have used it to edit a man page. At first, I had to run mandoc each time I modified to file to check the render. This was the first time I edited a man page so I had to modify it a lot to get what I wanted. I remembered about entr and this is how you use it: ls stagit.1 | entr mandoc /_  This simple command will run “mandoc stagit.1” each time stagit.1 is modified. The file names must be given by stdin to entr, and then use the characters sequence /_ to replace the names (like {} in find). The man page of entr is very well documented if you need more examples. # Port of the week : dnscrypt-proxy Written by Solène, on 19 October 2016. Tags: #unix #security #portoftheweek #dns Today I will talk about net/dnscrypt-proxy. This let you encrypt your DNS traffic between your resolver and the remote DNS recursive server. More and more countries and internet provider use DNS to block some websites, and now they tend to do “man in the middle” with DNS answers, so you can’t just use a remote DNS you find on the internet. While a remote dnscrypt DNS server can still be affected by such “man in the middle” hijack, there is a very little chance DNS traffic is altered in datacenters / dedicated server hosting. The article also deal with unbound as a dns cache because dnscrypt is a bit slow and asking multiple time the same domain in a few minutes is a waste of cpu/network/time for everyone. So I recommend setting up a DNS cache on your side (which can also permit to use it on a LAN). At the time I write this article, their is a very good explanation about “how to install it” is named dnscrypt-proxy–1.9.5p3 in the folder /usr/local/share/doc/pkg-readmes/. The following article is made from this file. (Article updated at the time of OpenBSD 6.3) While I write for OpenBSD this can be easily adapted to anthing else Unix-like. ### Install dnscrypt # pkg_add dnscrypt-proxy  ### Resolv.conf Modify your resolv.conf file to this /etc/resolv.conf : nameserver 127.0.0.1 lookup file bind options edns0  ### When using dhcp client If you use dhcp to get an address, you can use the following line to force having 127.0.0.1 as nameserver by modifying dhclient config file. Beware, if you use it, when upgrading the system from bsd.rd, you will get 127.0.0.1 as your DNS server but no service running. /etc/dhclient.conf : supersede domain-name-servers 127.0.0.1;  ### Unbound Now, we need to modify unbound config to tell him to ask DNS at 127.0.0.1 port 40. Please adapt your config, I will just add what is mandatory. Unbound configuration file isn’t in /etc because it’s chrooted /var/unbound/etc/unbound.conf: server: # this line is MANDATORY do-not-query-localhost: no forward-zone: name: "." forward-addr: 127.0.0.1@40 # address dnscrypt listen on  If you want to allow other to resolv through your unbound daemon, please see parameters interface and access-control. You will need to tell unbound to bind on external interfaces and allow requests on it. ### Dnscrypt-proxy Now we need to configure dnscrypt, pick a server in the following LIST /usr/local/share/dnscrypt-proxy/dnscrypt-resolvers.csv, the name is the first column. As root type the following (or use doas/sudo), in the example we choose dnscrypt.eu-nl as a DNS provider # rcctl enable dnscrypt_proxy # rcctl set dnscrypt_proxy flags -E -m1 -R dnscrypt.eu-nl -a 127.0.0.1:40 # rcctl start dnscrypt_proxy  ### Conclusion You should be able to resolv address through dnscrypt now. You can use tcpdump on your external interface to see if you see something on udp port 53, you should not see traffic there. If you want to use dig hostname -p 40 @127.0.0.1 to make DNS request to dnscrypt without unbound, you will need net/isc-bind which will provide /usr/local/bin/dig. OpenBSD base dig can’t use a port different than 53. # How to publish a git repository on http Written by Solène, on 07 October 2016. Tags: #unix #git Here is an how-to in order to make a git repository available for cloning through a simple http server. This method only allow people to fetch the repository, not to push. I wanted to set-up this to get my code, I don’t plan to have any commit on it from other people at this time so it’s enough. In a folder publicly available from your http server clone your repository in bare mode. As explained in the [https://git-scm.com/book/tr/v2/Git-on-the-Server-The-Protocols](man page): $ cd /var/www/htdocs/some-path/
$git clone --bare /path/to/git_project gitproject.git$ cd gitproject.git
$git update-server-info$ mv hooks/post-update.sample hooks/post-update
$chmod o+x hooks/post-update  Then you will be able to clone the repository with $ git clone https://your-hostname/some-path/gitproject.git


I’ve lost time because I did not execute git update-server-info so the clone wasn’t possible.

# Port of the week : rlwrap

Written by Solène, on 04 October 2016.
Tags: #unix #shell #portoftheweek

Today I will present misc/rlwrap which is an utility tool when you use some command-line software which doesn’t provide you a nice readline input. By using rlwrap, you will be able to use telnet, a language REPL or any command-line tool where you input text with an history of what you type, ability to use emacs bindings like C-a C-e M-Ret etc… I use it often with telnet or sbcl.

Usage :

$rlwrap telnet host port  # Redirect stdin into a variable in shell Written by Solène, on 12 September 2016. Tags: #shell #unix If you want to write a script reading stdin and put it into a variable, there is an very easy way to procede : #!/bin/sh var=cat echo$var


That’s all