About me: My name is Solène Rapenne, pronouns she/her. I like learning and sharing knowledge. Hobbies: '(BSD OpenBSD Qubes OS Lisp cmdline gaming security QubesOS internet-stuff). I love percent and lambda characters. Qubes OS core team member, former OpenBSD developer solene@. No AI is involved in this blog.

Contact me: solene at dataswamp dot org or @solene@bsd.network (mastodon).

I'm a freelance OpenBSD, FreeBSD, Linux and Qubes OS consultant, this includes DevOps, DevSecOps, technical writing or documentation work. If you enjoy this blog, you can sponsor my open source work financially so I can write this blog and contribute to Free Software as my daily job.

How to account systemd services bandwidth usage on NixOS

Written by Solène, on 20 July 2022.
Tags: #nixos #bandwidth #monitoring

Comments on Fediverse/Mastodon

1. Introduction §

Did you ever wonder how many bytes a system service is daily receiving from the network? Thanks to systemd, we can easily account this.

This guide targets NixOS, but the idea could be applied on any Linux system using systemd.

NixOS project website

In this article, we will focus on the nix-daemon service.

2. Setup §

We will enable the attribute IPAccounting on the systemd service nix-daemon, this will make systemd to account bytes and packets that received and sent by the service. However, when the service is stopped, the counters are reset to zero and the information logged into the systemd journal.

In order to efficiently gather the network information over time into a database, we will run a script just before the service stops using the preStop service hook.

The script checks the existence of a sqlite database /var/lib/service-accounting/nix-daemon.sqlite, creates it if required, and then inserts the received bytes information of the nix-daemon service about to stop. The script uses the service attribute InvocationID and the current day to ensure that a tuple won't be recorded more than once, because if we restart the service multiple times a day, we need to distinguish all the nix-daemon instances.

Here is the code snippet to add to your /etc/nixos/configuration.nix file before running nixos-rebuild test to apply the changes.

  systemd.services.nix-daemon = {
      serviceConfig.IPAccounting = "true";
      path = with pkgs; [ sqlite busybox systemd ];
      preStop = ''
#!/bin/sh

SERVICE="nix-daemon"
DEST="/var/lib/service-accounting"
DATABASE="$DEST/$SERVICE.sqlite"

mkdir -p "$DEST"

# check if database exists
if ! dd if="$DATABASE" count=15 bs=1 2>/dev/null | grep -Ea "^SQLite format.[0-9]$" >/dev/null
then
cat <<EOF | sqlite3 "$DATABASE"
CREATE TABLE IF NOT EXISTS accounting (
        id TEXT PRIMARY KEY,
        bytes INTEGER NOT NULL,
        day DATE NOT NULL
);
EOF
fi

BYTES="$(systemctl show "$SERVICE.service" -P IPIngressBytes | grep -oE "^[0-9]+$")"
INSTANCE="'$(systemctl show "$SERVICE.service" -P InvocationID | grep -oE "^[a-f0-9]{32}$")'"

cat <<EOF | sqlite3 "$DATABASE"
INSERT OR REPLACE INTO accounting (id, bytes, day) VALUES ($INSTANCE, $BYTES, date('now'));
EOF
     '';
  };

If you want to apply this to another service, the script has a single variable SERVICE that has to be updated.

3. Display the information from the database §

You can use the following command to display the bandwidth usage of the nix-daemon service with a day-by-date report:

$ echo "SELECT day, sum(bytes)/1024/1024 AS Megabytes FROM accounting group by day" | sqlite3 -header -column /var/lib/service-accounting/nix-daemon.sqlite
day         Megabytes
----------  ---------
2022-07-17  173
2022-07-19  3018
2022-07-20  84

Please note this command requires the sqlite package to be installed in your environment.

4. Enhancement §

I have some ideas to improve the setup:

  • The script could be improved to support multiple services within the database by using a new field
  • The command to display data could be improved and turned into a system package to make it easier to use
  • Provide an SQL query for monthly summary

5. Conclusion §

Systemd services are very flexible and powerful thanks to the hooks provided to run script at the right time. While I was interested into network usage accounting, it's also possible to achieve a similar result with CPU usage and I/O accesses.