1. Introduction §

This article is the second of a series about building my container images on my Forgejo server. In the previous article, I explained how I run a private registry with zot.

The images are now built by Forgejo Actions, which is the CI of Forgejo, compatible with the GitHub Actions syntax (mostly). The jobs are executed by the Forgejo runner, a program you install on another system which takes the jobs from Forgejo and runs them in containers or as a regular user, beware that using it as a regular user might never clean completely betweens builds and has less sandboxing.

Forgejo runner installation documentation

The biggest challenge of the whole setup was to have the Forgejo runner able to build container images, because the jobs already run inside a container.

In this blog post, you will learn how I run the Forgejo runner with Docker-in-Docker, and how a workflow builds, tests and pushes an image to zot.

2. The runner §

The runner runs in a dedicated VM. It is started as a container, next to a special container called "DinD" (Docker in Docker, not dinde which is a turkey in French), which is a Docker daemon running inside a container. The CI jobs talk to this Docker daemon to build and start containers.

The DinD container needs privileges to run, but it does not give any privilege to the runner. The jobs can build images, but they do not run as privileged containers. The whole thing is not easy to grasp, I have to admit.

2.1. Why a dedicated VM §

All my services run in Incus, and Docker-in-Docker would need a privileged container to work, which I do not want. The jobs also run code I do not control, like the dependencies of a project, so I prefer a better isolation than a container. This is why the runner lives in its own virtual machine, it has better security and I can allow priviliged containers to run.

The VM is created in Incus with OpenTofu, and it runs Ubuntu like the rest of my infrastructure. I gave it 4 vCPU, 8 GB of memory and 40 GiB of disk, but this is arbitrary, it depends on what you build. Compiling a kernel, generating a NixOS ISO or rendering a Hugo website do not need the same resources.

Because the VM only runs jobs, I consider it as something I can throw away and recreate at any time. For the same reason, I do not filter its outgoing network access, there are too many different things downloaded by the jobs for a filter to be practical, and it does not contain sensitive data.

2.2. Podman and Docker §

The runner itself is started by podman, and the DinD container is started by Docker. It may look strange to use both, but the reason is simple. I declare everything with podman quadlets, and Docker-in-Docker only works well with Docker. The Docker container is the exception, it is started at boot by a command run from a systemd service, while the podman containers use quadlets.

I could totally start the CI runner within Docker, but my whole deployment framework is using podman so I prefered to keep things simple for an extra dozen megabytes.

2.3. Registering the runner §

The get the runnre working with Forgejo, you need to authenticate the runner, there is a registration mechanism between the two. The network flow is that the runner is connected to Forgejo and receive orders, not Forgejo connecting to runners.

I registered the runner by hand, and I stored the registration secret with sops. I did not see a reason to automate it, because I only have to do it once, if I need to do it again once it would take a couple minutes.

The registration is well explained in the Forgejo documentation.

Forgejo runner installation documentation

2.4. Starting the runner and DinD §

The DinD container is started with this command, using the Docker image docker:dind:

docker run -p 192.168.200.200:2376:2375 --detach --privileged --restart always --name dind_container docker:dind dockerd -H tcp://0.0.0.0:2375 --tls=false

The container runs privileged, because this is required by Docker-in-Docker. The Docker daemon listens on the port 2375 and all interfaces inside the container, which is published on the port 2376 of the VM on the IP 192.168.200.200. TLS is disabled, so anyone who can reach this port controls the daemon but this is local traffic only. The port 2376 should only be reachable from the VM itself, so be careful with your firewall.

The runner has two ways to talk to a Docker daemon. In its configuration, docker_host is set to unix:///var/run/docker.sock, which is the podman socket, so the runner starts the job containers with podman. The jobs running in these containers need to build images, so they use the DinD daemon through TCP. This is done by setting the variable DOCKER_HOST for the jobs, in the envs section of the runner configuration file:

envs:
  DOCKER_HOST: tcp://192.168.200.200:2376

I had issues using 127.0.0.1 here, I think because of the nesting of the containers. So the VM has a dummy network interface, which is a virtual interface not connected to anything, with the address 192.168.200.200. This address is only reachable from the VM itself, and the jobs use it to reach DinD. You can use another private address if you prefer.

There is one thing to keep in mind with this setup. The image store of the DinD daemon is shared by all the repositories using the runner, so a job could reuse an image or a layer left by a job of another repository. This is why my builds always use --pull --no-cache.

3. The workflow §

The workflow building the image is in .forgejo/workflows/image.yml in the repository of my Airthings exporter, which is public (it is an IoT device to measure air quality).

airthings-prometheus-exporter repository

Here is a reduced version of it, without the verification of the base images, which is covered in the last article of this series:

name: image

on:
  push:
    branches: [main]
    paths: ['Containerfile', 'requirements.txt', 'airthings_exporter.py', 'airthings_site.py']
  pull_request:
    paths: ['Containerfile', 'requirements.txt', 'airthings_exporter.py', 'airthings_site.py', '.forgejo/workflows/image.yml']
  workflow_dispatch:

concurrency:
  group: image-${{ github.ref }}
  cancel-in-progress: false

jobs:
  publish:
    runs-on: alpine-node
    env:
      TAG: ${{ github.event_name == 'pull_request' && 'airthings-exporter:pr' || 'registry.example.org/airthings/airthings-exporter:latest' }}
    steps:
      - name: Tools
        run: apk add --no-cache docker-cli docker-cli-buildx

      - uses: actions/checkout@v4

      - name: Login to the registry
        if: github.event_name != 'pull_request'
        env:
          REGISTRY_CI: ${{ secrets.REGISTRY_CI }}
        run: printf '%s' "$REGISTRY_CI" | docker login registry.example.org -u ci-airthings --password-stdin

      - name: Build
        run: docker build --pull --no-cache -f Containerfile -t "$TAG" .

      - name: Start
        run: docker run --rm --entrypoint python "$TAG" -c 'import prometheus_client, requests, dotenv, airthings_exporter, airthings_site'

      - name: Push
        if: github.event_name != 'pull_request'
        run: docker push "$TAG"

Replace registry.example.org with the address of your registry, and create the secret REGISTRY_CI in the repository settings with the password of the ci-airthings account created in zot.

Some explanations:

• the workflow runs on every pull request, and when a change lands on main, but only if a file used in the image changed
• there is one concurrency group per branch, so checking a pull request does not wait for a push from main, this is overkill here but useful if you work with multiple people working on a project at the same time
• on a pull request, the image gets a local tag and the job never logs in to the registry, so the registry password is not available to the code of the pull request, the push only happens once the pull request is merged. The code in a PR in review should not be considered safe.
• the -f Containerfile part is needed because Docker only looks for a file named Dockerfile
• the step "Start" is a small test, it starts Python in the image and imports all the modules, if something is missing the job fails before the push

The repository is public, so you can see the runs of this workflow on the Actions page.

airthings-prometheus-exporter Actions page

4. Conclusion §

I really like this setup. I now use Renovate everywhere to keep the dependencies up to date (thanks to the fellow mastodon follower who suggested me Renovate). It opens a pull request, the workflow builds the image to check that it works, I merge the pull request, the image is built again and pushed to the registry. Then podman auto-update updates the container on the system, you can see an example in the airthings-prometheus-exporter repository.

The Docker-in-Docker setup was very complicated to get right, but since I build containers in a CI which already runs in containers, it is required.

If you need more CPU/memory for a project, you can also run a runner on your personal computer and use it only for this project. Having a runner not available 24/7 is not an issue, the jobs will be handled when the runner is started.

The runner itself is very easy to deploy. The hard part was DinD, it took me at least 2 hours to understand how to make it work. The Forgejo documentation has explanations, but it takes time to really understand what is happening between the runner, the job containers and the DinD daemon.

In the next article, I will explain how Renovate opens pull requests to update the dependencies and the base images, these pull requests are built and tested by this workflow before I merge them.