1. Introduction §

I wrote a few small programs for my home, mostly Prometheus exporters for IoT devices. They run in containers, and until recently I had to build their images on my laptop and push them to the server. This worked, but there was no test before deploying, and I was not updating the base image which is a security issue.

Now, the images are built on the CI of my Forgejo server, and pushed to a container registry in my network, then the servers pull them automatically when there is a new version.

This article is the first of a series of four about this setup:

• a private container registry with zot (this article)
• building container images with a Forgejo runner and Docker-in-Docker
• keeping the dependencies up to date with Renovate
• verifying the base images with cosign and hardening the images

The registry I use is zot, while I use "registry" (the reference registry from the Distribution project) as a pull-through cache. Zot provides authentication and access control per prefix (like on an object storage bucket policy), which "registry" does not.

zot official website

In this blog post, you will learn how to set up zot as a private registry, with one account per project.

2. zot §

zot is a container registry following the OCI specification, which is the standard format for container images. It is a single program written in Go, with a single JSON configuration. It supports per image access control, deduplication of the storage, a retention policy and Prometheus metrics, so I did not need anything else around it.

I could have used Forgejo internal registry but as it is an Internet facing instance, I did not want to expose all containers I push on it, which is not possible here. Alternatives seemed way too heavy for my simple need.

3. Running zot §

I run zot with podman as a quadlet, as usual.

In /etc/containers/systemd/, create the file zot.container:

[Container]
ContainerName=zot
# make sure to check the latest stable version instead of v2.1.22
Image=ghcr.io/project-zot/zot:v2.1.22
PublishPort=5010:5010
Volume=registry-data:/var/lib/registry
Volume=/etc/zot/config.json:/etc/zot/config.json:ro
Volume=/etc/zot/htpasswd:/auth/htpasswd:ro

[Service]
Restart=always

[Install]
WantedBy=default.target

The container images sent to zot are stored in the volume registry-data.

4. Configuration §

The configuration file can be daunting, but if you take a calm look on it, and use jq on it, it makes things a lot easier to grasp.

Here is my file /etc/zot/config.json, reduced to one project called airthings:

{
  "distSpecVersion": "1.1.1",
  "storage": {
    "rootDirectory": "/var/lib/registry",
    "dedupe": true,
    "gc": true,
    "gcDelay": "1h",
    "gcInterval": "24h",
    "retention": {
      "delay": "24h",
      "policies": [
        {
          "repositories": ["**"],
          "deleteReferrers": true,
          "deleteUntagged": true,
          "keepTags": [
            { "patterns": ["^latest$"] },
            { "patterns": [".*"], "mostRecentlyPushedCount": 10 }
          ]
        }
      ]
    }
  },
  "http": {
    "address": "0.0.0.0",
    "port": "5010",
    "realm": "registry",
    "compat": ["docker2s2"],
    "auth": {
      "htpasswd": { "path": "/auth/htpasswd" }
    },
    "accessControl": {
      "metrics": { "anonymousPolicy": ["read"] },
      "repositories": {
        "**": { "anonymousPolicy": ["read"] },
        "airthings/**": {
          "policies": [
            { "users": ["ci-airthings"], "actions": ["read", "create", "update"] }
          ],
          "anonymousPolicy": ["read"]
        }
      },
      "adminPolicy": {
        "users": ["admin"],
        "actions": ["read", "create", "update", "delete"]
      }
    }
  },
  "extensions": {
    "metrics": {
      "enable": true,
      "prometheus": { "path": "/metrics" }
    }
  }
}

Some explanations:

• any clients can pull the images without an account, this is fine because the registry is only reachable from my network and no container contains sensitive data in this context
• each project has its own account, which can only push images under its own prefix, so the CI of a project can not overwrite the images of another one
• only the admin account can delete images
• the retention keeps the tag latest and the 10 most recently pushed tags of each image, the images without a tag anymore are deleted after 24 hours
• docker2s2 accepts images in the Docker format, because my CI builds with Docker, this will be explained in the next article

The accounts are in a htpasswd file with bcrypt passwords. You can create an entry with htpasswd from the Apache tools (package httpd-tools on Fedora, apache2-utils on Debian), add -c the first time to create the file:

htpasswd -B /etc/zot/htpasswd ci-airthings

5. Pushing and pulling §

The CI pushes over HTTPS, through a reverse proxy only reachable from my VPN and from the CI runner. On each server pulling from it, create /etc/containers/registries.conf.d/internal-registry.conf:

[[registry]]
location = "192.168.1.50:5010"
insecure = true

I made it easier for this guide, but you might want to use a TLS certificate or a reverse proxy in front of it instead of using plain HTTP. In this case, remove insecure = true.

6. Metrics §

The configuration above enables the Prometheus metrics of zot on /metrics, and anyone in my network can read them without an account (this is the metrics entry in accessControl). They give the storage size of the images, the requests, and push and pull statistics, which can help to see the registry growing or to check the servers really pull the new images. The metrics access can be filtered on the reverse proxy in front of zot.

7. Conclusion §

zot was easy to setup, and the access control by prefix is exactly what I needed to give one account to each project. So far, I did not have any issue with it, but I will have to check how to handle pruning old data from it one day as it currently holds everything pushed to it indefinitely.

Forgejo internal registry is still a solid choice if you are fine with its limitations, I am using it for public projects providing containers.

In a future article, I will explain how the Forgejo runner builds the images and pushes them to zot.

8. Going further §

zot has a web interface and can scan the images for vulnerabilities with tools like Trivy, I did not enable these extensions yet.

This does not cover how to handle retention properly.