About me: My name is Solène Rapenne, pronouns she/her. I like learning and sharing knowledge. Hobbies: '(BSD OpenBSD Qubes OS Lisp cmdline gaming security QubesOS internet-stuff). I love percent and lambda characters. OpenBSD developer solene@. No AI is involved in this blog.

Contact me: solene at dataswamp dot org or @solene@bsd.network (mastodon).

I'm a freelance OpenBSD, FreeBSD, Linux and Qubes OS consultant, this includes DevOps, DevSecOps, technical writing or documentation work. If you enjoy this blog, you can sponsor my open source work financially so I can write this blog and contribute to Free Software as my daily job.

Faster packages updates with OpenBSD

Written by Solène, on 06 August 2021.
Tags: #openbsd

Comments on Fediverse/Mastodon

1. Introduction §

On OpenBSD, pkg_add is not the fastest package manager around but it is possible to make a simple change to make yours regular updates check faster.

Disclaimer: THIS DOES NOT WORK ON -current/development version!

2. Explanation §

When you configure the mirror url in /etc/installurl, on release/stable installations when you use "pkg_add", some magic happens to expand the base url into full paths usable by PKG_PATH.

http://ftp.fr.openbsd.org/pub/OpenBSD

becomes

http://ftp.fr.openbsd.org/pub/OpenBSD/%v/packages-stable/%a/:http://ftp.fr.openbsd.org/pub/OpenBSD/%v/packages/%a/

The built string passed to PKG_PATH is the concatenation (joined by a ":" character) of the URL toward /packages/ and /packages-stable/ directories for your OpenBSD version and architecture.

This is why when you use "pkg_info -Q foobar" to search for a package and that a package name matches "foobar" in /packages-stable/ pkg_info will stop, it search for a result in the first URL given by PKG_PATH, when you add -a like "pkg_info -aQ foobar", it will look in all URL available in PKG_PATH.

3. Why we can remove /packages/ §

When you run your OpenBSD system freshly installed or after an upgrade, when you have your packages sets installed from the repository of your version, the files in /packages/ in the mirrors will NEVER CHANGE. When you run "pkg_add -u", it's absolutely 100% sure nothing changed in the directory /packages/, so checking for changes against them every time make no sense.

Using "pkg_add -u" with the defaults makes sense when you upgrade from a previous OpenBSD version because you need to upgrade all your packages. But then, when you look for security updates, you only need to check against /packages-stable/.

4. How to proceed §

There are two ways, one reusing your /etc/installurl file and the other is hard coding it. Pick the one you prefer.

# reusing the content of /etc/installurl
env PKG_PATH="$(cat /etc/installurl)/%v/packages-stable/%a/" pkg_add -u

# hard coding the url
env PKG_PATH="http://ftp.fr.openbsd.org/pub/OpenBSD/%v/packages-stable/%a/" pkg_add -u

Be careful, you will certainly have a message like this:

Couldn't find updates for ImageMagick-6.9.12.2 adwaita-icon-theme-3.38.0 aom-2.0.2 argon2-20190702 aspell-0.60.6.1p10 .....

This is perfectly normal, as pkg_add didn't find the packages in /packages-stable/ it wasn't able to find the current version installed or an update, as we only want updates it's fine.

5. Simple benchmark §

On my server running 6.9 with 438 packages I get these results.

  • packages-stable only: 44 seconds
  • all the packages: 203 seconds

I didn't measure the bandwidth usage but it should scale with the time reduction.

6. Conclusion §

This is a very simple and reliable way to reduce the time and bandwidth required to check for updates on OpenBSD (non -current!). I wonder if this would be a good idea to provide it as a flag for pkg_add, like "only check for stable updates".