1. Introduction §

In this guide, I'll explain how to create a NixOS VM in the hosting company OpenBSD Amsterdam which only provides OpenBSD VMs hosted on OpenBSD.

I'd like to thank the team at OpenBSD Amsterdam who offered me a VM for this experiment. While they don't support NixOS officially, they are open to have customers running non-OpenBSD systems on their VMs.

OpenBSD Amsterdam hosting service website

2. The steps from OpenBSD to NixOS §

Here is a short description of the steps required to get NixOS installed on OpenBSD Amsterdam.

1. Generate a NixOS VM disk file or use the one I provide
2. Rent a VM at OpenBSD Amsterdam (5€ / month for 1 vCPU, 1GB of memory and 50 GB of hdd, with a dedicated IPv4, working IPv6 and reverse DNS)
3. Connect to the hypervisor in order to get the serial console access to your VM
4. Connect with ssh to your VM to reboot it
5. In the serial console, upon reboot, boot on bsd.rd (the OpenBSD installer ramdisk)
6. Overwrite the local disk by fetching your NixOS VM disk file through http/ftp and writing it on the disk file
7. Reboot on NixOS
8. Configure the network from the serial console, rebuild the system
9. Enjoy

3. How to proceed §

You need to order a VM at OpenBSD Amsterdam first. You will receive an email with your VM name, its network configuration (IPv4 and IPv6), and explanations to connect to the hypervisor. We will need to connect to the hypervisor to have a serial console access to the virtual machine. A serial console is a text interface to a machine, you get the machine output displayed in your serial console client, and what you type is sent to the machine as if you had a keyboard connected to it.

It can be useful to read the onboarding guide before starting.

OpenBSD Amsterdam onboarding guide

3.1. Get into the OpenBSD installer §

Our first step is to get into the OpenBSD installer, so we can use it to overwrite the disk with our VM.

Connect to the hypervisor, attach to your virtual machine serial console by using the following command, we admit your VM name is "vm40" in the example:

vmctl console vm40

You can leave the console anytime by typing "~~." to get back into your ssh shell. The keys sequence "~." is used to drop ssh or a local serial console, but when you need to leave a serial console from a ssh shell, you need to use "~~.".

You shouldn't see anything because you won't get anything displayed until something is showed in the machine virtual first tty, you can press "enter" and you should see a login prompt. We don't need it, but it confirms the serial console is working.

In parallel, connect to your VM using ssh, find the root password at the end of ~/.ssh/authorized_keys, use "su -" to become root and run "reboot".

You should see the shutdown sequence scrolling in the hypervisor ssh session displaying the serial console, wait for the machine to reboot to spot for the login prompt, in which you will type bsd.rd:

Using drive 0, partition 3.
Loading......
probing: pc0 com0 mem[638K 3838M 4352M a20=on]
disk: hd0+
>> OpenBSD/amd64 BOOT 3.53
com0: 115200 baud
switching console to com0
>> OpenBSD/amd64 BOOT 3.53
boot> bsd.rd [ENTER] # you need to type bsd.rd

3.2. Copy the NixOS VM from the installer §

In this step, we will use the installer to fetch the NixOS VM disk and overwrite the local disk with it.

• in the installer, type "S" to get a shell:

[...]
Welcome to the OpenBSD/amd64 7.2 installation program.
(I)nstall, (U)pgrade, (A)utoinstall or (S)hell?

• enable the network using DHCP with the command:

ifconfig vio0 up autoconf

• create the disk device in /dev because it's missing by default:

cd /dev
sh MAKEDEV sd0

• fetch the NixOS disk and overwrite the local drive with it:
• (remove the gunzip part if you didn't compress your VM disk file)

ftp -o - https://perso.pw/nixos/vm.disk.gz | gunzip -f -c | dd of=/dev/rsd0c bs=10M

• reboot using the command "reboot"

3.3. NixOS grub menu §

At this step, in the serial console you should see a GRUB boot menu, it will boot the first entry after a few seconds. Then NixOS will start booting. In this menu you can access older versions of your system.

After the text stopped scrolling press enter. You should see a login prompt, you can log in with the username "root" and the default password "nixos" if you used my disk image.

3.4. Configuring NixOS §

If you used my template, your VM still doesn't have network connectivity, you need to edit the file /etc/nixos/configuration.nix in which I've put the most important variables you want to customize at the top of the file. You need to configure your IPv4 and IPv6 addresses and their gateways, and also your username with an ssh key to connect to it, and the system name.

Once you are done, run "nixos-rebuild switch", you should have network if you configured it correctly.

After the rebuild, run "passwd your_user" if you want to assign a password to your newly declared user.

You should be able to connect to your VM using its public IP and your ssh key with your username.

EXTRA: You may want to remove the profile minimal.nix which is imported: it disables documentation and the use of X libraries, but this may trigger packages compilation as they are not always built without X support.

3.5. Resizing the partition (last step) §

Because we started with a small 2 GB raw disk to create the virtual machine, the partition still has 2 GB only. We will have to resize the partition /dev/vda1 to take all the disk space, and then resize the ext4 file system.

First step is to extend the partition to 50 GB, the size of the virtual disk offered at openbsd.amsterdam.

# nix-shell -p parted
# parted /dev/vda
(parted) resizepart 1
Warning: The partition /dev/vda1 is currently in use. Are you sure to continue?
Yes/No? yes
End? [2147MB]? 50GB
(parted) quit

Second step is to resize the file system to fill up the partition:

# resize2fs /dev/vda1
The file system /dev/vda1 is mounted on / ; Resizing done on the fly
old_desc_blocks = 1, new_desc_blocks = 6
The file system /dev/vda1 now has a size of 12206775 blocks (4k).

Done! "df -h /" should report the new size.

3.6. Congratulations §

You have a fully functional NixOS VM!

4. Creating the VM §

While I provide a bootable NixOS disk image at https://perso.pw/nixos/vm.disk.gz , you can generate yours with this guide.

• create a raw disk of 2 GB to install the VM in it

qemu-img create -f raw vm.disk 2G

• run qemu in a serial console to ensure it works, in the grub boot menu you will need to select the 4th choice enabling serial console in the installer. In this no graphics qemu mode, you can stop qemu by pressing "ctrl+a" and then "c" to drop into qemu's own console, and type "quit" to stop the process.

qemu-system-x86_64 \
  -smp 2 -m 4G \
  -enable-kvm \
  -display curses -nographic \
  -cdrom nixos-minimal*.iso \
  -drive file=vm.disk,if=virtio,format=raw

• we create the partitions and prepare the chroot

sudo -i
parted /dev/vda -- mklabel msdos
parted /dev/vda -- mkpart primary 1MiB 100%
mkfs.ext4 -L nixos /dev/vda1
mount /dev/disk/by-label/nixos /mnt
mkdir -p /mnt/etc/nixos/

• edit the file /mnt/etc/nixos/configuration.nix , the NixOS install has nano available by default, but you can have your favorite editor by using "nix-shell -p vim" if you prefer vim. Here is a configuration file that will work:

NixOS configuration.nix file for OpenBSD Amsterdam

• edit the file /mnt/etc/nixos/hardware-configuration.nix

NixOS hardware-configuration.nix file for OpenBSD Amsterdam

• we can run the installer, it will ask for the root password, and then we can shut down the VM

nixos-install
systemctl poweroff

Now, you have to host the disk file somewhere to make it available through http or ftp protocol in order to retrieve it from the openbsd.amsterdam VM. I'd recommend compressing the file by running gzip on it, that will drastically reduce its size from 2GB to ~500MB.

5. Full disk encryption §

The ext4 file system offers a way to encrypt specific directories, it can be enough for most users.

However, if you want to enable full disk encryption, you need to use the guide above to generate your VM, but you need to create a separate /boot partition and create a LUKS volume for the root partition. This is explained in the NixOS manual, in the installer section. You should adapt the according bits in the configuration file to match your new setup.

Don't forget you will need to connect to the hypervisor to type your password through the serial access every time you will reboot.

6. Known issue and workaround §

There is an issue with the OpenBSD hypervisor and Linux kernels at the moment, when you reboot your Linux VM, the VM process on the OpenBSD host crashes. Fortunately, it crashes after all the shutdown process is done, so it doesn't let the file system in a weird state.

This problem is fixed in OpenBSD -current as of August 2022, and won't happen in OpenBSD 7.2 hypervisors that will be available by the end of the year.

A simple workaround is to open a tmux session in the hypervisor to run an infinite loop regularly checking if your VM is running, and starting it when it's stopped:

while true ; do vmctl status vm40 | grep stopped && vmctl start vm40 ; sleep 30 ; done

Mailing list archives: vmx_fault_page: uvm_fault returns 14, GPA=0xfe001818, rip=0xffffffffc0d6bb96

Mailing list archives: vmm page fault with VM upgraded from Ubuntu 18LTS to 20LTS

7. Conclusion §

It's great to have more choice when you need a VM. The OpenBSD Amsterdam team is very kind, professional and regularly give money to the OpenBSD project.

8. Going further §

This method should work for other hosting providers, given you can access the VM disk from an live environment (installer, rescue system etc..). You may need to pay attention to the disk device, and if you can't obtain a serial console access to your system, you need to get the network done right in the VM before copying it to the disk.

In the same vein, you can use this method to install any operating system supported by the hypervisor. I chose NixOS because I love this system, and it's easy to reproduce a result with its declarative paradigm.