1. Introduction §

As a recent Qubes OS user, but also a NixOS user, I want to be able to reproduce my system configuration instead of fiddling with files everywhere by hand and being clueless about what I changed since the installation time.

Fortunately, Qubes OS is managed internally with Salt Stack (it's similar to Ansible if you didn't know about Salt), so we can leverage salt to modify dom0 or Qubes templates/VMs.

Qubes OS official project website

Salt Stack project website

Qubes OS documentation: Salt

2. Simple setup §

In this example, I'll show how to write a simple Salt state files, allowing you to create/modify system files, install packages, add repositories etc...

Everything will happen in dom0, you may want to install your favorite text editor in it. Note that I'm still trying to figure a nice way to have a git repository to handle this configuration, and being able to synchronize it somewhere, but I still can't find a solution I like.

The dom0 salt configuration can be found in /srv/salt/, this is where we will write:

• a .top file that is used to associate state files to apply to which hosts
• a state file that contain actual instructions to run

Quick extra explanation: there is a directory /srv/pillar/, where you store things named "pillars", see them as metadata you can associate to remote hosts (AppVM / Templates in the Qubes OS case). We won't use pillars in this guide, but if you want to write more advanced configurations, you will surely need them.

3. dom0 management §

Let's use dom0 to manage itself 🙃.

Create a text file /srv/salt/custom.top with the content (YAML format):

base:
  'dom0':
    - dom0

This tells that hosts matching dom0 (2nd line) will use the state named dom0.

We need to enable that .top file so it will be included when salt is applying the configuration.

qubesctl top.enable custom

Now, create the file /srv/salt/dom0.sls with the content (YAML format):

my packages:
  pkg.installed:
    - pkgs:
      - kakoune
      - git

This uses the salt module named pkg, and pass it options in order to install the packages "git" and "kakoune".

Salt Stack documentation about the pkg module

On my computer, I added the following piece of configuration to /srv/salt/dom0.sls to automatically assign the USB mouse to dom0 instead of being asked every time, this implements the instructions explained in the documentation link below:

Qubes OS documentation: USB mice

/etc/qubes-rpc/policy/qubes.InputMouse:
  file.line:
    - mode: ensure
    - content: "sys-usb dom0 allow"
    - before: "^sys-usb dom0 ask"

Salt Stack documentation: file line

This snippet makes sure that the line sys-usb dom0 allow in the file /etc/qubes-rpc/policy/qubes.InputMouse is present above the line matching ^sys-usb dom0 ask. This is a more reproducible way of adding lines to configuration file than editing by hand.

Now, we need to apply the changes by running salt on dom0:

qubesctl --target dom0 state.apply

You will obtain a list of operations done by salt, with a diff for each task, it will be easy to know if something changed.

Note: state.apply used to be named state.highstate (for people who used salt a while ago, don't be confused, it's the same thing).

4. Template management §

Using the same method as above, we will add a match for the fedora templates in the custom top file:

In /srv/salt/custom.top add:

  'fedora-*':
    - globbing: true
    - fedora

This example is slightly different than the one for dom0 where we matched the host named "dom0". As I want my salt files to require the least maintenance possible, I won't write the template name verbatim, but I'd rather use a globbing (this is the name for simpler wildcard like foo*) matching everything starting by fedora-, I currently have fedora-37 and fedora-38 on my computer, so they are both matching.

Create /srv/salt/fedora.sls:

custom packages:
  pkg.installed:
    - pkgs:
      - borgbackup
      - dino
      - evolution
      - fossil
      - git
      - pavucontrol
      - rsync
      - sbcl
      - tig

In order to apply, we can type qubesctl --all state.apply, this will work but it's slow as salt will look for changes in each VM / template (but we only added changes for fedora templates here, so nothing would change except for the fedora templates).

For a faster feedback loop, we can specify one or multiple targets, for me it would be qubesctl --targets fedora-37,fedora-38 state.apply, but it's really a matter of me being impatient.

5. Auto configure Split SSH §

An interesting setup with Qubes OS is to have your SSH key in a separate VM, and use Qubes OS internal RPC to use the SSH from another VM, with a manual confirmation on each use. However, this setup requires modifying files at multiple places, let's see how to manage everything with salt.

Qubes OS community documentation: Split SSH

Reusing the file /srv/salt/custom.top created earlier, we add split_ssh_client.sls for some AppVMs that will use the split SSH setup. Note that you should not deploy this state to your Vault, it would self reference for SSH and would prevent the agent to start (been there :P):

base:
  'dom0':
    - dom0
  'fedora-*':
    - globbing: true
    - fedora
  'MyDevAppVm or MyWebBrowserAppVM':
    - split_ssh_client

Create /srv/salt/split_ssh_client.sls: this will add two files to load the environment variables from /rw/config/rc.local and ~/.bashrc. It's actually easier to separate the bash snippets in separate files and use source, rather than using salt to insert the snippets directly in place where needed.

/rw/config/bashrc_ssh_agent:
  file.managed:
    - user: root
    - group: wheel
    - mode: 444
    - contents: |
        SSH_VAULT_VM="vault"
        if [ "$SSH_VAULT_VM" != "" ]; then
          export SSH_AUTH_SOCK="/home/user/.SSH_AGENT_$SSH_VAULT_VM"
        fi

/rw/config/rclocal_ssh_agent:
  file.managed:
    - user: root
    - group: wheel
    - mode: 444
    - contents: |
        SSH_VAULT_VM="vault"
        if [ "$SSH_VAULT_VM" != "" ]; then
          export SSH_SOCK="/home/user/.SSH_AGENT_$SSH_VAULT_VM"
          rm -f "$SSH_SOCK"
          sudo -u user /bin/sh -c "umask 177 && exec socat 'UNIX-LISTEN:$SSH_SOCK,fork' 'EXEC:qrexec-client-vm $SSH_VAULT_VM qubes.SshAgent'" &
        fi

/rw/config/rc.local:
  file.append:
    - text: source /rw/config/rclocal_ssh_agent

/rw/home/user/.bashrc:
  file.append:
    - text: source /rw/config/bashrc_ssh_agent

Edit /srv/salt/dom0.sls to add the SshAgent RPC policy:

/etc/qubes-rpc/policy/qubes.SshAgent:
  file.managed:
    - user: root
    - group: wheel
    - mode: 444
    - contents: |
        MyClientSSH vault ask,default_target=vault

Now, run qubesctl --all state.apply to configure all your VMs, which are the template, dom0 and the matching AppVMs. If everything went well, you shouldn't have errors when running the command.

6. Use a dedicated AppVM for web browsing §

Another real world example, using Salt to configure your AppVMs to open links in a dedicated AppVM (named WWW for me):

Qubes OS Community Documentation: Opening URLs in VMs

In your custom top file /srv/salt/custom.top, you need something similar to this (please adapt if you already have top files or state files):

  'dom0':
    - dom0
  'fedora-*':
     - globbing: true
     - fedora
  'vault or qubes-communication or qubes-devel':
    - default_www

Add the following text to /srv/salt/dom0.sls, this is used to configure the RPC:

/etc/qubes-rpc/policy/qubes.OpenURL:
  file.managed:
    - user: root
    - group: wheel
    - mode: 444
    - contents: |
        @anyvm @anyvm ask,default_target=WWW

Add this to /srv/salt/fedora.sls to create the desktop file in the template:

/usr/share/applications/browser_vm.desktop:
  file.managed:
    - user: root
    - group: wheel
    - mode: 444
    - contents: |
        [Desktop Entry]
        Encoding=UTF-8
        Name=BrowserVM
        Exec=qvm-open-in-vm browser %u
        Terminal=false
        X-MultipleArgs=false
        Type=Application
        Categories=Network;WebBrowser;
        MimeType=x-scheme-handler/unknown;x-scheme-handler/about;text/html;text/xml;application/xhtml+xml;application/xml;application/rss+xml;x-scheme-handler/http;x-scheme-handler/https;

Create /srv/salt/default_www.sls with the following content, this will run xdg-settings to set the default browser:

xdg-settings set default-web-browser browser_vm.desktop:
  cmd.run:
    - runas: user

Now, run qubesctl --target fedora-38,dom0 state.apply.

From there, you MUST reboot the VMs that will be configured to use the WWW AppVm as the default browser, they need to have the new file browser_vm.desktop available for xdg-settings to succeed. Run qubesctl --target vault,qubes-communication,qubes-devel state.apply.

Congratulations, now you will have a RPC prompt when an AppVM wants to open a file to ask you if you want to open it in your browsing AppVM.

7. Conclusion §

This method is a powerful way to handle your hosts, and it's ready to use on Qubes OS. Unfortunately, I still need to figure a nicer way to export the custom files written in /srv/salt/ and track the changes properly in a version control system.

Erratum: I found a solution to manage the files :-) stay tuned for the next article.