Nvidia card in eGPU and NixOS

Written by Solène, on 05 December 2021.
Tags: #linux #games #nixos #egpu

1. Updates
2. Introduction
3. What is it about?
4. Nvidia offloading
5. NixOS
6. How to use it
- 6. 1. Running a command
- 6. 2. In Steam
- 6. 3. In Lutris
7. Advanced usage / boot specialisation

1. Updates §

2022-01-02: add entry about specialization and how to use the eGPU as a display device

2. Introduction §

I previously wrote about using an eGPU on Gentoo Linux. It was working when using the eGPU display but I never got it to work for accelerating games using the laptop display.

Now, I'm back on NixOS and I got it to work!

3. What is it about? §

My laptop has a thunderbolt connector and I'm using a Razer Core X external GPU case that is connected to the laptop using a thunderbolt cable. This allows to use an external "real" GPU on a laptop but it has performance trade off and on Linux also compatibility issues.

There are three ways to use the nvidia eGPU:

- run the nvidia driver and use it as a normal card with its own display connected to the GPU, not always practical with a laptop

- use optirun / primerun to run programs within a virtual X server on that GPU and then display it on the X server (very clunky, originally created for Nvidia Optimus laptop)

- use Nvidia offloading module (it seems recent and I learned about it very recently)

The first case is easy, just install nvidia driver and use the right card, it should work on any setup. This is the setup giving best performance.

The most complicated setup is to use the eGPU to render what's displayed on the laptop, meaning the video signal has to come back from the thunderbolt cable, reducing the bandwidth.

4. Nvidia offloading §

Nvidia made work in their proprietary driver to allow a program to have its OpenGL/Vulkan calls to be done in a GPU that is not the one used for the display. This allows to throw optirun/primerun for this use case, which is good because they added performance penalty, complicated setup and many problems.

Official documentation about offloading with nvidia driver

5. NixOS §

I really love NixOS and for writing articles it's so awesome, because instead of a set of instructions depending on conditions, I only have to share the piece of config required.

This is the bits to add to your /etc/nixos/configuration.nix file and then rebuild system:

hardware.nvidia.modesetting.enable = true;
hardware.nvidia.prime.sync.allowExternalGpu = true;
hardware.nvidia.prime.offload.enable = true;
hardware.nvidia.prime.nvidiaBusId = "PCI:10:0:0";
hardware.nvidia.prime.intelBusId = "PCI:0:2:0";
services.xserver.videoDrivers = ["nvidia" ];

A few notes about the previous chunk of config:

- only add nvidia to the list of video drivers, at first I was adding modesetting but this was creating troubles

- the PCI bus ID can be found with lspci, it has to be translated in decimal, here my nvidia id is 10:0:0 but in lspci it's 0a:00:00 with 0a being 10 in hexadecimal

NixOS wiki about nvidia offload mode

6. How to use it §

The use of offloading is controlled by environment variables. What's pretty cool is that if you didn't connect the eGPU, it will still work (with integrated GPU).

6.1. Running a command §

We can use glxinfo to be sure it's working, add the environment as a prefix:

__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia glxinfo

6.2. In Steam §

Modify the command line of each game you want to run with the eGPU (it's tedious), by:

__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia %command%

6.3. In Lutris §

Lutris has a per-game or per-runner setting named "Enable Nvidia offloading", you just have to enable it.

7. Advanced usage / boot specialisation §

Previously I only explained how to use the laptop screen and the eGPU as a discrete GPU (not doing display). For some reasons, I've struggled a LOT to be able to use the eGPU display (which gives more performance because it's hitting less thunderbolt limitations).

I've discovered NixOS "specialisation" feature, allowing to add an alternative boot entry to start the system with slight changes, in this case, this will create a new "external-display" entry for using the eGPU as the primary display device:

  hardware.nvidia.modesetting.enable = true;
  hardware.nvidia.prime.sync.allowExternalGpu = true;
  hardware.nvidia.prime.offload.enable = true;
  hardware.nvidia.prime.nvidiaBusId = "PCI:10:0:0";
  hardware.nvidia.prime.intelBusId = "PCI:0:2:0";
  services.xserver.videoDrivers = ["nvidia" ];

  # external display on the eGPU card
  # otherwise it's discrete mode using laptop screen
  specialisation = {
    external-display.configuration = {
        system.nixos.tags = [ "external-display" ];
        hardware.nvidia.modesetting.enable = pkgs.lib.mkForce false;
        hardware.nvidia.prime.offload.enable = pkgs.lib.mkForce false;
        hardware.nvidia.powerManagement.enable = pkgs.lib.mkForce false;
        services.xserver.config = pkgs.lib.mkOverride 0
  ''
Section "Module"
    Load           "modesetting"
EndSection

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    BusID          "10:0:0"
    Option         "AllowEmptyInitialConfiguration"
    Option         "AllowExternalGpus" "True"
EndSection
'';
    };
  };

With this setup, the default boot is the offloading mode but I can choose "external-display" to use my nvidia card and the screen attached to it, it's very convenient.

I had to force the xserver configuration file because the one built by NixOS was not working for me.

Table of contents