1. Introduction §
If you are using SSH quite often, it is likely you use an SSH agent which stores your private key in memory so you do not have to type your password every time.
This method is convenient, but it comes at the expense of your SSH key use security, anyone able to use your session while the agent holds the key unlocked can use your SSH key. This scenario is most likely to happen when using a compromised build script.
However, it is possible to harden this process at a small expense of convenience, make your SSH agent ask for confirmation every time the key has to be used.
2. Setup §
The tooling provided with OpenSSH comes with a simple SSH agent named ssh-agent. On OpenBSD, the agent is automatically started and ask to unlock your key upon graphical login if it finds a SSH key in the default path (like ~/.ssh/id_rsa).
Usually, the method to run the ssh-agent is the following. In a shell script defining your environment at an early stage, either your interactive shell configuration file or the script running your X session, you use eval $(ssh-agent -s). This command runs ssh-agent and also enable the environment variables to make it work.
Once your ssh-agent is correctly configured, it is required to add a key into it, now, here are two methods to proceed.
2.1. OpenSSH ssh-add §
In addition to ssh-agent, OpenSSH provides ssh-add to load keys into the agent. It is simple of use, just run ssh-add /path/to/key.
ssh-add manual page
If you want to have a GUI confirmation upon each SSH key use, just add the flag -c to this command line: ssh-add -c /path/to/key.
In OpenBSD, if you have your key at a standard location, you can modify the script /etc/X11/xenodm/Xsession to change the first occurrence of ssh-add by ssh-add -c. You will still be greeting for your key password upon login, but you will be asked for each of its use.
2.2. KeepassXC §
It turns out the password manager KeepassXC can hold SSH keys, it works great for having used for a while. KeepassXC can either store the private key within its database or load a private key from the filesystem using a path and unlock it using a stored password, the choice is up to you.
You need to have the ssh-agent variables in your environment to have the feature work, as KeepassXC will replace ssh-add only, not the agent.
KeepassXC documentation has a "SSH Agent integration" section explaining how it works and how to configure it.
KeepassXC official documentation
In the key settings and "SSH Agent" tab, there is a checkbox to ask user confirmation upon each key use.
3. Other security features §
3.1. Timeout §
I would recommend to automatically delete the key from the agent after some time, this is especially useful if you do not actively use your SSH key.
In ssh-add, this can be achieved using -t time flag (it's tea time, if you want to remember about it), where time is a number of seconds or a time format specified in sshd_config, like 5s for 5 seconds, 10m for 10 minutes, 16h for 16 hours or 2d for 2 days.
In KeepassXC, it's in the key settings, within the SSH agent tab, you can configure the delay before the key is removed from the agent.
4. Conclusion §
The ssh-agent is a practical software that ease the use of SSH keys without much compromise with regards to security, but some extra security could be useful in certain scenarios, especially for developers running untrusted code as their user holding the SSH key.
While the extra confirmation could still be manipulated by a rogue script, it would come with a greater complexity at the cost of being spotted more easily. If you really want to protect your SSH keys, you should use them from a hardware token requiring a physical action to unlock it. While I find those tokens not practical and expensive, they have their use and they can not be beaten by a pure software solution.