Solene'%

Using a dedicated administration workstation for my infrastructure

Wed, 23 Oct 2024 00:00:00 GMT

1. Introduction
2. The need
3. Setup
- 3. 1. Workstation
- 3. 2. Servers
- 3. 3. File exchange
4. Conclusion

1. Introduction §

As I moved my infrastructure to a whole new architecture, I decided to only expose critical accesses to dedicated administration systems (I have just one). That workstation is dedicated to my infrastructure administration, it can only connect to my servers over a VPN and can not reach the Internet.

This blog post explains why I am doing this, and gives a high level overview of the setup. Implementation details are not fascinating as it only requires basics firewall, HTTP proxy and VPN configuration.

2. The need §

I wanted to have my regular computer not being able to handle any administration task, so I have a computer "like a regular person" without SSH keys, VPN and a password manager that does not mix personal credentials with administration credentials ... To prevent credentials leaks or malware risks, it makes sense to uncouple the admin role from the "everything else" role. So far, I have been using Qubes OS which helped me to do so at the software level, but I wanted to go further.

3. Setup §

This is a rather quick and simple explanation of what you have to do in order to set up a dedicated system for administration tasks.

3.1. Workstation §

The admin workstation I use is an old laptop, it only needs a web browser (except if you have no internal web services), a SSH client, and being able to connect to a VPN. Almost any OS can do it, just pick the one you are the most conformable with, especially with regard to the firewall configuration.

The workstation has its own SSH key that is deployed on the servers. It also has its own VPN to the infrastructure core. And its own password manager.

Its firewall is configured to block all in and out traffic except the following:

UDP traffic to allow WireGuard
HTTP proxy address:port through WireGuard interface
SSH through WireGuard

The HTTP proxy exposed on the infrastructure has a whitelist to allow some fqdn. I actually want to use the admin workstation for some tasks, like managing my domains through my registrar web console. Keeping the list as small as possible is important, you do not want to start using this workstation for browsing the web or reading emails.

On this machine, make sure to configure the system to use the HTTP proxy for updates and installing packages. The difficulty of doing so will vary from an operating system to another. While Debian required a single file in /etc/apt/apt.conf.d/ to configure apt to use the HTTP proxy, OpenBSD needed both http_proxy and https_proxy environment variables, but some scripts needed to be patched as they do not use the variables, I had to check fw_update, pkg_add, sysupgrade and syspatch were all working.

Ideally, if you can afford it, configure a remote logging of this workstation logs to a central log server. When available, auditd monitoring important files access/changes in /etc could give precious information.

3.2. Servers §

My SSH servers are only reachable through a VPN, I do not expose it publicly anymore. And I do IP filtering over the VPN, so only the VPN clients that have a use to connect over SSH will be allowed to connect.

When I have some web interfaces for services like Minio, Pi-Hole and the monitoring dashboard, all of that is restricted to the admin workstations only. Sometimes, you have the opportunity to separate the admin part by adding a HTTP filter on a /admin/ URI, or if the service uses a different port for the admin and the service (like Minio). When enabling a new service, you need to think about all the things you can restrict to the admin workstations only.

Depending on your infrastructure size and locations, you may want to use dedicated systems for SSH/VPN/HTTP proxy entry points, it is better if it is not shared with important services.

3.3. File exchange §

You will need to exchange data to the admin workstation (rarely the other way), I found nncp to be a good tool for that. You can imagine a lot of different setup, but I recommend picking one that:

does not require a daemon on the admin workstation: this does not increase the workstation attack surface
allows encryption at rest: so you can easily use any deposit system for the data exchange
is asynchronous: as a synchronous connection could be potentially dangerous because it establishes a link directly between the sender and the receiver

Previous blog post: Secure file transfer with NNCP

4. Conclusion §

I learned about this method while reading ANSSI (French cybersecurity national agency) papers. While it may sound extreme, it is a good practice I endorse. This gives a use to old second hand hardware I own, and it improves my infrastructure security while giving me peace of mind.

ANSSI website (in French)

In addition, if you want to allow some people to work on your infrastructure (maybe you want to set up some infra for an association?), you already have the framework to restrict their scope and trace what they do.

Of course, the amount of complexity and resources you can throw at this is up to you, you could totally have a single server and lock most of its services behind a VPN and call it a day, or have multiple servers worldwide and use dedicated servers to enter their software defined network.

Last thing, make sure that you can bootstrap into your infrastructure if the only admin workstation is lost/destroyed. Most of the time, you will have a physical/console access that is enough (make sure the password manager is reachable from the outside for this case).

Securing backups using S3 storage

Tue, 22 Oct 2024 00:00:00 GMT

1. Introduction
2. Quick intro to object storage
- 2. 1. Open source S3-compatible storage implementations
3. Configure your S3
4. Conclusion
5. Going further

1. Introduction §

In this blog post, you will learn how to make secure backups using Restic and a S3 compatible object storage.

Backups are incredibly important, you may lose important files that only existed on your computer, you may lose access to some encrypted accounts or drives, when you need backups, you need them to be reliable and secure.

There are two methods to handle backups:

pull backups: a central server connects to the system and pulls data to store it locally, this is how rsnapshot, backuppc or bacula work
push backups: each system run the backup software locally to store it on the backup repository (either locally or remotely), this is how most backups tool work

Both workflows have pros and cons. The pull backups are not encrypted, and a single central server owns everything, this is rather bad from a security point of view. While push backups handle all encryption and accesses to the system where it runs, an attacker could destroy the backup using the backup tool.

I will explain how to leverage S3 features to protect your backups from an attacker.

2. Quick intro to object storage §

S3 is the name of an AWS service used for Object Storage. Basically, it is a huge key-value store in which you can put data and retrieve it, there are very little metadata associated with an object. Objects are all stored in a "bucket", they have a path, and you can organize the bucket with directories and subdirectories.

Buckets can be encrypted, which is an important feature if you do not want your S3 provider to be able to access your data, however most backup tools already encrypt their repository, so it is not really useful to add encryption to the bucket. I will not explain how to use encryption in the bucket in this guide, although you can enable it if you want. Using encryption requires more secrets to store outside of the backup system if you want to restore, and it does not provide real benefits because the repository is already encrypted.

S3 was designed to be highly efficient for retrieving / storage data, but it is not a competitor to POSIX file systems. A bucket can be public or private, you can host your website in a public bucket (and it is rather common!). A bucket has permissions associated to it, you certainly do not want to allow random people to put files in your public bucket (or list the files), but you need to be able to do so.

The protocol designed around S3 was reused for what we call "S3-compatible" services on which you can directly plug any "S3-compatible" client, so you are not stuck with AWS.

This blog post exists because I wanted to share a cool S3 feature (not really S3 specific, but almost everyone implemented this feature) that goes well with backups: a bucket can be versioned. So, every change happening on a bucket can be reverted. Now, think about an attacker escalating to root privileges, they can access the backup repository and delete all the files there, then destroy the server. With a backup on a versioned S3 storage, you could revert your bucket just before the deletion happened and recover your backup. In order to prevent this, the attacker should also get access to the S3 storage credentials, which is different from the credentials required to use the bucket.

Finally, restic supports S3 as a backend, and this is what we want.

2.1. Open source S3-compatible storage implementations §

There is a list of open source and free S3-compatible storage, I played with them all, and they have different goals and purposes, they all worked well enough for me:

Seaweedfs GitHub project page

Garage official project page

Minio official project page

A quick note about those:

I consider seaweedfs to be the Swiss army knife of storage, you can mix multiple storage backends and expose them over different protocols (like S3, HTTP, WebDAV), it can also replicate data over remote instances. You can do tiering (based on last access time or speed) as well.
Garage is a relatively new project, it is quite bare bone in terms of features, but it works fine and support high availability with multiple instances, it only offers S3.
Minio is the big player, it has a paid version (which is extremely expensive) although the free version should be good enough for most users.

3. Configure your S3 §

You need to pick a S3 provider, you can self-host it or use a paid service, it is up to you. I like backblaze as it is super cheap, with $6/TB/month, but I also have a local minio instance for some needs.

Create a bucket, enable the versioning on it and define the data retention, for the current scenario I think a few days is enough.

Create an application key for your restic client with the following permissions: "GetObject", "PutObject", "DeleteObject", "GetBucketLocation", "ListBucket", the names can change, but it needs to be able to put/delete/list data in the bucket (and only this bucket!). After this process done, you will get a pair of values: an identifier and a secret key

Now, you will have to provide the following environment variables to restic when it runs:

AWS_DEFAULT_REGION which contains the region of the S3 storage, this information is given when you configure the bucket.
AWS_ACCESS_KEY which contains the access key generated when you created the application key.
AWS_SECRET_ACCESS_KEY which contains the secret key generated when you created the application key.
RESTIC_REPOSITORY which will look like s3:https://$ENDPOINT/$BUCKET with $ENDPOINT being the bucket endpoint address and $BUCKET the bucket name.
RESTIC_PASSWORD which contains your backup repository passphrase to encrypt it, make sure to write it down somewhere else because you need it to recover the backup.

If you want a simple script to backup some directories, and remove old data after a retention of 5 hourly, 2 daily, 2 weekly and 2 monthly backups:

restic backup -x /home /etc /root /var
restic forget --prune -H 5 -d 2 -w 2 -m 2

Do not forget to run restic init the first time, to initialize the restic repository.

4. Conclusion §

I really like this backup system as it is cheap, very efficient and provides a fallback in case of a problem with the repository (mistakes happen, there is not always need for an attacker to lose data ^_^').

If you do not want to use S3 backends, you need to know Borg backup and Restic both support an "append-only" method, which prevents an attacker from doing damages or even read the backup, but I always found the use to be hard, and you need to have another system to do the prune/cleanup on a regular basis.

5. Going further §

This approach could work on any backend supporting snapshots, like BTRFS or ZFS. If you can recover the backup repository to a previous point in time, you will be able to access to the working backup repository.

You could also do a backup of the backup repository, on the backend side, but you would waste a lot of disk space.

Snap integration in Qubes OS templates

Sat, 19 Oct 2024 00:00:00 GMT

1. Introduction
2. Setup on Fedora
- 2. 1. Snap installation
- 2. 2. Proxy configuration
- 2. 3. Run updates on template update
- 2. 4. Qube settings menu integration
- 2. 5. Snap store GUI
3. Debian
4. Conclusion

1. Introduction §

Snap package format is interesting, while it used to have a bad reputation, I wanted to make my opinion about it. After reading its design and usage documentation, I find it quite good, and I have a good experience using some programs installed with snap.

Snapcraft official website (store / documentation)

Snap programs can be either packaged as "strict" or "classic"; when it is strict there is some confinement at work which can be inspected on an installed snap using snap connections $appname, while a "classic" snap has no sandboxing at all. Snap programs are completely decorrelated from the host operating system where snap is running, so you can have old or new versions of a snap packaged program without having to handle shared library versions.

The following setup explains how to install snap programs in a template to run them from AppVMs, and not how to install snap programs in AppVMs as a user, if you need this, please us the Qubes OS guide linked below.

Qubes OS documentation explains how to setup snap in a template, but with a helper to allow AppVMs to install snap programs in the user directory.

Qubes OS official documentation: install snap packages in AppVMs

In a previous blog post, I explained how to configure a Qubes OS template to install flatpak programs in it, and how to integrate it to the template.

Previous blog post: Installing flatpak programs in a Qubes OS template

2. Setup on Fedora §

All commands are meant to be run as root.

2.1. Snap installation §

Snapcraft official documentation: Installing snap on Fedora

Installing snap is easy, run the following command:

dnf install snapd

To allow "classic" snaps to work, you need to run the following command:

sudo ln -s /var/lib/snapd/snap /snap

2.2. Proxy configuration §

Now, you have to configure snap to use the http proxy in the template, this command can take some time because snap will time out as it tries to use the network when invoked...

snap set system proxy.http="http://127.0.0.1:8082/"
snap set system proxy.https="http://127.0.0.1:8082/"

2.3. Run updates on template update §

You need to prevent snap from searching for updates on its own as you will run updates when the template is updated:

snap refresh --hold

To automatically update snap programs when the template is updating (or doing any dnf operation), create the file /etc/qubes/post-install.d/05-snap-update.sh with the following content and make it executable:

#!/bin/sh

if [ "$(qubesdb-read /type)" = "TemplateVM" ]
then
    snap refresh
fi

2.4. Qube settings menu integration §

To add the menu entry of each snap program in the qube settings when you install/remove snaps, create the file /usr/local/sbin/sync-snap.sh with the following content and make it executable:

#!/bin/sh

# when a desktop file is created/removed
# - links snap .desktop in /usr/share/applications
# - remove outdated entries of programs that were removed
# - sync the menu with dom0

inotifywait -m -r \
-e create,delete,close_write \
/var/lib/snapd/desktop/applications/ |
while  IFS=':' read event
do
    find /var/lib/snapd/desktop/applications/ -type l -name "*.desktop" | while read line
    do
        ln -s "$line" /usr/share/applications/
    done
    find /usr/share/applications/ -xtype l -delete
    /etc/qubes/post-install.d/10-qubes-core-agent-appmenus.sh
done

Install the package inotify-tools to make the script above working, and add this to /rw/config/rc.local to run it at boot:

/usr/local/bin/sync-snap.sh &

You can run the script now with /usr/local/bin/sync-snap.sh & if you plan to install snap programs.

2.5. Snap store GUI §

If you want to browse and install snap programs using a nice interface, you can install the snap store.

snap install snap-store

You can run the store with snap run snap-store or configure your template settings to add the snap store into the applications list, and run it from your Qubes OS menu.

3. Debian §

The setup on Debian is pretty similar, you can reuse the Fedora guide except you need to replace dnf by apt.

Snapcraft official documentation: Installing snap on Debian

4. Conclusion §

More options to install programs is always good, especially when it comes with features like quota or sandboxing. Qubes OS gives you the flexibility to use multiple templates in parallel, a new source of packages can be useful for some users.

Asynchronous secure file transfer with nncp

Sun, 06 Oct 2024 00:00:00 GMT

1. Introduction
2. Explanations
3. Real world example: Syncthing gateway
4. Setup
- 4. 1. Configuration file and private keys
- 4. 2. Public keys
- 4. 3. Import public keys
5. Usage
6. Workflow (how to use)
- 6. 1. Sending files
- 6. 2. Sync and file unpacking
7. Explanations about ACK
8. Conclusion
9. Going further

1. Introduction §

nncp (node to node copy) is a software to securely exchange data between peers. Is it command line only, it is written in Go and compiles on Linux and BSD systems (although it is only packaged for FreeBSD in BSDs).

The website will do a better job than me to talk about the numerous features, but I will do my best to explain what you can do with it and how to use it.

nncp official project website

2. Explanations §

nncp is a suite of tools to asynchronously exchange data between peers, using zero knowledge encryption. Once peers have exchanged their public keys, they are able to encrypt data to send to this peer, this is nothing really new to be honest, but there is a twist.

a peer can directly connect to another using TCP, you can even configure different addresses like a tor onion or I2P host and use the one you want
a peer can connect to another using ssh
a peer can generate plain files that will be carried over USB, network storage, synchronization software, whatever, to be consumed by a peer. Files can be split in chunks of arbitrary size in order to prevent anyone snooping from figuring how many files are exchanged or their name (hence zero knowledge).
a peer can generate data to burn on a CD or tape (it is working as a stream of data instead of plain files)
a peer can be reachable through another relay peer
when a peer receives files, nncp generates ACK files (acknowledgement) that will tell you they correctly received it
a peer can request files and/or trigger pre-configured commands you expose to this peer
a peer can send emails with nncp (requires a specific setup on the email server)
data transfer can be interrupted and resumed

What is cool with nncp is that files you receive are unpacked in a given directory and their integrity is verified. This is sometimes more practical than a network share in which you are never sure when you can move / rename / modify / delete the file that was transferred to you.

I identified a few "realistic" use cases with nncp:

exchange files between air gap environments (I tried to exchange files over sound or QR codes, I found no reliable open source solution)
secure file exchange over physical medium with delivery notification (the medium needs to do a round-trip for the notification)
start a torrent download remotely, prepare the file to send back once downloaded, retrieve the file at your own pace
reliable data transfer over poor connections (although I am not sure if it beats kermit at this task :D )
"simple" file exchange between computers / people over network

This let a lot of room for other imaginative use cases.

3. Real world example: Syncthing gateway §

My preferred workflow with nncp that I am currently using is a group of three syncthing servers.

Each syncthing server is running on a different computer, the location does not really matter. There is a single share between these syncthing instances.

The servers where syncthing are running have incoming and outgoing directories exposed over a NFS / SMB share, with a directory named after each peer in both directories. Deposing a file in the "outgoing" directory of a peer will make nncp to prepare the file for this peer, put it into the syncthing share and let it share, the file is consumed in the process.

In the same vein, in the incoming directory, new files are unpacked in the incoming directory of emitting peer on the receiver server running syncthing.

Why is it cool? You can just drop a file in the peer you want to send to, it disappears locally and magically appears on the remote side. If something wrong happens, due to ACK, you can verify if the file was delivered and unpacked. With three shares, you can almost have two connected at the same time.

It is a pretty good file deposit that requires no knowledge to use.

This could be implemented with pure syncthing, however you would have to:

for each peer, configure a one-way directory share in syncthing for each other peer to upload data to
for each peer, configure a one-way directory share in syncthing for each other peer to receive data from
for each peer, configure an encrypted share to relay all one way share from other peers

This does not scale well.

Side note, I am using syncthing because it is fun and requires no infrastructure. But actually, a webdav filesystem, a Nextcloud drive or anything to share data over the network would work just fine.

4. Setup §

4.1. Configuration file and private keys §

On each peer, you have to generate a configuration file with its private keys. The default path for the configuration file is /etc/nncp.hjson but nothing prevents you from storing this file anywhere, you will have to use the parameter -cfg /path/to/config file in that case.

Generate the file like this:

nncp-cfgnew > /etc/nncp.hjson

The file contains comments, this is helpful if you want to see how the file is structured and existing options. Never share the private keys of this file!

I recommend checking the spool and log paths, and decide which user should use nncp. For instance, you can use /var/spool/nncp to store nncp data (waiting to be delivered or unpacked) and the log file, and make your user the owner of this directory.

4.2. Public keys §

Now, generate the public keys (they are just derived from the private keys generated earlier) to share with your peers, there is a command for this that will read the private keys and output the public keys in a format ready to put in the nncp.hjson file of recipients.

nncp-cfgmin > my-peer-name.pub

You can share the generated file with anyone, this will allow them to send you files. The peer name of your system is "self", you can rename it, it is just an identifier.

4.3. Import public keys §

When import public keys, you just need to add the content generated by the command nncp-cfgmin of a peer in your nncp configuration file.

Just copy / paste the content in the neigh structure within the configuration file, just make sure to rename "self" by the identifier you want to give to this peer.

If you want to receive data from this peer, make sure to add an attribute line incoming: "/path/to/incoming/data" for that peer, otherwise you will not be able to unpack received file.

5. Usage §

Now you have peers who exchanged keys, they are able to send data to each other. nncp is a collection of tools, let's see the most common and what they do:

nncp-file: add a file in the spool to deliver to a peer
nncp-toss: unpack incoming data (files, commands, file request, emails) and generate ack
nncp-reass: reassemble files that were split in smaller parts
nncp-exec: trigger a pre-configured command on the remote peer, stdin data will be passed as the command parameters. Let's say a peer offers a "wget" service, you can use echo "https://some-domain/uri/" | nncp-exec peername wget to trigger a remote wget.

If you use the client / server model over TCP, you will also use:

nncp-daemon: the daemon waiting for connections
nncp-caller: a daemon occasionally triggering client connections (it works like a crontab)
nncp-call: trigger a client connection to a peer

If you use asynchronous file transfers, you will use:

nncp-xfer: generates to / consumes files from a directory for async transfer

6. Workflow (how to use) §

6.1. Sending files §

For sending files, just use nncp-file file-path peername:, the file name will be used when unpacked, but you can also give the filename you want to give once unpacked.

A directory could be used as a parameter instead of a file, it will be stored automatically in a .tar file for delivery.

Finally, you can send a stream of data using nncp-file stdin, but you have to give a name to the resulting file.

6.2. Sync and file unpacking §

This was not really clear from the documentation, so here it is how to best use nncp when exchanging files using plain files, the destination is /mnt/nncp in my examples (it can be an external drive, a syncthing share, a NFS mount...):

When you want to sync, always use this scheme:

nncp-xfer -rx /mnt/nncp
nncp-toss -gen-ack
nncp-xfer -keep -tx -mkdir /mnt/nncp
nncp-rm -all -ack

This receives files using nncp-xfer -rx, the files are stored in nncp spool directory. Then, with nncp-toss -gen-ack, the files are unpacked to the "incoming" directory of each peer who sent files, and ACK are generated (older versions of nncp-toss does not handle ack, you need to generate ack befores and remove them after tx, with nncp-ack -all 4>acks and nncp-rm -all -pkt < acks).

nncp-xfer -tx will put in the directory the data you want to send to peers, and also the ack files generated by the rx which happened before. The -keep flag is crucial here if you want to make use of ACK, with -keep, the sent data are kept in the pool until you receive the ACK for them, otherwise the data are removed from the spool and will not be retransmited if the files were not received. Finally, nncp-rm will delete all ACK files so you will not transmit them again.

7. Explanations about ACK §

From my experience and documentation reading, there are three cases with the spool and ACK:

the shared drive is missing the files you sent (that are still in pool), and you received no ACK, the next time you run nncp-xfer, the files will be transmitted again
when you receive ACK files for files in spool, they are deleted from the spool
when you do not use -keep when sending files with nncp-xfer, the files will not be stored in the spool so you will not be able to know what to retransmit if ACK are missing

ACKs do not clean up themselves, you need to use nncp-rm. It took me a while to figure this, my nodes were sending ACKs to each other repeatedly.

8. Conclusion §

I really like nncp as it allows me to securely transfer files between my computers without having to care if they are online. Rsync is not always possible because both the sender and receiver need to be up at the same time (and reachable correctly).

The way files are delivered is also practical for me, as I already shared above, files are unpacked in a defined directory by peer, instead of remembering I moved something in a shared drive. This removes the doubt about files being in a shared drive: why is it there? Why did I put it there? What was its destination??

I played with various S3 storage to exchange nncp data, but this is for another blog post :-)

9. Going further §

There are more features in nncp, I did not play with all of them.

You can define "areas" in parallel of using peers, you can use emails notifications when a remote receives data from you to have a confirmation, requesting remote files etc... It is all in the documentation.

I have the idea to use nncp on a SMTP server to store encrypted incoming emails until I retrieve them (I am still working at improving the security of email storage), stay tuned :)

I moved my emails to Proton Mail

Sun, 15 Sep 2024 00:00:00 GMT

1. Introduction
2. My needs
3. Proton Mail
- 3. 1. Benefits
- 3. 2. Interesting features
- 3. 3. Shortcomings
- 3. 4. Alternatives
4. My ideal email setup
5. Conclusion
- 5. 1. Update 2024-09-14

1. Introduction §

I recently took a very hard decision: I moved my emails to Proton Mail.

This is certainly a shock for people following this blog for a long time, this was a shock for me as well! This was actually pretty difficult to think this topic objectively, I would like to explain how I came up to this decision.

I have been self-hosting my own email server since I bought my first domain name, back in 2009. The server have been migrated multiple times, from hosting companies to another and regularly changing the underlying operating system for fun. It has been running on: Slackware, NetBSD, FreeBSD, NixOS and Guix.

2. My needs §

First, I need to explain my previous self-hosted setup, and what I do with my emails.

I have two accounts:

one for my regular emails, mailing lists, friends, family
one for my company to reach client, send quotes and invoices

Ideally, having all the emails retrieved locally and not stored on my server would be ideal. But I am using a lot of devices (most are disposable), and having everything on a single computer will not work for me.

Due to my emails being stored remotely and containing a lot of private information, I have never been really happy with how emails work at all. My dovecot server has access to all my emails, unencrypted and a single password is enough to connect to the server. Adding a VPN helps to protect dovecot if it is not exposed publicly, but the server could still be compromised by other means. OpenBSD smtpd server got critical vulnerabilities patched a few years ago, basically allowing to get root access, since then I have never been really comfortable with my email setup.

I have been looking for ways to secure my emails, this is how I came to the setup encrypting incoming emails with GPG. This is far from being ideal, and I stopped using it quickly. This breaks searches, the server requires a lot of CPU and does not even encrypt all information.

Emails encryption at rest on OpenBSD using dovecot and GPG

Someone shown me a dovecot plugin to encrypt emails completely, however my understanding of the encryption of this plugin is that the IMAP client must authenticate the user using a plain text password that is used by dovecot to unlock an asymmetric encryption key. The security model is questionable: if the dovecot server is compromised, users passwords are available to the attacker and they can decrypt all the emails. It would still be better than nothing though, except if the attacker has root access.

Dovecot encryption plugin: TREES

One thing I need from my emails is to arrive to the recipients. My emails were almost always considered as spam by big email providers (GMail, Microsoft), this has been an issue for me for years, but recently it became a real issue for my business. My email servers were always perfectly configured with everything required to be considered as legit as possible, but it never fully worked.

3. Proton Mail §

Why did I choose Proton Mail over another email provider? There are a few reasons for it, I evaluated a few providers before deciding.

Proton Mail is a paid service, actually this is an argument in itself, I would not trust a good service to work for free, this would be too good to be true, so it would be a scam (or making money on my data, who knows).

They offer zero-knowledge encryption and MFA, which is exactly what I wanted. Only me should be able to read my email, even if the provider is compromised, adding MFA on top is just perfect because it requires two secrets to access the data. Their zero-knowledge security could be criticized for a few things, ultimately there is no guarantee they do it as advertised.

Long story short, when making your account, Proton Mail generates an encryption key on their server that is password protected with your account password. When you use the service and log-in, the encrypted key is sent to you so all crypto operations happens locally, but there is no way to verify if they kept your private key unencrypted at the beginning, or if they modified their web apps to key log the password typed. Applications are less vulnerable to the second problem as it would impact many users and this would leave evidences. I do trust them for doing the things right, although I have no proof.

I did not choose Proton Mail for end-to-end encryption, I only use GPG occasionally and I could use it before.

IMAP is possible with Proton Mail when you have a paid account, but you need to use a "connect bridge", it is a client that connects to Proton with your credentials and download all encrypted emails locally, then it exposes an IMAP and SMTP server on localhost with dedicated credentials. All emails are saved locally and it syncs continuously, it works great, but it is not lightweight. There is a custom implementation named hydroxide, but it did not work for me. The bridge does not support caldav and cardav, which is not great but not really an issue for me anyway.

GitHub project page: hydroxide

Before migrating, I verified that reversibility was possible, aka being able to migrate my emails away from Proton Mail. In case they stop providing their export tool, I would still have a local copy of all my IMAP emails, which is exactly what I would need to move it somewhere else.

There are certainly better alternatives than Proton with regard to privacy, but Proton is not _that_ bad on this topic, it is acceptable enough for me.

3.1. Benefits §

Since I moved my emails, I do not have deliverability issues. Even people on Microsoft received my emails at first try! Great success for me here.

The anti-spam is more efficient that my spamd trained with years of spam.

Multiple factor authentication is required to access my account.

3.2. Interesting features §

I did not know I would appreciate scheduling emails sending, but it's a thing and I do not need to keep the computer on.

It is possible to generate aliases (10 or unlimited depending on the subscription), what's great with it is that it takes a couple seconds to generate a unique alias, and replying to an email received on an alias automatically uses this alias as the From address (webmail feature). On my server, I have been using a lot of different addresses using a "+" local prefix, it was rarely recognized, so I switched to a dot, but these are not real aliases. So I started managing smtpd aliases through ansible, and it was really painful to add a new alias every time I needed one. Did I mention I like this alias feature? :D

If I want to send an end-to-end encrypted email without GPG, there is an option to use a password to protect the content, the email would actually send a link to the recipient, leading to a Proton Mail interface asking for the password to decrypt the content, and allow that person to reply. I have no idea if I will ever use it, but at least it is a more user-friendly end-to-end encryption method. Tuta is offering the same feature, but it is there only e2e method.

Proton offer logs of login attempts on my account, this was surprising.

There is an onion access to their web services in case you prefer to connect using tor.

The web interface is open source, one should be able to build it locally to connect to Proton servers, I guess it should work?

GitHub project page: ProtonMail webclients

3.3. Shortcomings §

Proton Mail cannot be used as an SMTP relay by my servers, except through the open source bridge hydroxide.

The calendar only works on the website and the smartphone app. The calendar it does not integrate with the phone calendar, although in practice I did not find it to be an issue, everything works fine. Contact support is less good on Android, they are restrained in the Mail app and I still have my cardav server.

The web app is first class citizen, but at least it is good.

Nothing prevents Proton Mail from catching your incoming and outgoing emails, you need to use end-to-end encryption if you REALLY need to protect your emails from that.

I was using two accounts, this would require a "duo" subscription on Proton Mail which is more expensive. I solved this by creating two identities, label and filter rules to separate my two "accounts" (personal and professional) emails. I actually do not really like that, although it is not really an issue at the moment as one of them is relatively low traffic.

The price is certainly high, the "Mail plus" plan is 4€ / month (48€ / year) if you subscribe for 12 months, but is limited to 1 domain, 10 aliases and 15 GB of storage. The "Proton Unlimited" plan is 10€ / month (120€ / year) but comes with the kitchen sink: infinite aliases, 3 domains, 500 GB storage, and access to all Proton services (that you may not need...) like VPN, Drive and Pass. In comparison, hosting your email service on a cheap server should not cost you more than 70€ / year, and you can self-host a nextcloud / seafile (equivalent to Drive, although it is stored encrypted there), a VPN and a vaultwarden instance (equivalent to Pass) in addition to the emails.

Emails are limited to 25MB, which is low given I always configured my own server to allow 100 MB attachments, but it created delivery issues on most recipient servers, so it is not a _real_ issue, but I prefer when I can decide of this kind of limitation.

3.4. Alternatives §

I evaluated Tuta too, but for the following reasons I dropped the idea quickly:

they don't support email import (it's "coming soon" since years on their website)
you can only use their app or website
there is no way to use IMAP
there is no way to use GPG because their client does not support it, and you cannot connect using SMTP with your own client

Their service is cool though, but not for me.

4. My ideal email setup §

If I was to self-host again (which may be soon! Who knows), I would do it differently to improve the security:

one front server with the SMTP server, cheap and disposable
one server for IMAP
one server to receive and analyze the logs

Only the SMTP server would be publicly available, all ports would be closed on all servers, servers would communicate between each other through a VPN, and exports their logs to a server that would only be used for forensics and detecting security breaches.

Such setup would be an improvement if I was self-hosting again my emails, but the cost and time to operate is non-negligible. It is also an ecological nonsense to need 3 servers for a single person emails.

5. Conclusion §

I started this blog post with the fact that the decision was hard, so hard that I was not able to decide up to a day before renewing my email server for one year. I wanted to give Proton a chance for a month to evaluate it completely, and I have to admit I like the service much more than I expected...

My Unix hacker heart hurts terribly on this one. I would like to go back to self-hosting, but I know I cannot reach the level of security I was looking for, simply because email sucks in the first place. A solution would be to get rid of this huge archive burden I am carrying, but I regularly search information into this archive and I have not found any usable "mail archive system" that could digest everything and serve it locally.

5.1. Update 2024-09-14 §

I wrote this blog post two days ago, and I cannot stop thinking about this topic since the migration.

The real problem certainly lies in my use case, not having my emails on the remote server would solve my problems. I need to figure how to handle it. Stay tuned :-)

Self-hosting at home and privacy

Thu, 12 Sep 2024 00:00:00 GMT

1. Introduction
2. Public information
- 2. 1. Domain WHOIS
- 2. 2. TLS certificates using ACME
- 2. 3. Domain name
- 2. 4. Public IP
3. Mitigations
4. Conclusion

1. Introduction §

You may self-host services at home, but you need to think about the potential drawbacks for your privacy.

Let's explore what kind of information could be extracted from self-hosting, especially when you use a domain name.

2. Public information §

2.1. Domain WHOIS §

A domain name must expose some information through WHOIS queries, basically who is the registrar responsible for it, and who could be contacted for technical or administration matters.

Almost every registrar will offer you feature to hide your personal information, you certainly not want to have your full name, full address and phone number exposed on a single WHOIS request.

You can perform a WHOIS request on the link below, directly managed by ICANN.

ICANN Lookup

2.2. TLS certificates using ACME §

If you use TLS certificates for your services, and ACME (Let's Encrypt or alternatives), all the domains for which a certificate was emitted can easily be queried.

You can visit the following website, type a domain name, and you will immediately have a list of existing domain names.

crt.sh Certificate Search

In such situation, if you planned to keep a domain hidden by not sharing it with anyone, you got it wrong.

2.3. Domain name §

If you use a custom domain in your email, it is highly likely that you have some IT knowledge and that you are the only user of your email server.

Using this statement (IT person + only domain user), someone having access to your email address can quickly search for anything related to your domain and figure it is related to you.

2.4. Public IP §

Anywhere you connect, your public IP is known of the remote servers.

Some bored sysadmin could take a look at the IPs in their logs, and check if some public service is running on it, polling for secure services (HTTPS, IMAPS, SMTPS) will immediately give associated domain name on that IP, then they could search even further.

3. Mitigations §

There are not many solutions to prevent this, unfortunately.

The public IP situation could be mitigated by either continuing hosting at home by renting a cheap server with a public IP and establish a VPN between the two and use the public IP of the server for your services, or to move your services to such remote server. This is an extract cost of course. When possible, you could expose the service over Tor hidden service or I2P if it works for your use case, you would not need to rent a server for this.

The TLS certificates names being public could be easily solved by generating self-signed certificates locally, and deal with it. Depending on your services, it may be just fine, but if you have strangers using the services, the fact to accept to trust the certificate on first use (TOFU) may appear dangerous. Some software fail to connect to self-signed certificates and do not offer a bypass...

4. Conclusion §

Self-hosting at home can be practical for various reasons: reusing old hardware, better local throughput, high performance for cheap... but you need to be aware of potential privacy issues that could come with it.

How to use Proton VPN port forwarding

Tue, 03 Sep 2024 00:00:00 GMT

1. Introduction
2. Feature explanation
3. Setup
- 3. 1. OpenBSD
  - 3. 1. 1. Using supervisord
  - 3. 1. 2. Without supervisord
- 3. 2. Linux
4. Conclusion

1. Introduction §

If you use Proton VPN with the paid plan, you have access to their port forwarding feature. It allows you to expose a TCP and/or UDP port of your machine on the public IP of your current VPN connection.

This can be useful for multiple use cases, let's see how to use it on Linux and OpenBSD.

Proton VPN documentation: port forwarding setup

If you do not have a privacy need with regard to the service you need to expose to the Internet, renting a cheap VPS is a better solution: cheaper price, stable public IP, no weird script for port forwarding, use of standard ports allowed, reverse DNS, etc...

2. Feature explanation §

Proton VPN port forwarding feature is not really practical, at least not as practical as doing a port forwarding with your local router. The NAT is done using NAT-PMP protocol (an alternative to UPnP), you will be given a random port number for 60 seconds. The random port number is the same for TCP and UDP.

Wikipedia page about NAT Port Mapping Protocol

There is a NAT PMPC client named natpmpc (available almost everywhere as a package) that need to run in an infinite loop to renew the port lease before it expires.

This is rather not practical for multiple reasons:

you get a random port assigned, so you must configure your daemon every time
the lease renewal script must run continuously
if something wrong happens (script failing, short network failure) that prevent renewing the lease, you will get a new random port

Although it has shortcomings, it is a useful feature that was dropped by other VPN providers because of abuses.

3. Setup §

Let me share a script I am using on Linux and OpenBSD that does the following:

get the port number
reconfigure the daemon using the port forwarding feature
infinite loop renewing the lease

You can run the script from supervisord (a process manager) to restart it upon failure.

Supervisor official project website

In the example, the Java daemon I2P will be used to demonstrate the configuration update using sed after being assigned the port number.

3.1. OpenBSD §

Install the package natpmpd to get the NAT-PMP client.

Create a script with the following content, and make it executable:

#!/bin/sh

PORT=$(natpmpc -a 1 0 udp 60 -g 10.2.0.1 | awk '/Mapped public/ { print $4 }')

# check if the current port is correct
grep "$PORT" /var/i2p/router.config || /etc/rc.d/i2p stop

# update the port in I2P config
sed -i -E "s,(^i2np.udp.port).*,\1=$PORT, ; s,(^i2np.udp.internalPort).*,\1=$PORT," /var/i2p/router.config

# make sure i2p is started (in case it was stopped just before)
/etc/rc.d/i2p start

while true
do
    date # use for debug only
    natpmpc -a 1 0 udp 60 -g 10.2.0.1 && natpmpc -a 1 0 tcp 60 -g 10.2.0.1 || { echo "error Failure natpmpc $(date)"; break ; }
    sleep 45
done

The script will search for the port number in I2P configuration, stop the service if the port is not found. Then the port line is modified with sed (in all cases, it does not matter much). Finally, i2p is started, this will only do something in case i2p was stopped before, otherwise nothing happens.

Then, in an infinite loop with a 45 seconds frequency, there is a renewal of the TCP and UDP port forwarding happening. If something wrong happens, the script exits.

3.1.1. Using supervisord §

If you want to use supervisord to start the script at boot and maintain it running, install the package supervisor and create the file /etc/supervisord.d/nat.ini with the following content:

[program:natvpn]
command=/etc/supervisord.d/continue_nat.sh ; choose the path of your script
autorestart=unexpected ; when to restart if exited after running (def: unexpected)

Enable supervisord at boot, start it and verify it started (a configuration error prevents it from starting):

rcctl enable supervisord
rcctl start supervisord
rcctl check supervisord

3.1.2. Without supervisord §

Open a shell as root and execute the script and keep the terminal opened, or run it in a tmux session.

3.2. Linux §

The setup is exactly the same as for OpenBSD, just make sure the package providing natpmpc is installed.

Depending on your distribution, if you want to automate the script running / restart, you can run it from a systemd service with auto restart on failure, or use supervisord as explained above.

If you use a different network namespace, just make sure to prefix the commands using the VPN with ip netns exec vpn.

Here is the same example as above but using a network namespace named "vpn" to start i2p service and do the NAT query.

#!/bin/sh

PORT=$(ip netns exec vpn natpmpc -a 1 0 udp 60 -g 10.2.0.1 | awk '/Mapped public/ { print $4 }')

FILE=/var/i2p/.i2p/router.config

grep "$PORT" $FILE || sudo -u i2p /var/i2p/i2prouter stop
sed -i -E "s,(^i2np.udp.port).*,\1=$PORT, ; s,(^i2np.udp.internalPort).*,\1=$PORT," $FILE

ip netns exec vpn sudo -u i2p /var/i2p/i2prouter start

while true
do
    date
    ip netns exec vpn natpmpc -a 1 0 udp 60 -g 10.2.0.1 && ip netns exec vpn natpmpc -a 1 0 tcp 60 -g 10.2.0.1 || { echo "error Failure natpmpc $(date)"; break ; }
    sleep 45
done

4. Conclusion §

Proton VPN port forwarding feature is useful when need to expose a local network service on a public IP. Automating it is required to make it work efficiently due to the unusual implementation.

Emails encryption at rest on OpenBSD using dovecot and GPG

Mon, 19 Aug 2024 00:00:00 GMT

1. Introduction
2. Threat model
3. Setup
- 3. 1. GPGit
- 3. 2. Sieve
- 3. 3. Dovecot
- 3. 4. User GPG setup
- 3. 5. Anti-spam service
4. Conclusion

1. Introduction §

In this blog post, you will learn how to configure your email server to encrypt all incoming emails using user's GPG public keys (when it exists). This will prevent anyone from reading the emails, except if you own the according GPG private key. This is known as "encryption at rest".

This setup, while effective, has limitations. Headers will not be encrypted, search in emails will break as the content is encrypted, and you obviously need to have the GPG private key available when you want to read your emails (if you read emails on your smartphone, you need to decide if you really want your GPG private key there).

Encryption is CPU consuming (and memory too for emails of a considerable size), I tried it on an openbsd.amsterdam virtual machine, and it was working fine until someone sent me emails with 20MB attachments. On a bare-metal server, there is absolutely no issue. Maybe GPG makes use of hardware acceleration cryptography, and it is not available in virtual machines hosted under the OpenBSD hypervisor vmm.

This is not an original idea, Etienne Perot wrote about a similar setup in 2012 and enhanced the gpgit script we will use in the setup. While his blog post is obsolete by now because of all the changes that happened in Dovecot, the core idea remains the same. Thank you very much Etienne for your job!

Etienne Perot: Encrypt specific incoming emails using Dovecot and Sieve

gpgit GitHub project page

gpgit mirror on tildegit.org

This guide is an extension of my recent email server setup guide:

2024-07-24 Full-featured email server running OpenBSD

2. Threat model §

This setup is useful to protect your emails stored on the IMAP server. If the server or your IMAP account are compromised, the content of your emails will be encrypted and unusable.

You must be aware that emails headers are not encrypted: recipients / senders / date / subject will remain in clear text even after encryption. If you already use end-to-end encryption with your recipients, there are no benefits using this setup.

An alternative is to not let any emails on the IMAP server, although they could be recovered as they are written in the disk until you retrieve them.

Personally, I keep many emails of my server, and I am afraid that a 0day vulnerability could be exploited on my email server, allowing an attacker to retrieve the content of all my emails. OpenSMTPD had critical vulnerabilities a few years ago, including a remote code execution, so it is a realistic threat.

I wrote a privacy guide (for a client) explaining all the information shared through emails, with possible mitigations and their limitations.

IVPN: The Technical Realities of Email Privacy

3. Setup §

This setup makes use of the program gpgit which is a Perl script encrypt emails received over the standard input using GPG, it is a complicated task because the email structure can be very complicated. I have not been able to find any alternative to this script. In gpgit repository there is a script to encrypt an existing mailbox (maildir format), that script must be run on the server, I did not test it yet.

You will configure a specific sieve rule which is "global" (not user-defined) that will process all emails before any other sieve filter. This sieve script will trigger a filter (a program allowed to modify the email) and pass the email on the standard input of the shell script encrypt.sh, which in turn will run gpgit with the according username after verifying a gnupg directory existed for them. If there is no gnupg directory, the email is not encrypted, this allows multiple users on the email server without enforcing encryption for everyone.

If a user has multiple addresses, this is the system account name that is used in the local part of the GPG key address.

3.1. GPGit §

Some packages are required for gpgit to work, they are all available on OpenBSD:

pkg_add p5-Mail-GnuPG p5-List-MoreUtils

Download gpgit git repository and copy its gpgpit script into /usr/local/bin/ as an executable:

cd /tmp/
git clone https://github.com/EtiennePerot/gpgit
cd gpgit
install -o root -g wheel -m 555 gpgit /usr/local/bin/

3.2. Sieve §

All the following paths will be relative to the directory /usr/local/lib/dovecot/sieve/, you can cd into it now.

Create the file encrypt.sh with this content, replace the variable DOMAIN with the domain configured in the GPG key:

#!/bin/sh

DOMAIN="puffy.cafe"

NOW=$(date +%s)
DATA="$(cat)"

if test -d ~/.gnupg
then
    echo "$DATA" | /usr/local/bin/gpgit "${USER}@${DOMAIN}"
    NOW2=$(date +%s)
    echo "Email encryption for user ${USER}: $(( NOW2 - NOW )) seconds" | logger -p mail.info
else
    echo "$DATA"
    echo "Email encryption for user for ${USER} none" | logger -p mail.info
fi

Make the script executable with chmod +x encrypt.sh. This script will create a new log line in your email logs every time an email is processed, including the username and the time required for encryption (in case of encryption). You could extend the script to discard the Subject header from the email if you want to hide it, I do not provide the implementation as I expect this task to be trickier than it looks like if you want to handle all corner cases.

Create the file global.sieve with the content:

require ["vnd.dovecot.filter"];
filter "encrypt.sh";

Compile the sieve rules with sievec global.sieve.

3.3. Dovecot §

Edit the file /etc/dovecot/conf.d/90-plugin.conf to add the following code within the plugin block:

  sieve_filter_bin_dir = /usr/local/lib/dovecot/sieve
  sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment +vnd.dovecot.filter
  sieve_before = /usr/local/lib/dovecot/sieve/global.sieve
  sieve_filter_exec_timeout = 200s

You may have sieve_global_extensions already set, in that case update its value.

The variable sieve_filter_exec_timeout allows the script encrypt.sh to run for 200 seconds before being stopped, you should adapt the value to your system. I came up with 200 seconds to be able to encrypt email with 20MB attachments on an openbsd.amsterdam virtual machine. On a bare metal server with a Ryzen 5 CPU, it takes less than one second for the same email.

The full file should look like the following (in case you followed my previous email guide):

##
## Plugin settings
##

# All wanted plugins must be listed in mail_plugins setting before any of the
# settings take effect. See  for list of plugins and
# their configuration. Note that %variable expansion is done for all values.

plugin {
  sieve_plugins = sieve_imapsieve sieve_extprograms

  # From elsewhere to Spam folder
  imapsieve_mailbox1_name = Spam
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_before = file:/usr/local/lib/dovecot/sieve/report-spam.siev

  # From Spam folder to elsewhere
  imapsieve_mailbox2_name = *
  imapsieve_mailbox2_from = Spam
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve

  sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve

  # for GPG encryption
  sieve_filter_bin_dir = /usr/local/lib/dovecot/sieve
  sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment +vnd.dovecot.filter
  sieve_before = /usr/local/lib/dovecot/sieve/global.sieve
  sieve_filter_exec_timeout = 200s
}

Open the file /etc/dovecot/conf.d/10-master.conf and uncomment the variable default_vsz_limit and set its value to 1024M. This is required as GPG uses a lot of memory and without this, the process will be killed and the email lost. I found 1024M to works with attachments up to 45 MB, however you should raise this value higher value if you plan to receive bigger attachments.

Restart dovecot to take account of the changes: rcctl restart dovecot.

3.4. User GPG setup §

You need to create a GPG keyring for each users you want use encryption, the simplest method is to setup a passwordless keyring and import your public key:

$ gpg --quick-generate-key --passphrase '' --batch "$USER"
$ gpg --import public-key-file.asc
$ gpg --edit-key FINGERPRINT_HERE
gpg> sign
[....]
gpg> save

If you want to disable GPG encryption for the user, remove the directory ~/.gnupg.

3.5. Anti-spam service §

If you use a spam filter such as rspamd or spamassassin relying on bayes filter, it will only work if it process the emails before arriving at dovecot, for instance in my email setup this is the case as rspamd is a filter of opensmtpd and pass the email before being delivered to Dovecot.

Such service can have privacy issues, especially if you use encryption. Bayes filter works by splitting an email content into tokens (not really words but almost) and looking for patterns using these tokens, basically each emails is split and stored in the anti-spam local database in small parts. I am not sure one could recreate the emails based on tokens, but if someone like an attacker is able to access the token list, they may have some insights about your email content. If this is part of your threat model, disable your anti-spam Bayes filter.

4. Conclusion §

This setup is quite helpful if you want to protect all your emails on their storage. Full disk encryption on the server does not prevent anyone able to connect over SSH (as root or the email user) from reading the emails, even file recovery is possible when the volume is unlocked (not on the real disk, but the software encrypted volume), this is where encryption at rest is beneficial.

I know from experience it is complicated to use end-to-end encryption with tech-savvy users, and that it is even unthinkable with regular users. This is a first step if you need this kind of security (see the threat model section), but you need to remember a copy of all your emails certainly exist on the servers used by the persons you exchange emails with.

Using Firefox remote debugging feature

Thu, 08 Aug 2024 00:00:00 GMT

1. Introduction
2. Setup
3. Remote connection
4. Conclusion

1. Introduction §

Firefox has an interesting features for developers, its ability to connect a Firefox developers tools to a remote Firefox instance. This can really interesting in the case of a remote kiosk display for instance.

The remote debugging does not provide a display of the remote, but it gives you access to the developer tools for tabs opened on the remote.

2. Setup §

The remote firefox you want to connect to must be started using the command line parameter --start-debugger-server. This will make it listen on the TCP port 6000 on 127.0.0.1. Be careful, there is another option named remote-debugging-port which is not what you want here, but the names can be confusing (trust me, I wasted too much time because of this).

Before starting Firefox, a few knobs must be modified in its configuration. Either search for the options in about:config or create a user.js file in the Firefox profile directory with the following content:

user_pref("devtools.chrome.enabled", true);
user_pref("devtools.debugger.remote-enabled", true);
user_pref("devtools.debugger.prompt-connection", false);

This enables the remote management and removes a prompt upon each connection, while this is a good safety measure, it is not practical for remote debugging.

When you start Firefox, the URL input bar should have a red background.

3. Remote connection §

Now, you need to make a SSH tunnel to that remote host where Firefox is running in order to connect to the port. Depending on your use case, a local NAT could be done to expose the port to a network interface or VPN interface, but pay attention to security as this would allow anyone on the network to control the Firefox instance.

The SSH tunnel is quite standard: ssh -L 6001:127.0.0.1:6000, the remote port 6000 is exposed locally as 6001, this is important because your own Firefox may be using the port 6000 for some reasons.

In your own local Firefox instance, visit the page about:debugging, add the remote instance localhost:6001 and then click on Connect on its name on the left panel. Congratulations, you have access to the remote instance for debugging or profiling websites.

Input the remote address localhost:6001 and click on Add

Click on connect on the left

Enjoy your remote debugging session

4. Conclusion §

While it can be tricky to debug a system you can directly see, especially if it is a kiosk in production that you can see / use in case of a problem.

Full-featured email server running OpenBSD

Thu, 25 Jul 2024 00:00:00 GMT

1. Introduction
2. Quick reminder
3. Packet Filter (PF)
4. DNS
- 4. 1. MX records
- 4. 2. SPF
- 4. 3. DKIM
- 4. 4. DMARC
- 4. 5. PTR (Reverse DNS)
5. System configuration
- 5. 1. Acme-client
- 5. 2. Rspamd
  - 5. 2. 1. Alternatives
- 5. 3. OpenSMTPD
  - 5. 3. 1. TLS
  - 5. 3. 2. User management
  - 5. 3. 3. Handling extra domains
  - 5. 3. 4. Without Dovecot
- 5. 4. Dovecot
  - 5. 4. 1. IMAP
  - 5. 4. 2. POP
  - 5. 4. 3. JMAP
  - 5. 4. 4. Sieve (filtering rules)
  - 5. 4. 5. Manage Sieve
  - 5. 4. 6. Start the service
6. Webmail
- 6. 1. Roundcube mail setup
7. Hardening
- 7. 1. Always allow the sender per email or domain
- 7. 2. Block bots
- 7. 3. Split the stack
- 7. 4. Network attack surface reduction
8. Email client configuration
9. Verify the setup
10. Maintenance
- 10. 1. Running processes
- 10. 2. Certificates renewal
- 10. 3. All about logs
- 10. 4. Disk space
11. Conclusion

1. Introduction §

This blog post is a guide explaining how to setup a full-featured email server on OpenBSD 7.5. It was commissioned by a customer of my consultancy who wanted it to be published on my blog.

Setting up a modern email stack that does not appear as a spam platform to the world can be a daunting task, the guide will cover what you need for a secure, functional and low maintenance email system.

The features list can be found below:

email access through IMAP, POP or Webmail
secure SMTP server (mandatory server to server encryption, personal information hiding)
state-of-the-art setup to be considered as legitimate as possible
firewall filtering (bot blocking, all ports closes but the required ones)
anti-spam

In the example, I will set up a temporary server for the domain puffy.cafe with a server using the subdomain mail.puffy.cafe. From there, you can adapt with your own domain.

2. Quick reminder §

I prepared a few diagrams explaining how all the components are used together, in three cases: when sending an email, when the SMTP servers receives an email from the outside and when you retrieve your emails locally.

Authenticated user sending an email to the outside

Outside sending an email to one of our users

User retrieving emails for reading

3. Packet Filter (PF) §

Packet Filter is OpenBSD's firewall. In our setup, we want all ports to be blocked except the few ones required for the email stack.

The following ports will be required:

opensmtpd 25/tcp (smtp): used for email delivery from other servers, supports STARTTLS
opensmtpd 465/tcp (smtps): used to establish a TLS connection to the SMTP server to receive or send emails
opensmtpd 587/tcp (submission): used to send emails to external servers, supports STARTTLS
httpd 80/tcp (http): used to generate TLS certificates using ACME
dovecot 993/tcp (imaps): used to connect to the IMAPS server to read emails
dovecot 995/tcp (pop3s): used to connect to the POP3S server to download emails
dovecot 4190/tcp (sieve): used to allow remote management of an user SIEVE rules

Depending on what services you will use, only the opensmtpd ports are mandatory. In addition, we will open the port 22/tcp for SSH.

set block-policy drop
set loginterface egress
set skip on lo0

# normalisation des paquets
match in all scrub (no-df random-id max-mss 1440)
antispoof quick for { egress }

tcp_ports = "{ smtps smtp submission imaps pop3s sieve ssh http }"

block all
pass out inet
pass out inet6

# allow ICMP (ping)
pass in proto icmp

# allow IPv6 to work
pass in on egress inet6 proto icmp6 all icmp6-type { routeradv neighbrsol neighbradv }
pass in on egress inet6 proto udp from fe80::/10 port dhcpv6-server to fe80::/10 port dhcpv6-client no state

# allow our services
pass in on egress proto tcp from any to any port $tcp_ports

# default OpenBSD rules
# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010

# Port build user does not need network
block return out log proto {tcp udp} user _pbuild

4. DNS §

If you want to run your own email server, you need a domain name configured with a couple of DNS records about the email server.

4.1. MX records §

Wikipedia page: MX record

The MX records list the servers that should be used by outside SMTP servers to send us emails, this is the public list of our servers accepting emails for a given domain. They have a weight associated to each of them, the server with the lowest weight should be used first and if it does not respond, the next server used will be the one with a slightly higher weight. This is a simple mechanism that allow setting up a hierarchy.

I highly recommend setting up at least two servers, so if your main server fails is unreachable (host outage, hardware failure, upgrade ongoing) the emails will be sent to the backup server. Dovecot bundles a program to synchronize mailboxes between servers, one way or two-way, one shot or continuously.

If you have no MX records in your domain name, it is not possible to send you emails. It is like asking someone to send you a post card without giving them any clue about your real address.

Your server hostname can be different from the domain apex (raw domain name without a subdomain), a simple example would be to use mail.domain.example for the server name, this will not prevent it from receiving/sending emails using @domain.example in email addresses.

In my example, the domain puffy.cafe mail server will be mail.puffy.cafe, giving this MX record in my DNS zone:

        IN MX     10 mail.puffy.cafe.

4.2. SPF §

Wikipedia page: SPF record

The SPF record is certainly the most important piece of the email puzzle to detect spam. With the SPF, the domain name owner can define which servers are allowed to send emails from that domain. A properly configured spam filter will give a high spam score to incoming emails that are not in the sender domain SPF.

To ease the configuration, that record can automatically include all MX defined for a domain, but also A/AAAA records, so if you only use your MX servers for sending, a simple configuration allowing MX servers to send is enough.

In my example, only mail.puffy.cafe should be legitimate for sending emails, any future MX server should also be allowed to send emails, so we configure the SPF to allow all MX defined servers to be senders.

    600 IN TXT     "v=spf1 mx -all"

4.3. DKIM §

Wikipedia page: DKIM signature

When used, the DKIM is a system allowing a receiver to authenticate a sender, based on an asymmetric cryptographic keys. The sender publishes its public key on a TXT DNS record before signing all outgoing emails using the private key. By doing so, receivers can validate the email integrity and make sure it was sent from a server of the domain claimed in the From header.

DKIM is mandatory to not be classified as a spamming server.

The following set of commands will create a 2048 bits RSA key in /etc/mail/dkim/private/puffy.cafe.key with its public key in /etc/mail/dkim/puffy.cafe.pub, the umask 077 command will make sure any file created during the process will only be readable by root. Finally, you need to make the private key readable to the group _rspamd.

Note: the umask command will persist in your shell session, if you do not want to create files/directory only readable by root after this, either spawn a new shell, or run the set of commands in a new shell and then exit from it once you are done.

umask 077
install -d -o root -g wheel -m 755 /etc/mail/dkim
install -d -o root -g _dkim -m 775 /etc/mail/dkim/private
openssl genrsa -out /etc/mail/dkim/private/puffy.cafe.key 2048
openssl rsa -in /etc/mail/dkim/private/puffy.cafe.key -pubout -out /etc/mail/dkim/puffy.cafe.pub
chgrp _rspamd /etc/mail/dkim/private/puffy.cafe.key /etc/mail/dkim/private/
chmod 440 /etc/mail/dkim/private/puffy.cafe.key
chmod 775 /etc/mail/dkim/private/

In this example, we will name the DKIM selector dkim to keep it simple. The selector is the name of the key, this allows having multiple DKIM keys for a single domain.

Add the DNS record like the following, the value in p is the public key in the file /etc/mail/dkim/puffy.cafe.pub, you can get it as a single line with the command awk '/PUBLIC/ { $0="" } { printf ("%s",$0) } END { print }' /etc/mail/dkim/puffy.cafe.pub:

Your registrar may offer to add the entry using a DKIM specific form. There is nothing wrong doing so, just make sure the produced entry looks like the entry below.

dkim._domainkey IN TXT "v=DKIM1;k=rsa;p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAo3tIFelMk74wm+cJe20qAUVejD0/X+IdU+A2GhAnLDpgiA5zMGiPfYfmawlLy07tJdLfMLObl8aZDt5Ij4ojGN5SE1SsbGC2MTQGq9L2sLw2DXq+D8YKfFAe0KdYGczd9IAQ9mkYooRfhF8yMc2sMoM75bLxGjRM1Fs1OZLmyPYzy83UhFYq4gqzwaXuTvxvOKKyOwpWzrXzP6oVM7vTFCdbr8E0nWPXWKPJhcd10CF33ydtVVwDFp9nDdgek3yY+UYRuo/iJvdcn2adFoDxlE6eXmhGnyG4+nWLNZrxIgokhom5t5E84O2N31YJLmqdTF+nH5hTON7//5Kf/l/ubwIDAQAB"

4.4. DMARC §

Wikipedia page: DMARC record

The DMARC record is an extra mechanism that comes on top of SPF/DKIM, while it does not do much by itself, it is important to configure it.

DMARC could be seen as a public notice explaining to servers receiving emails whose sender looks like your domain name (legit or not) what they should do if SPF/DKIM does not validate.

As of 2024, DMARC offers three actions for receivers:

do nothing but make a report to the domain owner
"quarantine" mode: tell the receiver to be suspicious without rejecting it, the result will depend on the receiver (most of the time it will be flagged as spam) and make a report
"reject" mode: tell the receiver to not accept the email and make a report

In my example, I want invalid SPF/DKIM emails to be rejected. It is quite arbitrary, but I prefer all invalid emails from my domain to be discarded rather than ending up in a spam directory, so p and sp are set to reject. In addition, if my own server is misconfigured I will be notified about delivery issues sooner than if emails were silently put into quarantine.

An email address should be provided to receive DMARC reports, they are barely readable and I never made use of them, but the email address should exist so this is what the rua field is for.

The field aspf is set to r (relax), basically this allows any servers with a hostname being a subdomain of .puffy.cafe to send emails for @puffy.cafe, while if this field is set to s (strict), the domain of the sender should match the domain of the email server (mail.puffy.cafe would only be allowed to send for @mail.puffy.cafe).

Mx Toolbox website: DMARC tags list

_dmarc        IN TXT     "v=DMARC1;p=reject;rua=mailto:dmarc@puffy.cafe;sp=reject;aspf=r;"

4.5. PTR (Reverse DNS) §

Wikipedia page: PTR record

An older mechanism used to prevent spam was to block, or consider as spam, any SMTP server whose advertised hostname did not match the result of the reverse lookup of its IP.

Let's say "mail.foobar.example" (IP: A.B.C.D) is sending an email to my server, if the result of the DNS request to resolve the PTR of A.B.C.D is not "mail.foobar.example", the email would be considered as spam or rejected. While this is superseded by SPF/DKIM and annoying as it is not always possible to define a PTR for a public IP, the reverse DNS setup is still a strong requirement to not be considered as a spamming platform.

Make sure the PTR matches the system hostname and not the domain name itself, in the example above the PTR should be mail.foobar.example and not foobar.example.

5. System configuration §

5.1. Acme-client §

The first step is to obtain a valid TLS certificate, this requires configuring acme-client, httpd and start httpd daemon.

Copy the acme-client example cp /etc/examples/acme-client.conf /etc/

Modify /etc/acme-client.conf and edit only the last entry to configure your own domain, mine looks like this:

#
# $OpenBSD: acme-client.conf,v 1.5 2023/05/10 07:34:57 tb Exp $
#
authority letsencrypt {
	api url "https://acme-v02.api.letsencrypt.org/directory"
	account key "/etc/acme/letsencrypt-privkey.pem"
}

authority letsencrypt-staging {
	api url "https://acme-staging-v02.api.letsencrypt.org/directory"
	account key "/etc/acme/letsencrypt-staging-privkey.pem"
}

authority buypass {
	api url "https://api.buypass.com/acme/directory"
	account key "/etc/acme/buypass-privkey.pem"
	contact "mailto:me@example.com"
}

authority buypass-test {
	api url "https://api.test4.buypass.no/acme/directory"
	account key "/etc/acme/buypass-test-privkey.pem"
	contact "mailto:me@example.com"
}

domain mail.puffy.cafe {
    # you can remove the line "alternative names" if you do not need extra subdomains
    # associated to this certificate
    # imap.puffy.cafe is purely an example, I do not need it
	alternative names { imap.puffy.cafe pop.puffy.cafe }
	domain key "/etc/ssl/private/mail.puffy.cafe.key"
	domain full chain certificate "/etc/ssl/mail.puffy.cafe.fullchain.pem"
	sign with letsencrypt
}

Now, configure httpd, starting from the OpenBSD example: cp /etc/examples/httpd.conf /etc/

Edit /etc/httpd.conf, we want the first block to match all domains but not "example.com", and we do not need the second block listen on 443/tcp (except if you want to run a https server with some content, but you are on your own then). The resulting file should look like the following:

# $OpenBSD: httpd.conf,v 1.22 2020/11/04 10:34:18 denis Exp $

server "*" {
	listen on * port 80
	location "/.well-known/acme-challenge/*" {
		root "/acme"
		request strip 2
	}
	location * {
		block return 302 "https://$HTTP_HOST$REQUEST_URI"
	}
}

Enable and start httpd with rcctl enable httpd && rcctl start httpd.

Run acme-client -v mail.puffy.cafe to generate the certificate with some verbose output (if something goes wrong, you will have a clue).

If everything went fine, you should have the full chain certificate in /etc/ssl/mail.puffy.cafe.fullchain.pem and the private key in /etc/ssl/private/mail.puffy.cafe.key.

5.2. Rspamd §

You will use rspamd to filter spam and sign outgoing emails for DKIM.

Install rspamd and the filter to plug it to opensmtpd:

pkg_add rspamd-- opensmtpd-filter-rspamd

You need to configure rspamd to sign outgoing emails with your DKIM private key, to proceed, create the file /etc/rspamd/local.d/dkim_signing.conf (the filename is important):

# our usernames does not contain the domain part
# so we need to enable this option
allow_username_mismatch = true;

# this configures the domain puffy.cafe to use the selector "dkim"
# and where to find the private key
domain {
    puffy.cafe {
        path = "/etc/mail/dkim/private/puffy.cafe.key";
        selector = "dkim";
    }
}

For better performance, you need to use redis as a cache backend for rspamd:

rcctl enable redis
rcctl start redis

Now you can start rspamd:

rcctl enable rspamd
rcctl start rspamd

For extra information about rspamd (like statistics or its web UI), I wrote about it in 2021:

Older blog post: 2024-07-13 Filtering spam using Rspamd and OpenSMTPD on OpenBSD

5.2.1. Alternatives §

If you do not want to use rspamd, it is possible to replace the DKIM signing part using opendkim, dkimproxy or opensmtpd-filter-dkimsign. The spam filter could be either replaced by the featureful spamassassin available as a package, or partially with the base system program spamd (it does not analyze emails).

This guide only focus on rspamd, but it is important to know alternatives exist.

5.3. OpenSMTPD §

OpenSMTPD configuration file on OpenBSD is /etc/mail/smtpd.conf, here is a working configuration with a lot of comments:

## this defines the paths for the X509 certificate
pki puffy.cafe cert "/etc/ssl/mail.puffy.cafe.fullchain.pem"
pki puffy.cafe key "/etc/ssl/private/mail.puffy.cafe.key"
pki puffy.cafe dhe auto

## this defines how the local part of email addresses can be split
# defaults to '+', so solene+foobar@domain matches user
# solene@domain. Due to the '+' character being a regular source of issues
# with many online forms, I recommend using a character such as '_',
# '.' or '-'. This feature is very handy to generate infinite unique emails
# addresses without pre-defining aliases.
# Using '_', solene_openbsd@domain and solene_buystuff@domain lead to the
# same address
smtp sub-addr-delim '_'

## this defines an external filter
# rspamd does dkim signing and spam filter
filter rspamd proc-exec "filter-rspamd"

## this defines which file will contain aliases
# this can be used to define groups or redirect emails to users
table aliases file:/etc/mail/aliases

## this defines all the ports to use
# mask-src hides system hostname, username and public IP when sending an email
listen on all port 25  tls         pki "puffy.cafe" filter "rspamd"
listen on all port 465 smtps       pki "puffy.cafe" auth mask-src filter "rspamd"
listen on all port 587 tls-require pki "puffy.cafe" auth mask-src filter "rspamd"

## this defines actions
# either deliver to lmtp or to an external server
action "local" lmtp "/var/dovecot/lmtp" alias 
action "outbound" relay

## this defines what should be done depending on some conditions
# receive emails (local or from external server for "puffy.cafe")
match from any for domain "puffy.cafe" action "local"
match from local for local action "local"

# send email (from local or authenticated user)
match from any auth for any action "outbound"
match from local for any action "outbound"

In addition, you can configure the advertised hostname by editing the file /etc/mail/mailname: for instance my machine's hostname is ryzen so I need this file to advertise it as mail.puffy.cafe.

Restart OpenSMTPD with rcctl restart smtpd.

5.3.1. TLS §

For ports using STARTTLS (25 and 587), there are different options with regard to TLS encryption.

do not allow STARTTLS
offer STARTTLS but allow not using it (option tls)
require STARTTLS: drop connection when the remote peer does ask for STARTTLS (option tls-require)
require STARTTLS: drop connection when no STARTTLS, and verify the remote certificate (option tls-require verify)

It is recommended to enforce STARTTLS on port 587 as it is used by authenticated users to send emails, preventing them to send emails without network encryption.

On port 25, used by external servers to reach yours, it is important to allow STARTTLS because most server will deliver emails over an encrypted TLS session, however it is your choice to enforce it or not.

Enforcing STARTTLS might break email delivery from some external servers that are outdated or misconfigured (or bad actors).

5.3.2. User management §

By default, OpenSMTPD is configured to deliver email to valid users in the system. In my example, if user solene exists, then email address solene@puffy.cafe will deliver emails to solene user mailbox.

Of course, as you do not want the system daemons to receive emails, a file contains aliases to redirect emails from a user to another, or simply discard it.

In /etc/mail/aliases, you can redirect emails to your username by adding a new line, in the example below I will redirect root emails to my user.

root: solene

It is possible to redirect to multiple users using a comma to separate them, this is handful if you want to create a local group delivering emails to multiple users.

Instead of a user, it is possible to append the incoming emails to a file, pipe them to a command or return an SMTP code. The aliases(5) man pages contains all you need to know.

OpenBSD manual pages: aliases(5)

Every time you modify this file, you need to run the command smtpctl update table aliases to reload the aliases table in OpenSMTPD memory.

You can add a new email account by creating a new user with a shell preventing login:

useradd -m -s /sbin/nologin username_here
passwd username_here

This user will not be able to do anything on the server but connecting to SMTP/IMAP/POP. They will not be able to change their password either!

5.3.3. Handling extra domains §

If you need to handle emails for multiple domains, this is rather simple:

Add this line to the file /etc/mail/smtpd.conf by changing puffy.cafe to the other domain name: match from any for domain "puffy.cafe" action "local"
Configure the other domain DNS MX/SPF/DKIM/DMARC
Configure /etc/rspamd/local.d/dkim_signing.conf to add a new block with the other domain, the dkim selector and the dkim key path
The PTR does not need to be modified as it should match the machine hostname advertised over SMTP, and it is an unique value anyway

If you want to use a different aliases table for the other domain, you need to create a new aliases file and configure /etc/mail/smtpd.conf accordingly where the following lines should be added:

table lambda file:/etc/mail/aliases-lambda

action "local_mail_lambda" lmtp "/var/dovecot/lmtp" alias 

match from any for domain "lambda-puffy.eu" action "local_mail_lambda"

Note that the users will be the same for all the domains configured on the server. If you want to have separate users per domains, or that "user a" on domain A and "user a" on domain B could be different persons / logins, you would need to setup virtual users instead of using system users. Such setup is beyond the scope of this guide.

5.3.4. Without Dovecot §

It is possible to not use Dovecot. Such setup can suit users who would like to download the maildir directory using rsync on their local computer, this is a one-way process and does not allow sharing a mailbox across multiple devices. This reduces maintenance and attack surface at the cost of convenience.

This may work as a two-way access (untested) when using a software such as unison to keep both the local and remote directories synchronized, but be prepared to manage file conflicts!

If you want this setup, replace the following line in smtpd.conf

action "local" lmtp "/var/dovecot/lmtp" alias

by this line: if you want to store the emails into a maildir format (a directory per email folder, a file per email), emails will be stored in the directory "Maildir" in user's homes.

action "local" maildir "~/Maildir/" junk alias

or this line if you want to keep the mbox format (a single file with emails appended to it, not practical), the emails will be stored in /var/mail/$user.

action "local" mbox alias

Wikipedia page: Maildir format

Wikipedia page: Mbox format

5.4. Dovecot §

Dovecot is an important piece of software for the domain end users, it provides protocols like IMAP or POP3 to read emails from a client. It is the most popular open source IMAP/POP server available (the other being Cyrus IMAP).

Install dovecot with the following command line:

pkg_add dovecot-- dovecot-pigeonhole--

Dovecot has a lot of configuration files in /etc/dovecot/conf.d/ although most of them are commented and ready to be modified, you will have to edit a few of them. This guide provides the content of files with empty lines / comments stripped so you can quickly check if your file is ok, you can use the command awk '$1 !~ /^#/ && $1 ~ /./' on a file to display its "useful" content only (awk will not modify the file).

Modify /etc/dovecot/conf.d/10-ssl.conf and search the lines ssl_cert and ssl_key, change their values to your certificate full chain and private key.

Generate a Diffie-Hellman file for perfect forward secrecy, this will make each TLS negociation unique, so if the private key ever leak, every past TLS communication will remain safe.

openssl dhparam -out /etc/dovecot/dh.pem 4096
chown _dovecot:_dovecot /etc/dovecot/dh.pem
chmod 400 /etc/dovecot/dh.pem

The file (filtered of all comments/empty lines) should look like the following:

ssl_cert =


Modify /etc/dovecot/conf.d/10-mail.conf, search for a commented line mail_location, uncomment it and set the value to maildir:~/Maildir, this will tell Dovecot where users mailboxes are stored and in which format, we want to use the maildir format.

The resulting file should look like:

mail_location = maildir:~/Maildir
namespace inbox {
  inbox = yes
}
mmap_disable = yes
first_valid_uid = 1000
mail_plugin_dir = /usr/local/lib/dovecot
protocol !indexer-worker {
}
mbox_write_locks = fcntl

Modify the file /etc/dovecot/conf.d/20-lmtp.conf, LMTP is the protocol used by opensmtpd to transmit incoming emails to dovecot.  Search for the commented variable mail_plugins and uncomment it with the value mail_plugins = $mail_plugins sieve:

The resulting file should look like:

protocol lmtp {
  mail_plugins = $mail_plugins sieve
}

If you do not want to use IMAP or POP3, you do not need Dovecot.  There is an explanation above how to proceed without Dovecot.

5.4.1. IMAP §
Wikipedia page: IMAP protocol
IMAP is an efficient protocol that returns headers of emails per directory, so you do not have to download all your emails to view the directory list, emails are downloaded upon read (by default in most email clients).  It allows some cool features like server side search, incoming email sorting with sieve filters or multi devices access.

Edit /etc/dovecot/conf.d/20-imap.conf and configure the last lines accordingly to the result file:

protocol imap {
  mail_plugins = $mail_plugins imap_sieve
  mail_max_userip_connections = 25
}

The number of connections per user/IP should be high if you have an email client tracking many folders, in IMAP a connection is required for each folder, so the number of connections can quickly increase.  On top of that, if you have multiple devices under the same public IP you could quickly reach the limit.  I found 25 worked fine for me with 3 devices.

5.4.2. POP §
Wikipedia page: POP protocol
POP3 is a pretty old protocol that is rarely considered by users, I still consider it a viable alternative to IMAP depending on your needs.

A major incentive for using POP is that it downloads all emails locally before removing them from the server.  As we have no tooling to encrypt emails stored on remote email servers, POP3 is a must if you want to not leave any email on the server.  POP3 does not support remote folders, so you can not use Sieve filters on the server to sort your emails and then download them as-this.  A POP3 client downloads the Inbox and then sorts the emails locally.

It can support multiple devices under some conditions: if you delete the emails after X days, your devices should synchronize before the emails are removed.  In such case they will have all the emails stored locally, but they will not be synced together: if both computers A and B are up-to-date, when deleting an email on A, it will still be in B.

There are no changes required for POP3 in Dovecot as the defaults are good enough.

5.4.3. JMAP §
For information, a replacement for IMAP called JMAP is in development, it is meant to be better than IMAP in every way and also include calendars and address book management.

JMAP Implementations are young but exist, although support in email clients is almost non-existent.  For instance, it seems Mozilla Thunderbird is not interested in it, an issue in their bug tracker about JMAP from December 2016 only have a couple of comments from people who would like to see it happening, nothing more.

Issue 1322991: Add support for new JMAP protocol
From the JMAP website page listing compatible clients, I only recognized the name "aerc" which is a modern console email client.

JMAP project website: clients list
5.4.4. Sieve (filtering rules) §
Wikipedia page: Sieve
Dovecot has a plugin to offer Sieve filters, they are rules applied to received emails going into your mailbox, whether you want to sort them into dedicated directories, mark them read or block some addresses.  That plugin is called pigeonhole.

You will need Sieve to enable the spam filter learning system when moving emails from/to the Junk folder as it is triggered by a Sieve rule.  This improves rspamd Bayes (a method using tokens to understand information, the story of the person behind it is interesting) filter ability to detect spam accurately.

Edit /etc/dovecot/conf.d/90-plugin.conf with the following content:

plugin {
  sieve_plugins = sieve_imapsieve sieve_extprograms

  # From elsewhere to Spam folder
  imapsieve_mailbox1_name = Spam
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_before = file:/usr/local/lib/dovecot/sieve/report-spam.sieve

  # From Spam folder to elsewhere
  imapsieve_mailbox2_name = *
  imapsieve_mailbox2_from = Spam
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve

  sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve

  sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment
}

This piece of configuration was taken from the official Dovecot documentation: https://doc.dovecot.org/configuration_manual/howto/antispam_with_sieve/ .  It will trigger shell scripts calling rspamd to make it learn what does a spam look like, and what is legit (ham).  One script will run when an email is moved out of the spam directory (ham), another one when an email is moved to the spam directory (spam).

Modify /etc/dovecot/conf.d/15-mailboxes.conf to add the following snippet inside the block namespace inbox { ... }, it will associate the Junk directory as the folder containing spam and automatically create it if it does not exist:

  mailbox Spam {
    auto = create
    special_use = \Junk
  }

To make this work completely, you need to write the two extra sieve filters that will run trigger the scripts:

Create /usr/local/lib/dovecot/sieve/report-spam.sieve

require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"];

if environment :matches "imap.user" "*" {
  set "username" "${1}";
}

pipe :copy "sa-learn-spam.sh" [ "${username}" ];

Create /usr/local/lib/dovecot/sieve/report-ham.sieve

require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"];

if environment :matches "imap.mailbox" "*" {
  set "mailbox" "${1}";
}

if string "${mailbox}" "Trash" {
  stop;
}

if environment :matches "imap.user" "*" {
  set "username" "${1}";
}

pipe :copy "sa-learn-ham.sh" [ "${username}" ];

Create /usr/local/lib/dovecot/sieve/sa-learn-ham.sh

#!/bin/sh
exec /usr/local/bin/rspamc -d "${1}" learn_ham

Create /usr/local/lib/dovecot/sieve/sa-learn-spam.sh

#!/bin/sh
exec /usr/local/bin/rspamc -d "${1}" learn_spam

Make the two scripts executable with chmod +x /usr/local/lib/dovecot/sieve/sa-learn-spam.sh /usr/local/lib/dovecot/sieve/sa-learn-ham.sh.

Run the following command to compile the sieve filters:

sievec /usr/local/lib/dovecot/sieve/report-spam.sieve
sievec /usr/local/lib/dovecot/sieve/report-ham.sieve

5.4.5. Manage Sieve §
By default, Sieves rules are a file located on the user home directory, however there is a standard protocol named "managesieve" to manage Sieve filters remotely from an email client.

It is enabled out of the box in Dovecot configuration, although you need to make sure you open the port 4190/tcp in the firewall if you want to allow users to use it.

5.4.6. Start the service §
Once you configured everything, make sure that dovecot service is enabled, and then start / restart it:

rcctl enable dovecot
rcctl start dovecot

6. Webmail §
A webmail will allow your users to read / send emails from a web interface instead of having to configure a local email client.  While they can be convenient, they enable a larger attack surface and are often affected by vulnerability issues, you may prefer to avoid webmail on your server.

The two most popular open source webmail are Roundcube mail and Snappymail (a fork of the abandoned rainloop) and Roundcube, they both have pros and cons.

6.1. Roundcube mail setup §
Roundcube is packaged in OpenBSD, it will pull in all required dependencies and occasionally receive backported security updates.

Install the package:

pkg_add roundcubemail

When installing the package, you will be prompted for a database backend for PHP.  If you have one or two users, I highly recommend choosing SQLite as it will work fine without requiring a running daemon, thus less maintenance and server resources locked.  If you plan to have a lot of users, there are no wrong picks between MySQL or PostgreSQL, but if you already have one of them running it would be better to reuse it for Roundcube.

Specific instructions for installing Roundcube are provided by the package README in /usr/local/share/doc/pkg-readmes/roundcubemail.

We need to enable a few PHP modules to make Roundcube mail working:

ln -s /etc/php-8.2.sample/zip.ini /etc/php-8.2/
ln -s /etc/php-8.2.sample/intl.ini /etc/php-8.2/
ln -s /etc/php-8.2.sample/opcache.ini /etc/php-8.2/
ln -s /etc/php-8.2.sample/pdo_sqlite.ini /etc/php-8.2/

Note that more PHP modules may be required if you enable extra features and plugins in Roundcube.

PHP is ready to be started:

rcctl enable php82_fpm
rcctl start php82_fpm

Add the following blocks to /etc/httpd.conf, make sure you opened the port 443/tcp in your pf.conf and that you reloaded it with pfctl -f /etc/pf.conf:

server "mail.puffy.cafe" {

    listen on egress tls

    tls key "/etc/ssl/private/mail.puffy.cafe.key"
    tls certificate "/etc/ssl/mail.puffy.cafe.fullchain.pem"

    root "/roundcubemail"

    directory index index.php

    location "*.php" {
        fastcgi socket "/run/php-fpm.sock"
    }
}

types {
    include "/usr/share/misc/mime.types"
}

Restart httpd with rcctl restart httpd.

You need to configure Roundcube to use a 24 bytes security key and configure the database: edit the file /var/www/roundcubemail/config/config.inc.php:

Search for the variable des_key, replace its value by the output of the command tr -dc [:print:] < /dev/urandom | fold -w 24 | head -n 1 which will generate a 24 byte random string.  If the string contains a quote character, either escape this character by prefixing it with a \ or generate a new string.

For the database, you need to search the variable db_dsnw.

If you use SQLite, change this line

$config['db_dsnw'] = 'sqlite:///roundcubemail/db/sqlite.db?mode=0660';

by this line:

$config['db_dsnw'] = 'sqlite:///db/sqlite.db?mode=0660';

If you chose MySQL/MariaDB or PostgreSQL, modify this line:

$config['db_dsnw'] = 'mysql://roundcube:pass@localhost/roundcubemail';

by

$config['db_dsnw'] = 'mysql://USER:PASSWORD@DATABASE_NAME';

Where USER, PASSWORD and DATABASE_NAME must match a new user and database created into the backend.

Because PHP is chrooted on OpenBSD and that the OpenSMTPD configuration enforces TLS on port 587, it is required to enable TLS to work in the chroot:

mkdir -p /var/www/etc/ssl
cp -p /etc/ssl/cert.pem /etc/ssl/openssl.cnf /var/www/etc/ssl/

To make sure the files cert.pem and openssl.cnf stay in sync after upgrades, add the two commands to a file /etc/rc.local and make this file executable.  This script always starts at boot and is the best place for this kind of file copy.

If your IMAP and SMTP hosts are not on the same server where Roundcube is installed, adapt the variables imap_host and smtp_host to the server name.

If Roundcube mail is running on the same server where OpenSMTPD is running, you need to disable certificate validation because localhost will not match the certificate and authentication will fail.  Change smtp_host line to $config['smtp_host'] = 'tls://127.0.0.1:587'; and add this snippet to the configuration file:

$config['smtp_conn_options'] = array(
'ssl' => array('verify_peer' => false, 'verify_peer_name' => false),
'tls' => array('verify_peer' => false, 'verify_peer_name' => false));

From here, Roundcube mail should work when you load the domain configured in httpd.conf.

For a more in-depth guide to install and configure Roundcube mail, there is an excellent guide available which was written by Bruno Flückiger:

Install Roundcube on OpenBSD
7. Hardening §
It is always possible to improve the security of this stack, all the following settings are not mandatory, but they can be interesting depending on your needs.

7.1. Always allow the sender per email or domain §
It is possible to configure rspamd to force it to accept emails from a given email address or domain, bypassing the anti-spam.

To proceed, edit the file /etc/rspamd/local.d/multimap.conf to add this content:

local_wl_domain {
        type = "from";
        filter = "email:domain";
        map = "$CONFDIR/local.d/whitelist_domain.map";
        symbol = "LOCAL_WL_DOMAIN";
        score = -10.0;
        description = "domains that are always accepted";
}

local_wl_from {
        type = "from";
        map = "$CONFDIR/local.d/whitelist_email.map";
        symbol = "LOCAL_WL_FROM";
        score = -10.0;
        description = "email addresses that are always accepted";
}

Create the files /etc/rspamd/local.d/whitelist_domain.map and /etc/rspamd/local.d/whitelist_email.map using the command touch.

Restart the service rspamd with rcctl restart rspamd.

The created files use a simple syntax, add a line for each entry you want to allow:



  a domain name in /etc/rspamd/local.d/whitelist_domain.map to allow the domain
  an email address in /etc/rspamd/local.d/whitelist_email.map to allow this address


There is no need to restart or reload rspamd after changing the files.

Reusing the same technique can be done to block domains/addresses directly in rspamd by giving a high positive score.

7.2. Block bots §
I published on my blog a script and related configuration to parse OpenSMTPD logs and block the bad actors with PF.

2023-06-22 Ban scanners IPs from OpenSMTP logs
This includes an ignore file if you do not want some IPs to be blocked.

7.3. Split the stack §
If you want to improve your email setup security further, the best method is to split each part into dedicated systems.

As dovecot is responsible for storing and exposing emails to users, this component would be safer in a dedicated system, so if a component of the email stack (other than dovecot) is compromised, the mailboxes will not be exposed.

7.4. Network attack surface reduction §
If this does not go against usability of the email server users, I strongly recommend limiting the publicly opened ports in the firewall to the minimum: 25, 80, 465, 587.  This would prevent attackers to exploit any network related 0day or unpatched vulnerabilities of non-exposed services such as Dovecot.

A VPN should be deployed to allow users to reach Dovecot services (IMAP, POP) and other services if any.

SSH port could be removed from the public ports as well, however, it would be safer to make sure your hosting provider offers a serial access / VNC / remote access to the system because if the VPN stops working, you will not be able to log in into the system using SSH to debug it.

8. Email client configuration §
If everything was done correctly so far, you should have a complete email stack fully functional.

Here are the connection information to use your service:



  IMAP/POP3/SMTP login: username on the remote system (the username does not include the @ part)
  IMAP/POP3/SMTP password: password of the remote system user
  IMAP/POP3 server: dovecot server hostname
  IMAP/POP3 port: 993 for IMAPS and 995 for POP3S (TLS is enabled)
  SMTP server: opensmtpd server hostname
  SMTP port: either 465 in SSL/TLS mode (encryption forced), or 587 in STARTTLS mode (encryption not enforced depending on OpenSMTPD configuration)


The webmail, if any, will be available at the address configured in httpd.conf, using the same credentials as above.

9. Verify the setup §
There is an online service providing you a random email address to send a test email to, then you can check the result on their website displaying if the SPF, DKIM, DMARC and PTR records are correctly configured.

www.mail-tester.com
The score you want to be displayed on their website is no least than 10/10.  The service can report meaningless issues like "the email was poorly formatted" or "you did not include an unsubscribe link", they are not relevant for the current test.

While it used to be completely free last time I used it, I found it would ask you to pay after three free checks if you do not want to wait 24h.  It uses your public IP address for the limit.

10. Maintenance §
10.1. Running processes §
The following processes list should always be running: using a program like monit, zabbix or reed-alert to notify you when they stop working could be a good idea.



  dovecot
  httpd
  redis
  rspamd
  smtpd


10.2. Certificates renewal §
In addition, the TLS certificate should be renewed regularly as ACME generated certificates are valid for a few months.  Edit root crontab with crontab -e as root to add this line:

10 4 * * 0 -s acme-client mail.puffy.cafe && rcctl restart dovecot httpd smtpd

This will try to renew the certificate for mail.puffy.cafe every Sunday at 04h10 and upon renewal restart the services using the certificate: dovecot, httpd and smtpd.

10.3. All about logs §
If you need to find some logs, here is a list of paths where to find information:



  dovecot: /var/log/maillog
  httpd: /var/log/daemon for the daemon, access logs in /var/www/logs/access.log and errors logs in /var/www/logs/error.log
  redis: /var/log/daemon
  rspamd: /var/log/rspamd/rspamd.log and its web UI on port 11334 (only on localhost by default, a SSH tunnel can be handy)
  smtpd: /var/log/maillog
  roundcube: /var/www/roundcubemail/logs/errors.log and /var/www/roundcubemail/logs/sendmail.log


A log rotation of the new logs can be configured in /etc/newsyslog.conf with these lines (take only what you need):

/var/log/rspamd/rspamd.log		600  7     500  *     Z "pkill -USR1 -u root -U root -x rspamd"
/var/www/roundcubemail/logs/errors.log	600  7     500  *     Z
/var/www/roundcubemail/logs/sendmail.log 600 7     500  *     Z

10.4. Disk space §
Finally, OpenSMTPD will stop delivering emails locally if the /var partition has less than 4% of free disk space, be sure to monitor the disk space of this partition otherwise you will not receive emails anymore for a while before noticing something is wrong.

11. Conclusion §
Congratulations, you configured a whole email stack that will allow you to send emails to the world, using your own domain and hardware.  Keeping your system up to date is important as you have network services exposed to the wild Internet.

Even with a properly configured setup featuring SPF/DKIM/DMARC/PTR, it is not guaranteed to not end in the spam directory of our recipients.  The IP reputation of your SMTP server also account, and so is the domain name extension (I have a .pw domain which I learned too late that it was almost always considered as spam because it is not mainstream).

Solene'%

Using a dedicated administration workstation for my infrastructure

Table of contents

1. Introduction §

2. The need §

3. Setup §

3.1. Workstation §

3.2. Servers §

3.3. File exchange §

4. Conclusion §

Securing backups using S3 storage

Table of contents

1. Introduction §

2. Quick intro to object storage §

2.1. Open source S3-compatible storage implementations §

3. Configure your S3 §

4. Conclusion §

5. Going further §

Snap integration in Qubes OS templates

Table of contents

1. Introduction §

2. Setup on Fedora §

2.1. Snap installation §

2.2. Proxy configuration §

2.3. Run updates on template update §

2.4. Qube settings menu integration §

2.5. Snap store GUI §

3. Debian §

4. Conclusion §

Asynchronous secure file transfer with nncp

Table of contents

1. Introduction §

2. Explanations §

3. Real world example: Syncthing gateway §

4. Setup §

4.1. Configuration file and private keys §

4.2. Public keys §

4.3. Import public keys §

5. Usage §

6. Workflow (how to use) §

6.1. Sending files §

6.2. Sync and file unpacking §

7. Explanations about ACK §

8. Conclusion §

9. Going further §

I moved my emails to Proton Mail

Table of contents

1. Introduction §

2. My needs §

3. Proton Mail §

3.1. Benefits §

3.2. Interesting features §

3.3. Shortcomings §

3.4. Alternatives §

4. My ideal email setup §

5. Conclusion §

5.1. Update 2024-09-14 §

Self-hosting at home and privacy

Table of contents

1. Introduction §

2. Public information §

2.1. Domain WHOIS §

2.2. TLS certificates using ACME §

2.3. Domain name §

2.4. Public IP §

3. Mitigations §

4. Conclusion §

How to use Proton VPN port forwarding

Table of contents

1. Introduction §

2. Feature explanation §

3. Setup §

3.1. OpenBSD §

3.1.1. Using supervisord §

3.1.2. Without supervisord §

3.2. Linux §

4. Conclusion §

Emails encryption at rest on OpenBSD using dovecot and GPG

Table of contents

1. Introduction §