About me: My name is Solène Rapenne, pronouns she/her. I like learning and sharing knowledge. Hobbies: '(BSD OpenBSD Qubes OS Lisp cmdline gaming security QubesOS internet-stuff). I love percent and lambda characters. Qubes OS core team member, former OpenBSD developer solene@. No AI is involved in this blog.

Contact me: solene at dataswamp dot org or @solene@bsd.network (mastodon).

I'm a freelance OpenBSD, FreeBSD, Linux and Qubes OS consultant, this includes DevOps, DevSecOps, technical writing or documentation work. If you enjoy this blog, you can sponsor my open source work financially so I can write this blog and contribute to Free Software as my daily job.

Aggregate internet links with mlvpn

Written by Solène, on 28 March 2020.
Tags: #openbsd

Comments on Fediverse/Mastodon

In this article I’ll explain how to aggregate internet access bandwidth using mlvpn software. I struggled a lot to set this up so I wanted to share a how-to.

Pre-requisites

mlvpn is meant to be used with DSL / fiber links, not wireless or 4G links with variable bandwidth or packet loss.

mlvpn requires to be run on a server which will be the public internet access and on the client on which you want to aggregate the links, this is like doing multiples VPN to the same remote server with a VPN per link, and aggregate them.

Multi-wan roundrobin / load balancer doesn’t allow to stack bandwidth but doesn’t require a remote server, depend on what you want to do, this may be enough and mlvpn may not be required.

mlvpn should be OS agnostic between client / server but I only tried between two OpenBSD hosts, your setup may differ.

Some network diagram

Here is a simple network, the client has access to 2 ISP through two ethernet interfaces.

em0 and em1 will have to be on different rdomains (it’s a feature to separate routing tables).

Let’s say the public ip of the server is 1.2.3.4.

                [internet]
                    ↑
                    | (public ip on em0)
             #-------------#
             |             |
             |   Server    |
             |             |
             #-------------#
                |       |
                |       |
                |       |
                |       |
    (internet)  |       | (internet)
    #-------------#   #-------------#
    |             |   |             |
    |   ISP 1     |   |  ISP 2      |
    |             |   |             |  (you certainly don't control those)
    #-------------#   #-------------#
                |       |
                |       |
  (dsl1 via em0)|       | (dsl1 via em1)
             #-------------#
             |             |
             |   Client    |
             |             |
             #-------------#

Network configuration

As said previously, em0 and em1 must be on different rdomains, it can easily be done by adding rdomain 1 and rdomain 2 to the interfaces configuration.

Example in /etc/hostname.em0

rdomain 1
dhcp

mlvpn installation

On OpenBSD the installation is as easy as pkg_add mlvpn (should work starting from 6.7 because it required patching).

mlvpn configuration

Once the network configuration is done on the client, there are 3 steps to do to get aggregation working:

  1. mlvpn configuration on the server
  2. mlvpn configuration on the client
  3. activating NAT on the client

Server configuration

On the server we will use the UDP ports 5080 et 5081.

Connections speed must be defined in bytes to allow mlvpn to correctly balance the traffic over the links, this is really important.

The line bandwidth_upload = 1468006 is the maximum download bandwidth of the client on the specified link in bytes. If you have a download speed of 1.4 MB/s then you can choose a value of 1.4*1024*1024 => 1468006.

The line bandwidth_download = 102400 is the maximum upload bandwidth of the client on the specified link in bytes. If you have an upload speed of 100 kB/s then you can choose a value of 100*1024 => 102400.

The password line must be a very long random string, it’s a shared secret between the client and the server.

# config you don't need to change
[general]
statuscommand = "/etc/mlvpn/mlvpn_updown.sh"
protocol = "tcp"
loglevel = 4
mode = "server"
tuntap = "tun"
interface_name = "tun0"
cleartext_data = 0
ip4 = "10.44.43.2/30"
ip4_gateway = "10.44.43.1"
    
# things you need to change
password = "apoziecxjvpoxkvpzeoirjdskpoezroizepzdlpojfoiezjrzanzaoinzoi"
    
[dsl1]
bindhost = "1.2.3.4"
bindport = 5080
bandwidth_upload = 1468006
bandwidth_download = 102400
    
[dsl2]
bindhost = "1.2.3.4"
bindport = 5081
bandwidth_upload = 1468006
bandwidth_download = 102400

Client configuration

The password value must match the one on the server, the values of ip4 and ip4_gateway must be reversed compared to the server configuration (this is so in the following example).

The bindfib lines must correspond to the according rdomain values of your interfaces.

# config you don't need to change
[general]
statuscommand = "/etc/mlvpn/mlvpn_updown.sh"
loglevel = 4
mode = "client"
tuntap = "tun"
interface_name = "tun0"
ip4 = "10.44.43.1/30"
ip4_gateway = "10.44.43.2"
timeout = 30
cleartext_data = 0
    
password = "apoziecxjvpoxkvpzeoirjdskpoezroizepzdlpojfoiezjrzanzaoinzoi"
    
[dsl1]
remotehost = "1.2.3.4"
remoteport = 5080
bindfib = 1
    
[dsl2]
remotehost = "1.2.3.4"
remoteport = 5081
bindfib = 2

NAT configuration (server side)

As with every VPN you must enable packet forwarding and create a pf rule for the NAT.

Enable forwarding

Add this line in /etc/sysctl.conf:

net.inet.ip.forwarding=1

You can enable it now with sysctl net.inet.ip.forwarding=1 instead of waiting for a reboot.

In pf.conf you must allow the UDP ports 5080 and 5081 on the public interface and enable nat, this can be done with the following lines in pf.conf but you should obviously adapt to your configuration.

# allow NAT on VPN
pass in on tun0
pass out quick on em0 from 10.44.43.0/30 to any nat-to em0
    
# allow mlvpn to be reachable
pass in on egress inet proto udp from any to (egress) port 5080:5081

Start mlvpn

On both server and client you can run mlvpn with rcctl:

rcctl enable mlvpn
rcctl start mlvpn

You should see a new tun0 device on both systems and being able to ping them through tun0.

Now, on the client you have to add a default gateway through the mlvpn tunnel with the command route add -net default 10.44.43.2 (adapt if you use others addresses). I still didn’t find how to automatize it properly.

Your client should now use both WAN links and being visible with the remote server public IP address.

mlvpn can be used for more links, you only need to add new sections. mlvpn also support IPv6 but I didn’t take time to find how to make it work, si if you are comfortable with ipv6 it may be easy to set up IPv6 with the variables ip6 and ip6_gateway in mlvpn.conf.