About me: My name is Solène Rapenne. I like learning and sharing my knowledge related to IT stuff. Hobbies: '(BSD OpenBSD h+ Lisp cmdline gaming internet-stuff Crossbow). I love percent and lambda characters. OpenBSD developer solene@.

Contact me: solene on Freenode, solene+www at dataswamp dot org or solene@bsd.network (mastodon). If for some reason you want to give me some money, I accept paypal at the address donate@perso.pw.

Aggregate internet links with mlvpn

Written by Solène, on 28 March 2020.
Tags: #openbsd67

Comments on Mastodon

In this article I’ll explain how to aggregate internet access bandwidth using mlvpn software. I struggled a lot to set this up so I wanted to share a how-to.

Pre-requisites

mlvpn is meant to be used with DSL / fiber links, not wireless or 4G links with variable bandwidth or packet loss.

mlvpn requires to be run on a server which will be the public internet access and on the client on which you want to aggregate the links, this is like doing multiples VPN to the same remote server with a VPN per link, and aggregate them.

Multi-wan roundrobin / load balancer doesn’t allow to stack bandwidth but doesn’t require a remote server, depend on what you want to do, this may be enough and mlvpn may not be required.

mlvpn should be OS agnostic between client / server but I only tried between two OpenBSD hosts, your setup may differ.

Some network diagram

Here is a simple network, the client has access to 2 ISP through two ethernet interfaces.

em0 and em1 will have to be on different rdomains (it’s a feature to separate routing tables).

Let’s say the public ip of the server is 1.2.3.4.

                [internet]
                    ↑
                    | (public ip on em0)
             #-------------#
             |             |
             |   Server    |
             |             |
             #-------------#
                |       |
                |       |
                |       |
                |       |
    (internet)  |       | (internet)
    #-------------#   #-------------#
    |             |   |             |
    |   ISP 1     |   |  ISP 2      |
    |             |   |             |  (you certainly don't control those)
    #-------------#   #-------------#
                |       |
                |       |
  (dsl1 via em0)|       | (dsl1 via em1)
             #-------------#
             |             |
             |   Client    |
             |             |
             #-------------#

Network configuration

As said previously, em0 and em1 must be on different rdomains, it can easily be done by adding rdomain 1 and rdomain 2 to the interfaces configuration.

Example in /etc/hostname.em0

rdomain 1
dhcp

mlvpn installation

On OpenBSD the installation is as easy as pkg_add mlvpn (should work starting from 6.7 because it required patching).

mlvpn configuration

Once the network configuration is done on the client, there are 3 steps to do to get aggregation working:

  1. mlvpn configuration on the server
  2. mlvpn configuration on the client
  3. activating NAT on the client

Server configuration

On the server we will use the UDP ports 5080 et 5081.

Connections speed must be defined in bytes to allow mlvpn to correctly balance the traffic over the links, this is really important.

The line bandwidth_upload = 1468006 is the maximum download bandwidth of the client on the specified link in bytes. If you have a download speed of 1.4 MB/s then you can choose a value of 1.4*1024*1024 => 1468006.

The line bandwidth_download = 102400 is the maximum upload bandwidth of the client on the specified link in bytes. If you have an upload speed of 100 kB/s then you can choose a value of 100*1024 => 102400.

The password line must be a very long random string, it’s a shared secret between the client and the server.

# config you don't need to change
[general]
statuscommand = "/etc/mlvpn/mlvpn_updown.sh"
protocol = "tcp"
loglevel = 4
mode = "server"
tuntap = "tun"
interface_name = "tun0"
cleartext_data = 0
ip4 = "10.44.43.2/30"
ip4_gateway = "10.44.43.1"

# things you need to change
password = "apoziecxjvpoxkvpzeoirjdskpoezroizepzdlpojfoiezjrzanzaoinzoi"

[dsl1]
bindhost = "1.2.3.4"
bindport = 5080
bandwidth_upload = 1468006
bandwidth_download = 102400

[dsl2]
bindhost = "1.2.3.4"
bindport = 5081
bandwidth_upload = 1468006
bandwidth_download = 102400

Client configuration

The password value must match the one on the server, the values of ip4 and ip4_gateway must be reversed compared to the server configuration (this is so in the following example).

The bindfib lines must correspond to the according rdomain values of your interfaces.

# config you don't need to change
[general]
statuscommand = "/etc/mlvpn/mlvpn_updown.sh"
loglevel = 4
mode = "client"
tuntap = "tun"
interface_name = "tun0"
ip4 = "10.44.43.1/30"
ip4_gateway = "10.44.43.2"
timeout = 30
cleartext_data = 0

password = "apoziecxjvpoxkvpzeoirjdskpoezroizepzdlpojfoiezjrzanzaoinzoi"

[dsl1]
remotehost = "1.2.3.4"
remoteport = 5080
bindfib = 1

[dsl2]
remotehost = "1.2.3.4"
remoteport = 5081
bindfib = 2

NAT configuration (server side)

As with every VPN you must enable packet forwarding and create a pf rule for the NAT.

Enable forwarding

Add this line in /etc/sysctl.conf:

net.inet.ip.forwarding=1

You can enable it now with sysctl net.inet.ip.forwarding=1 instead of waiting for a reboot.

In pf.conf you must allow the UDP ports 5080 and 5081 on the public interface and enable nat, this can be done with the following lines in pf.conf but you should obviously adapt to your configuration.

# allow NAT on VPN
pass in on tun0
pass out quick on em0 from 10.44.43.0/30 to any nat-to em0

# allow mlvpn to be reachable
pass in on egress inet proto udp from any to (egress) port 5080:5081

Start mlvpn

On both server and client you can run mlvpn with rcctl:

rcctl enable mlvpn
rcctl start mlvpn

You should see a new tun0 device on both systems and being able to ping them through tun0.

Now, on the client you have to add a default gateway through the mlvpn tunnel with the command route add -net default 10.44.43.2 (adapt if you use others addresses). I still didn’t find how to automatize it properly.

Your client should now use both WAN links and being visible with the remote server public IP address.

mlvpn can be used for more links, you only need to add new sections. mlvpn also support IPv6 but I didn’t take time to find how to make it work, si if you are comfortable with ipv6 it may be easy to set up IPv6 with the variables ip6 and ip6_gateway in mlvpn.conf.

Stream live video using nginx

Written by Solène, on 26 August 2019.
Tags: #openbsd67 #openbsd #gaming #nginx

Comments on Mastodon

This blog post is about a nginx rtmp module for turning your nginx server into a video streaming server.

The official website of the project is located on github at: https://github.com/arut/nginx-rtmp-module/

I use it to stream video from my computer to my nginx server, then viewers can use mpv rtmp://perso.pw/gaming in order to view the video stream. But the nginx server will also relay to twitch for more scalability (and some people prefer viewing there for some reasons).

The module will already be installed with nginx package since OpenBSD 6.6 (not already out at this time).

There is no package for install the rtmp module before 6.6. On others operating systems, check for something like “nginx-rtmp” or “rtmp” in an nginx context.

Install nginx on OpenBSD:

pkg_add nginx

Then, add the following to the file /etc/nginx/nginx.conf

load_module modules/ngx_rtmp_module.so;
rtmp {
    server {
        listen 1935;
        buflen 10s;

        application gaming {
            live on;
            allow publish 176.32.212.34;
            allow publish 175.3.194.6;
            deny publish all;
            allow play all;

            record all;
            record_path /htdocs/videos/;
            record_suffix %d-%b-%y_%Hh%M.flv;

        }
    }
}

The previous configuration sample is a simple example allowing 172.32.212.34 and 175.3.194.6 to stream through nginx, and that will record the videos under /htdocs/videos/ (nginx is chrooted in /var/www).

You can add the following line in the “application” block to relay the stream to your Twitch broadcasting server, using your API key.

push rtmp://live-ams.twitch.tv/app/YOUR_API_KEY;

I made a simple scripts generating thumbnails of the videos and generating a html index file.

Every 10 minutes, a cron check if files have to be generated, make thumbnails for videos (tries at 05:30 of the video and then 00:03 if it doesn’t work, to handle very small videos) and then create the html.

The script checking for new stuff and starting html generation:

#!/bin/sh

cd /var/www/htdocs/videos

for file in $(find . -mmin +1 -name '*.flv')
do
        echo $file
        PIC=$(echo $file | sed 's/flv$/jpg/')
        if [ ! -f "$PIC" ]
        then
                ffmpeg -ss 00:05:30 -i "$file" -vframes 1 -q:v 2 "$PIC"
                if [ ! -f "$PIC" ]
                then
                        ffmpeg -ss 00:00:03 -i "$file" -vframes 1 -q:v 2 "$PIC"
                        if [ ! -f "$PIC" ]
                        then
                                echo "problem with $file" | mail user@my-tld.com
                        fi
                fi
        fi
done
cd ~/dev/videos/ && sh html.sh

This one makes the html:

#!/bin/sh

cd /var/www/htdocs/videos

PER_ROW=3
COUNT=0

cat << EOF > index.html
<html>
  <body>
<h1>Replays</h1>
<table>
EOF

for file in $(find . -mmin +3 -name '*.flv')
do
        if [ $COUNT -eq 0 ]
        then
                echo "<tr>" >> index.html
                INROW=1
        fi
        COUNT=$(( COUNT + 1 ))
        SIZE=$(ls -lh $file  | awk '{ print $5 }')
        PIC=$(echo $file | sed 's/flv$/jpg/')

        echo $file
        echo "<td><a href=\"$file\"><img src=\"$PIC\" width=320 height=240 /><br />$file ($SIZE)</a></td>" >> index.html
        if [ $COUNT -eq $PER_ROW ]
        then
                echo "</tr>" >> index.html
                COUNT=0
                INROW=0
        fi
done

if [ $INROW -eq 1 ]
then
        echo "</tr>" >> index.html
fi

cat << EOF >> index.html
    </table>
  </body>
</html>
EOF