Port of the week: mbuffer

This Port of the week is a bit special because sadly, the port isn’t available on OpenBSD. The port is mbuffer (which you can find in misc/mbuffer).

I discovered it while looking for a way to enhance one of my network stream scripts. I have some scripts that get a dump of a postgresql base through SSH, copy it from stdin to a file with tee and send it out to the local postgres, the command line looks like

$ ssh remote-base-server "pg_dump my_base | gzip -c -f -" | gunzip -f | tee dumps/my_base.dump | psql my_base

I also use the same kind of command to receive a ZFS snapshot from another server.

But there is an issue, the end server is relatively slow, postgresql and ZFS will eat lot of data from stdin and then it will stop for sometimes writing on the disk, when they are ready to take new data, it’s slow to fill them. This is where mbuffer takes places. This tool permit to add a buffer that will take data from stdin and fill its memory (that you set on the command line), so when the slowest part of the command is ready to take data, mbuffer will empty its memory into the pipe, so the slowlest command isn’t waiting to get filled before working again.

The new command looks like that for a buffer of 300 Mb

ssh remote-base-server "pg_dump my_base | gzip -c -f -" |  gunzip -f | tee dumps/my_base.dump | mbuffer -s 8192 -m 300M | psql my_base

mbuffer also comes with a nice console output, showing

• bandwith in

• bandwith out

• percentage/consumption of memory filled

• total transfered

  in @ 1219 KiB/s, out @ 1219 KiB/s, 906 MiB total, buffer 0% full

In this example the server is too fast so there is no wait, the buffer isn’t used (0% full).

mbuffer can also listen on TCP, unix socket and have a lot of parameters that I didn’t try, if you think that can be useful for you, just go for it !