The container is called ubuntu-focal-interfaces and the wireguard interface is called wg1.

Here’s the steps needed to accomplish that.

Adding the Wireguard interface on the host with systemd-networkd

If you’re not using systemd-networkd on the host machine, just create a wireguard interface as described in the wireguard docs but don’t set an IP address on the interface.

systemd-networkd can create wireguard interfaces itself using a .netdev file. Create one in /etc/systemd/network/wg1.netdev:

[NetDev]
Name=wg1
Kind=wireguard
Description=Wireguard client interface, used in a container

[WireGuard]
# Bonus points if you actually use PrivateKeyFile
PrivateKey=XXXXXX
ListenPort=51820

[WireGuardPeer]
Endpoint=some-host.example.com:51820
PublicKey=YYYYYY
AllowedIPs=192.0.2.0/24

# Ensure the NAT knows about this connection
PersistentKeepalive=60

As it contains private keys, systemd-networkd will refuse to load it if the permissions are too wide, so

sudo chown root:systemd-network /etc/systemd/network/wg1.netdev
sudo chmod 640 /etc/systemd/network/wg1.netdev`.

Now create the interface:

sudo networkctl reload

Creating the image

Now we actually need to run a container, for this we need an image that nspawn can run. Fortunately, mkosi has us covered. Download/install it and create a directory where we’ll add the image build config.

Now create a mkosi.default file:

[Distribution]
Distribution=ubuntu
Release=focal

[Packages]
Packages=iproute2

We’ll bake in the setting of the IP address on the interface into the image, mkosi.postinst:

#!/bin/bash
systemctl enable systemd-networkd
echo "[Match]
Name=wg1

[Network]
Address=192.0.2.2/24" > /etc/systemd/network/wg1.network

Now build the image and import it into machined:

chmod +x mkosi.postinst
sudo mkosi
# wait......
sudo machinectl import-raw image.raw ubuntu-focal-interfaces

Adding the right interfaces to the container

Configure nspawn to pass the wg1 interface to the container and set up a veth interface as well. Create a file /etc/systemd/nspawn/ubuntu-focal-interfaces.nspawn

[Network]
VirtualEthernet=true
Interface=wg1

Boot the container

Now start the container:

sudo machinectl start ubuntu-focal-interfaces

If you now run ip address show on the host, you’ll see the wg1 interface is gone, as it has moved to the container’s network namespace.

When you open a shell in the container (sudo machinectl shell ubuntu-focal-interfaces) and run ip address show there, you’ll see two interfaces, one veth interface that has internet access and one wg1 interface with the configured address.

Should you stop the container (sudo machinectl stop ubuntu-focal-interfaces), the wg1 interface is moved back into the host’s namespace. The IP address, however, is no longer configured on the interface. This means I can now now safely tell the people who need access the public key of the wireguard interface, start the container, install SSH and give them access to do whatever they need to do.

Security Considerations

Should the other party have root permissions inside your container, they can still edit the wireguard interface’s parameters. These changes will be lost once the host is rebooted of course, but keep this in mind when moving interfaces between namespaces and containers.

Footnotes

1. “infrastructure” sounds impressive, this whole thing ran on the Raspberry Pi 4 that is my house’s wiring closet conencted to the uplink switch.

The keynote on what Site Reliability Engineering (SRE) means by David Blank-Edelmann from Microsoft was a treat to listen to.

After that the multi-track part of the day started. It was hard to pick what talk to attend, but I was at no point disappointed by the choices made.

Michael Boelen did a highly interactive talk on improving the shell-scripting game of those in the audience.

After that, Roland van Rijswijk-Deij donned a proper shirt (“I was considering a suit”) to explain the Quantum Blockchain Cloud and why not all crypto is doomed with the coming Quantum Supremecy.

Later in the afternoon, Anco Scholte ter Horst gave a small history of the absorbsion of XS4All into KPN, why and how Freedom Internet started, and what the plans for the near-future are.

Just before I had to hit the stage, Wim ter Have from Oracle gave a highly technical talk on NUMA awareness in libvirt. Allowing the mapping of NUMA nodes from the host to guests for blazing performance.

After that it was my time to present an updated version of my LOADays 2019 on systemd-nspawn and mkosi that I accidentally started in Dutch (sorry!). The slides are available here.

After that, it was time for the closing keynote on security fails and hack stories by Edwin van Andel.

All the talks were recorded, and I’ll certainly view some of the ones I missed. Hats off to the NLUUG program committee for putting together such a highly informative and densely packed conference.

I must admit I did not watch all the talks, as it was sunny outside and the discussions were good as well.

Frank Louwers kicked off day 1 with his talk “GDPR for nerds”. It was a good introduction into the terms, rules and consequences of the European General Data Protection Regulation that will go into effect May 25th of 2018. The main take-away was “When in doubt, you are a data processor”.

Friend of PowerDNS and all-round excellent person Jan-Piet Mens gave an introduction to Ansible AWX. AWX is the Open-Source version of the commercial Ansible Tower product, a centralized dashboard from where one can control their infrastructure.

On the security-front, Jean-François Maes gave a short pen-test walk-through of a (purposely) badly secured Docker deployment that allowed him to gain root. During this walk-through, several easy to make mistakes in application deployments were discussed.

After this, it was time for more sunny discussions and the famous beer and fresh-baked pizza social.

The morning of the second day was filled with theoretical talks. First there was Bram Vogelaar discussing Object-Oriented principles for Infrastructure as Code, using Puppet as an example. Followed by Serge van Ginderachter who talked about a non-existing project that would handle inventory and variables in an intuitive way, discussing theory and practical pitfalls alike.

In the afternoon it was up to me to tell the assembled collection of system administrators about the upcoming removal of EDNS work arounds by the 4 large Open-Source DNS Recursor vendors. The most important message was “run up-to-date software, check you firewalls and middleboxes and test your domains”. For those interested, the slide deck is available.

Peter Czanik was up next to talk about “making sense of syslog-ng”, where to me it was clear that it can do almost everything LogStash does, which was refreshing.

Last talk that I attended was Kristof Provost’s humorous overview of FreeBSD. He discussed its history, differences with GNU/Linux and project philosophy. As expected at a Linux-oriented conference, shenanigans were aplenty.

All in all, it was a very successful, sunny and densely-packed Loadays. Thanks to Toshaan Bharvani, Kris Buytaert and the rest of the team for putting it together and allowing me the opportunity to speak.

Update 2018-05-06: @maniacnl has uploaded a video of the presentation, embedded below.

The Plan and Execution

On the 10th of October we started a small internal project to change the set up of our authoritative infrastructure. This project had as a goal to have a more representable DNS authoritative infrastructure we could use to better dogfood our own software. In short, this was the work that had to be done:

• Upgrade to 4.1.0-rc on pdns-public-ns1.powerdns.com
• Switch to the gsqlite3 backend from the BIND backend
• Add a hidden master on pdns-public-ns1.powerdns.com and slave via the local loopback

The upgrade was easy enough with the use of the repositories. The whole migration from BIND to gsqlite3 was very straight-forward thanks to pdnsutil b2b-migrate, although some bugs were found and subsequently fixed. Adding the hidden master was also effortless and after changing the zones from ‘MASTER’ to ‘SLAVE’ and setting the master addresses to the local loopback address the whole chain of zone transfers was tested and worked properly: NOTIFY mesages were sent, AXFRs were performed and dig was happy.

Yay, Time to stop working for the day!

The Missed Warning Signs

The next morning I was pretty sleepy on a train on my way to Open-Xchange Summit and saw a bunch of monitoring emails come in regarding the DNSSEC validation of our domains:

Info:    CRITICAL powerdns.com: validation failure <powerdns.com. SOA IN>: RSA signature verification failed from 2604:a880:1:20::132:5001

And yes, we actually monitor SOA freshness between master and slaves and the DNSSEC chain of trust for all our domains (as one should). In my sleepy state, I decided these were probably emails from yesterday and just stared out the window until I arrived in Brussels. Once there, I got a message from the cool people at internet.nl that the RRSIG sent alongside the SOA for powerdns.com was not valid. Oops!

The Issue

In the PowerDNS Authoritative Server, we try really hard to ensure that slave server will fetch fresh RRSIGs when zones are resigned. One of these ways is the SOA-EDIT domain metadata. This piece of metadata lets the nameserver process bump the SOA serial artificially when serving the record from the backend.

There is also another piece of metadata called PRESIGNED. This indicates that the zone was AXFR’d from a master and already has DNSSEC signatures in the backend, it is set automatically when we see RRSIGs in the incoming AXFR.

During the b2b-migrate, the domain metadata for all zones was correctly migrated. This included the SOA-EDIT we had in place for powerdns.com.

So after AXFR’ing the zone, we had the following metadata in the database:

select * from domainmetadata where domain_id=41;
9|41|SOA-EDIT|INCEPTION-INCREMENT
11|41|PRESIGNED|1

When querying the nameserver the following records were returned:

; <<>> DiG 9.11.2 <<>> +dnssec soa powerdns.com @pdns-public-ns1.powerdns.com.
...
;; ANSWER SECTION:
powerdns.com.      3600  IN   SOA     pdns-public-ns1.powerdns.com. pieter\.lexis.powerdns.com. 2017101203 10800 3600 604800 3600
powerdns.com.      3600  IN   RRSIG   SOA 8 2 3600 20171026000000 20171005000000 36021 powerdns.com. C/VixIC.....

After some debugging, the PRESIGNED metadata was removed. And lo and behold, the following was returned:

;; ANSWER SECTION:
powerdns.com.      3600  IN   RRSIG   SOA 8 2 3600 20171026000000 20171005000000 36021 powerdns.com. C/VixIC.....
powerdns.com.      3600  IN   SOA     pdns-public-ns1.powerdns.com. pieter\.lexis.powerdns.com. 2017101201 10800 3600 604800 3600

Notice that the SOA serial is reduced by 2. This SOA record actually validates correctly:

Info:    OK powerdns.com: the chain of trust is valid.

So it appears PowerDNS honors the SOA-EDIT metadata for pre-signed zones, oops! To prevent others from running into this issue, the problem was clearly documented and a solution was writen within an hour. This fix will land in the next release of the PowerDNS Authoritative Server, and will be backported to the 4.0 release train as well.

What We Learned

Several things were learned in the course of migrating and figuring out what was going:

The SOA-EDIT interaction with PRESIGNED was not known and thus a corner case that was not foreseen in the migration plan.

Our DNSSEC trust-chain monitoring only found this issue because it queries for SOA. In the future we will write a checking script that will attempt to check all (or at least many) RRSIGs in the zone.

And most importantly: eating your own dogfood is an amazing way to find bugs before your users do!

As a beta feature, Mattermost allows one-on-one video calls using WebRTC. A WebRTC proxy is needed to allow the calling parties to establish communication. Janus is the recommended proxy software. Unfortunately, the documentation is sparse on how to configure Janus to achieve this.

This guide assumes a Janus 0.2.3 installation on Debian or Ubuntu, but should be applicable to other versions and operating systems. It only shows settings that should be modified, modify all others at you own discretion. The configuration files for Janus have many comments for each setting.

Configuring Janus

Janus support tons of transports and has many settings. For Mattermost, the secure WebSockets transport and the HTTPS admin need to be configured.

After installing Janus, edit /etc/janus/janus.cfg to enable the admin:

[general]
token_auth = True
admin_secret = V3ryS3cr3t
server_name = webrtc-proxy

And use valid certificates for DTLS (used to exchange key material inside the RTP session):

[certificates]
cert_pem = /etc/ssl/private/webrtc-proxy.example.com.pem
cert_key = /etc/ssl/private/webrtc-proxy.example.com.key

If you need IPv6 support, enable this as well:

[media]
ipv6 = yes

To create tokens, Mattermost needs to access the admin endpoint on the proxy. This endpoint can be reached over HTTP and HTTPS (from Mattermost). In this configuration the admin HTTP and the HTTP and HTTPS proxying are disabled. The configuration file for the HTTP transport is /etc/janus/janus.transport.http.cfg.

[general]
http = no
https = no

[admin]
admin_http = no
admin_base_path = /admin
admin_https = yes
admin_secure_port = 7889

[certificates]
cert_pem = /etc/ssl/private/webrtc-proxy.example.com.pem
cert_key = /etc/ssl/private/webrtc-proxy.example.com.key

The last bit of required configuration for Janus is the WebSockets transport. As the admin over WebSockets is not used by Mattermost, this is disabled. This configuration file is called /etc/janus/janus.transport.websockets.cfg.

[general]
ws = no
wss = yes
wss_port = 8989

[admin]
admin_ws = False
admin_wss = False

[certificates]
cert_pem = /etc/ssl/private/webrtc-proxy.example.com.pem
cert_key = /etc/ssl/private/webrtc-proxy.example.com.key

Now that Janus is configured, restart the service (systemctl restart janus.service) and move on to Mattermost.

Configuring Mattermost

Mattermost can be configured via the web interface (System Console > WebRTC (Beta)) or via the config.json file. See the WebRTC (Beta) documentation which settings in the web interface match the ones from the configuration file.

Some things to keep in mind:

• The GatewayAdminUrl’s port must match the the admin_secure_port set in /etc/janus/janus.transport.http.cfg
• The GatewayAdminUrl’s path must match the the admin_base_path set in /etc/janus/janus.transport.http.cfg
• The GatewayWebsocketUrl must use the wss:// scheme for WebRTC to work
• The GatewayWebsocketUrl’s port must match the wss_port from /etc/janus/janus.transport.websockets.cfg
• The GatewayAdminSecret must match the admin_secret from /etc/janus/janus.cfg

The WebrtcSettings should look like this after configuring:

  "WebrtcSettings": {
    "Enable": true,
    "GatewayAdminSecret": "V3ryS3cr3t",
    "GatewayAdminUrl": "https://webrtc-proxy.example.com:7889/admin",
    "GatewayWebsocketUrl": "wss://webrtc-proxy.example.com:8989",
    "StunURI": "",
    "TurnSharedKey": "",
    "TurnURI": "",
    "TurnUsername": ""
  }

Mattermost should pick up the configuration changes by itself. If not, restart it.

Now the users can enable WebRTC for themselves (Account Settings > Advanced > Preview pre-release features > Enable the ability to make and receive one-on-one WebRTC calls) and start calling each other.

What about STUN and TURN?

In the testing we have done with the video calls, no NAT-punching was needed. Your mileage may vary, of course.

Bonus: Installing Janus on Debian Stretch

There is no Janus package available on Debian Stretch. As I like to run one distribution in my infrastructure, I had to backport Janus from Buster to Stretch.

To install Janus on Debian Stretch, add the following to /etc/apt/sources.list.d/janus.list:

deb https://repo.plexis.eu/debian stretch-janus main

And add the public key to the keyring:

curl -L https://repo.plexis.eu/6A8573EDDC4A8842.asc | sudo apt-key add -

Now install Janus:

sudo apt-get update
sudo apt-get install janus janus-tools

For this purpose, I wrote a small Nagios/Icinga script a few years ago that does just this, but never released it publicly. The code is up on GitHub for your downloading pleasure.

For those who don’t know, dnsdist is a DNS firewall/loadbalancer that is fully configured in Lua and provides (among many things) live DNS traffic inspection without slowing down the server. It also has a minimal memory-footprint.

In true daredevil fashion, there was a live-demo involved. Below I’ve reproduced the dnsdist config:

controlSocket('0.0.0.0') -- for the console
setKey('MXNeLFWHUe4363BBKrY06cAsH8NWNb+Se2eXU5+Bb74=') -- for crypto
webserver("127.0.0.1:8080", "geheim2") -- Enable the webserver for live-graphs and overview
carbonServer('37.252.122.50', 'lieter-demo', 10) -- Send stats to the public PowerDNS metronome (https://metrics.powerdns.com)

-- Add 11 listen sockets so we can have multiple clients connecting
addLocal('127.0.0.1:5300')
for x in pairs({1,2,3,4,5,6,7,8,9,10}) do
  addLocal('10.0.0.' .. x .. ':5300')
end

-- Default pool with 3 recursors
newServer({address='127.0.0.1:5301', name='backend1'})
newServer({address='127.0.0.1:5302', name='backend2'})
newServer({address='127.0.0.1:5303', name='backend3'})

-- One abuse recursor
newServer({address='127.0.0.1:5304', name='backend4', pool='abuse'})

function maintenance()
  addresses=exceedQTypeRate(dnsdist.TLSA, 2, 10) -- Get all addresses that asked for more than 2TLSA records in the last 10 seconds
  addDynBlocks(addresses,"Exceeded TLSA",60) -- Block them for 60 seconds
end

And here are some of the commands entered into the dnsdist console:

-- Block all .ru domains
addDomainBlock("ru.")

-- Create a netmaskgroup and add a single address to it
nmg = newNMG()
nmg:addMask('127.0.0.1/32')

-- Match on the netmaskgroup and QTYPE AAAA
selector = AndRule({NetmaskGroupRule(nmg), QTypeRule(dnsdist.AAAA)})

-- Delay these queries by 1500 millisecond
addAction(selector, DelayAction(1500))

-- Allow a maximum of 3 queries per second to powerdns.com
addQPSLimit('powerdns.com', 3)

-- Sens add traffic for google.com (and everything below it) to the abue pool
addPoolRule({'google.com'}, 'abuse')

-- Add a packet cache of 10000 entries to the default pool
pc = newPacketCache(10000, 86400, 0, 60, 60)
getPool(""):setCache(pc)

What struck me as really awesome, was that during the demo I was processing around 10000 queries per second through dnsdist. While this was going on, memory-usage did not exceed 30 Megabytes and (even with the recursors doing all the heavy lifting) my CPU was only used for 20% tops.

Cron only has a one-minute accuracy, so that won’t help in this case. Fortunately, systemd has the possiblity to run services at configured times or with a monotonic interval using timer units.

[Unit]
Description=A job that stores the time

[Service]
Type=oneshot
ExecStart=/usr/local/bin/foo.sh

/etc/systemd/system/foo.service

[Unit]
Description=Runs every 15 secs

[Timer]
OnActiveSec=0 # Run when activated
OnUnitActiveSec=15 # Run every 15 seconds
AccuracySec=500msec # Don't delay too much

[Install]
WantedBy = multiuser.target

/etc/systemd/system/foo.timer

#!/bin/sh
date >> /tmp/timestamp

/usr/local/bin/foo.sh

Enable and start the timer: systemctl enable --now foo.timer and lo and behold:

Wed Jun 15 16:26:44 CEST 2016
Wed Jun 15 16:27:00 CEST 2016
Wed Jun 15 16:27:15 CEST 2016
Wed Jun 15 16:27:31 CEST 2016
Wed Jun 15 16:27:46 CEST 2016
Wed Jun 15 16:28:02 CEST 2016

/tmp/timestamp

Note that the timestamps are not exactly 15 seconds apart. This is because AccuracySec is set to half a second. Systemd attempts to limit the number of wake ups and to schedule timers simultaniously. For a more precise interval, AccuracySec can be set to 1us.

An advantage of this approach (compared to to cron) is the fact that the stdout/stderr output of foo.service goes directly to the journal. This allows foo.sh (or whatever script used) to be extremely simple.

The Arch Linux wiki on systemd timers has a good section on using systemd timers as a replacement for cron, including sending emails on job failure.

At PowerDNS, we had a demonstration setup that was rebuilt several times in the span of two weeks, and I got tired of editing the zonefile* by hand, so let’s automate the hell out of it.

Fortunately, Digital Ocean has an API that allows you to retrieve Droplet (VPS) information. This API has bindings in several languages, so it’s easy to generate these DNS-entries.

Here’s the quick ‘n dirty python script I whipped up:

#!/usr/bin/env python2
import sys
import time
import os

try:
    import json
except ImportError:
    import simplejson as json

try:
    from dopy.manager import DoError, DoManager
except ImportError as e:
    print("dopy library required for this script")
    sys.exit(1)

conf = {}
################################################################################
# Change these settings
################################################################################
conf['APIkey'] = os.environ.get('API_KEY', 'changeme')
conf['zone'] = os.environ.get('ZONE', 'thisiswrong.example')
conf['ns'] = ['ns1.%s' % conf.get('zone'), 'ns2.%s' % conf.get('zone')]
conf['SOA'] = "%s. hostmaster.%s. %s 360 60 8640 360" % (conf.get('ns')[0], conf.get('zone'), int(time.time()))

################################################################################
# Begin script
################################################################################
manager = DoManager(None, conf.get('APIkey'), api_version=2)
droplets = manager.all_active_droplets()

sys.stdout.write("$ORIGIN %s.\n" % conf.get('zone'))
sys.stdout.write("$TTL 120\n")
sys.stdout.write("\n")
sys.stdout.write("@\tIN\tSOA\t%s\n" % conf.get('SOA'))
for ns in conf.get('ns'):
    sys.stdout.write("@\tIN\tNS\t%s.\n" % ns)
sys.stdout.write("\n")

for droplet in droplets:
    if not droplet['name'].endswith('.%s' % conf.get('zone')):
        continue

    name = droplet['name'].split('.')[0]

    for net in droplet['networks']['v4']:
        if net['type'] == 'public':
            sys.stdout.write("%s\tIN\tA\t%s\n" % (name, net['ip_address']))

    for net in droplet['networks']['v6']:
        if net['type'] == 'public':
            sys.stdout.write("%s\tIN\tAAAA\t%s\n" % (name, net['ip_address']))

You’ll need the dopy and six(required by dopy) python modules installed.

The easiest way to generate the zone is through cron:

*/5 * * * * ZONE=myzone.example; API_KEY=s3cr1t; /usr/local/bin/do-zone-generator.py > /tmp/$ZONE.zone && cp /tmp/$ZONE.zone /var/lib/powerdns/zones && pdns_control bind-reload-now $ZONE >/dev/null

This will generate the zone, copy it to the right place and tell PowerDNS to reload the zone (sending out NOTIFYs).

Of course, the script could be extended to insert the data into a database-backed PowerDNS using the PowerDNS API. But this is left as an exercise to the reader.

*: As a sidenote, even though the PowerDNS Authoritative Server is deployed most of the time because of its excellent database backends, for my own small-scale deployments, I prefer zone-files over a database. But this might change, as pdnsutil supports an edit-zone command that will create a faux zonefile from the zone in the database, open it in $EDITOR and will insert it into the database after editing. See Bert Hubert’s blogpost on this.