JIT WireGuard · The Fly Weblog

Considered one of many odd choices we’ve made at Fly.io is how we use WireGuard. It’s not simply that we use it in lots of locations the place different retailers would use HTTPS and REST APIs. We’ve gone a step past that: each time you run flyctl, our lovable, sprawling CLI, it conjures a TCP/IP stack out of skinny air, with its personal IPv6 deal with, and speaks on to Fly Machines working on our networks.

There are plusses and minuses to this method, which we talked about in a blog post a couple years back. Some issues, like remote-operated Docker builders, get simpler to precise (a Fly Machine, so far as flyctl is anxious, may as effectively be on the identical LAN). However every thing typically will get trickier to maintain working reliably.

It was a call. We personal it.

Anyhow, we’ve made some enhancements just lately, and I’d like to speak about them.

Where we left off

I copy-pasted those last two bullet points from that two-year-old post, as a result of when it really works, it does simply work fairly effectively. (We finally did find yourself defaulting everyone to WireGuard-over-WebSockets, although.)

But when it all the time labored, we wouldn’t be right here, would we?

We bumped into two annoying issues:

One: NATS is quick, however doesn’t assure supply. Again in 2022, Fly.io was fairly massive on NATS internally. We’ve moved away from it. For example, our internal flyd API was once pushed by NATS; right now, it’s HTTP. Our NATS cluster was shedding too many messages to host a dependable API on it. Scaling again our use of NATS made WireGuard gateways higher, however nonetheless not nice.

Two: When flyctl exits, the WireGuard peer it created sticks round on the gateway. Nothing cleans up previous friends. In any case, you’re doubtless going to return again tomorrow and deploy a brand new model of your app, or fly ssh console into it to debug one thing. Why take away a peer simply to re-add it the following day?

Sadly, the overwhelming majority of friends are created by flyctl in CI jobs, which don’t have persistent storage and might’t reconnect to the identical peer the following run; they generate new friends each time, it doesn’t matter what.

So, we ended up with a not-reliable-enough provisioning system, and gateways with a whole lot of hundreds of friends that may by no means be used once more. The excessive stale peer rely made kernel WireGuard operations very sluggish – particularly loading all of the friends again into the kernel after a gateway server reboot – in addition to some kernel panics.

There needed to be

A better way.

It is possible to JIT WireGuard peers

The Linux kernel’s interface for configuring WireGuard is Netlink (which is mainly a method to create a userland socket to speak to a kernel service). Right here’s a summary of it as a C API. Word that there’s no API name to subscribe for “incoming connection try” occasions.

That’s OK! We will simply make our personal occasions. WireGuard connection requests are packets, they usually’re simply identifiable, so we will effectively snatch them with a BPF filter and a packet socket.

We personal the daemon code for that, and might simply hook the packet obtain operate to snarf WireGuard packets.

It’s not apparent, however WireGuard doesn’t have notions of “consumer” or “server”. It’s a pure point-to-point protocol; friends join to one another after they have visitors to ship. The primary peer to attach is named the initiator, and the peer it connects to is the responder.

For Fly.io, flyctl is often our initiator, sending a single UDP packet to the gateway, which is the responder. In accordance to the WireGuard paper, this primary packet is a handshake initiation. It will get higher: the packet kind is recorded in a single plaintext byte. So this easy BPF filter catches all of the incoming connections: udp and dst port 51820 and udp[8] = 1.

In most different protocols, we’d be completed at this level; we’d simply scrape the username or whatnot out of the packet, go fetch the matching configuration, and set up it within the kernel. With WireGuard, not so quick. WireGuard is predicated on Trevor Perrin’s Noise Protocol Framework, and Noise goes manner out of its method to hide identities throughout handshakes. To establish incoming requests, we’ll have to run sufficient Noise cryptography to decrypt the identification.

The code to do that is fussy, however it’s comparatively brief (about 200 traces). Helpfully, the kernel Netlink interface will give a privileged course of the non-public key for an interface, so the secrets and techniques we have to unwrap WireGuard are simple to get. Then it’s only a matter of working the primary little bit of the Noise handshake. When you’re that type of nerdy, here’s the code.

At this level, we have now the occasion feed we wished: the general public keys of each consumer making an attempt to make a WireGuard connection to our gateways. We maintain a rate-limited cache in SQLite, and once we see new friends, we’ll make an inside HTTP API request to fetch the matching peer data and set up it. This matches properly into the little daemon that already runs on our gateways to handle WireGuard, and permits us to ruthlessly and recklessly take away stale friends with a cron job.

See Also

A chemical sign in human feminine tears lowers aggression in males

However wait! There’s extra! We bounced this plan off Jason Donenfeld, and he tipped us off on a sneaky characteristic of the Linux WireGuard Netlink interface.

Our API fetch for brand spanking new friends is usually not going to be quick sufficient to reply to the primary handshake initiation message a brand new consumer sends us. That’s OK; WireGuard is fairly quick about retrying. However we will do higher.

Once we get an incoming initiation message, we have now the 4-tuple deal with of the specified connection, together with the ephemeral supply port flyctl is utilizing. We will set up the peer as if we’re the initiator, and flyctl is the responder. The Linux kernel will provoke a WireGuard connection again to flyctl. This works; the protocol doesn’t care an entire lot who’s the server and who’s the consumer. We get new connections established about as quick as they will presumably be put in.

Look at this graph