Improving Wireguard Throughput

[David Bell <david@dbell.dev>]

Hey all,

I'm investigating Wireguard as an alternative to mTLS in our
infrastructure and I'm trying to understand what's achievable in terms
of throughput and incremental cpu overhead, though at the moment I'm
focused on throughput.

We run 48 core machines with 25gbit NICs using the 5.10.113 kernel.
Benchmarking unencrypted traffic between hosts in the same dc achieves
line rate, and initially Wireguard was hitting ~2gbit. However, After
bumping up Wireguard's default MTU to 8920 I'm able to get to ~12gbit
throughput consistently over repeated benchmarks.

At this point I'm at a bit of a loss where to look for increasing
throughput. Independent results seem to be sparse for this kind of
workload, but looking at cilium's benchmark they were able to get
around 17.5gbit[0] with the following setup[1]. One thing I'm
wondering (probably incorrectly?) is whether we're being impacted by
Wireguard's lack of cpu affinity. Our NIC has 8 queues which are bound
to explicit cores, and my understanding with Wireguard is that
encrypt/decrypt handling is going to get splayed across all 48.

[0] https://docs.cilium.io/en/latest/operations/performance/benchmark/#wireguard-vs-ipsec
[1] https://docs.cilium.io/en/latest/operations/performance/benchmark/#test-hardware