Wireguard wg-quick defaults conflict with Kubernetes firewall

[Sebastian Rose <sebastian.rose@mailbox.org>]

Hello,

wg-quick in its default configuration causes routing conflicts when the same host is also running a kubernetes
master node. The issue seems to be how wg-quick marks the traffic to route to the Wireguard peer: 
https://www.wireguard.com/netns/#routing-all-your-traffic
This leads to a loss of network connectivity when Kubernetes and Wireguard are running simultaneously on the
same host.

wg-quick (when not instructed otherwise) generates an wg0 interface like this:

> interface: wg0
>   public key: [REMOVED]
>   private key: (hidden)
>   listening port: 37827
>   fwmark: 0xca6c
> 
> peer: [REMOVED]
>   endpoint: [IPV4]:51820
>   allowed ips: 0.0.0.0/0, ::/0
>   latest handshake: 1 minute, 21 seconds ago
>   transfer: 161.54 KiB received, 185.11 KiB sent

Additionally, the following routing rule gets created
> rule 32765:	not from all fwmark 0xca6c lookup 51820

> # ip route show table 51820
> default dev wg0 scope link

This got created with the following configuration:

> # wg showconf wg0
> [Interface]
> ListenPort = 36768
> FwMark = 0xca6c
> PrivateKey = [REMOVED]
> 
> [Peer]
> PublicKey = [REMOVED]
> AllowedIPs = 0.0.0.0/0, ::/0
> Endpoint = [IPV4]:51820

As part of a kubernetes installation, the following firewall rules get added:

> # iptables -L KUBE-FIREWALL
> Chain KUBE-FIREWALL (2 references)
> target     prot opt source               destination         
> DROP       all  -- !127.0.0.0/8          127.0.0.0/8          /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT
> DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

As far as I can tell, Wireguard marks its traffic with 0xca6c and Kubernetes wants to drop traffic that's
marked 0x8000. Since 0xca6c ∧ 0x8000 = 0x8000 all Wireguard traffic gets dropped and there is no network
connectivity.

Proposed solution:

Change FwMark in the [Interface] section of the wg0.conf to something like

> FwMark = 0x4a6c

This will cause the kubernetes firewall not to conflict anymore with the defaults of wg-quick.

Best,
Sebastian