Development discussion of WireGuard
 help / color / mirror / Atom feed
* eBPF + IPv6 + WireGuard
@ 2021-12-18  0:06 Alex
  2021-12-20 21:49 ` Mike O'Connor
  2021-12-23 15:25 ` Toke Høiland-Jørgensen
  0 siblings, 2 replies; 3+ messages in thread
From: Alex @ 2021-12-18  0:06 UTC (permalink / raw)
  To: wireguard

Hi all,

I am championing WireGuard at work, and I have been granted permission
to use it for establishing remote access to a private IPv6 VLAN for all
employees. I have experimented with different approaches and ran in to
an issue.

With the help of a machine dedicated fully to the job of remote access
(Ubuntu 20.04 / Linux 5.4), I've managed to establish end-to-end
connectivity between my work laptop and the servers. The VPN gateway
has a wg0 interface for employees and a "private0" interface for
private VLAN connectivity. The goal is to link the two.

I noticed an odd quirk: It only works when
"net.ipv6.conf.all.forwarding" is 1. If this value is 0,
"net.ipv6.conf.[wg0 | private0].forwarding" has no effect. In other
words, I cannot seem to enable forwarding for *only* the interfaces
that need it. It only works when forwarding is enabled for *all*
interfaces. This is a problem because when the "all" value is set to 1,
the machine will start to behave as a router on VLANs for which it is
most definitely not a router.

For the servers, I am using the officially defined[0] subnet-router
anycast address as the default IPv6 gateway, and it works well.
However, when I flipped the switch on "net.ipv6.conf.all.forwarding",
the VPN gateway started using NDP to announce that it was a router
across *all* VLANs! Specifically, it was claiming ownership over our
global subnet anycast address 2001:aaaa:bbbb:cccc::. This is neither
true nor desired. The VPN gateway started to receive outbound non-VPN
Internet traffic which broke Internet connectivity for all servers. 

This leads me to my first question: Why does
"net.ipv6.conf.wg0.forwarding" have no effect?

In researching a solution, I decided to give eBPF a try. I attached an
XDP program to "private0" (a layer 2 device, naturally) and
successfully redirected packets to "wg0" (a layer 3 device) with
bpf_redirect[1]. If the packets are unmodified, WireGuard will happily
pass them to the remote hosts. This is an issue because the remote
hosts are expecting to receive IP(v6) packets, not Ethernet frames. If
I use bpf_xdp_adjust_head[1] to strip off the Ethernet frame, WireGuard
will drop the packet[2] before it can be sent to the remote host.

My theory is that bpf_xdp_adjust_head is modifying the data pointer
only and not any underlying structures that may be associated with the
packet (sk_buff perhaps).

This leads me to my second question: Why can't I redirect traffic
received on an L2 interface to an L3 interface, *even after stripping
off the Ethernet frame?*

Finally, during this whole process I was using WireShark to inspect
traffic received by the remote host (i.e. my work laptop). With the
help of the extract-handshakes.sh script, I was able to decrypt traffic.
I did discover a bug though. I believe "index_hashtable_insert" should
actually be "wg_index_hashtable_insert"[3].

Does anyone have any insights as to what a proper solution would look
like? Is there a way to achieve my goal without introducing eBPF? Is
XDP completely unsuited for this particular purpose? Do I actually need
to operate on the sk_buff level, as opposed to the xdp_buff level?

Thank you all for your time.

 - Alex

[0] https://datatracker.ietf.org/doc/rfc1884/ (Section 2.5.1)
[1] https://www.man7.org/linux/man-pages/man7/bpf-helpers.7.html
[2] https://git.zx2c4.com/wireguard-linux/tree/drivers/net/wireguard/device.c#n131
[3] https://git.zx2c4.com/wireguard-tools/tree/contrib/extract-handshakes/extract-handshakes.sh#n46

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: eBPF + IPv6 + WireGuard
  2021-12-18  0:06 eBPF + IPv6 + WireGuard Alex
@ 2021-12-20 21:49 ` Mike O'Connor
  2021-12-23 15:25 ` Toke Høiland-Jørgensen
  1 sibling, 0 replies; 3+ messages in thread
From: Mike O'Connor @ 2021-12-20 21:49 UTC (permalink / raw)
  To: Alex; +Cc: wireguard

Hi

Wireguard is a layer 3 system. As a result you must route traffic not bridge, this also means that ip forwarding must be enabled.

You will need to firewall the traffic or setup a separate routing table. 

Other routers in your network will need to know about the IP addresses allocated to Wireguard and have a route for this range.

Mike

> On 21 Dec 2021, at 3:21 am, Alex <wireguard@centromere.net> wrote:
> 
> Hi all,
> 
> I am championing WireGuard at work, and I have been granted permission
> to use it for establishing remote access to a private IPv6 VLAN for all
> employees. I have experimented with different approaches and ran in to
> an issue.
> 
> With the help of a machine dedicated fully to the job of remote access
> (Ubuntu 20.04 / Linux 5.4), I've managed to establish end-to-end
> connectivity between my work laptop and the servers. The VPN gateway
> has a wg0 interface for employees and a "private0" interface for
> private VLAN connectivity. The goal is to link the two.
> 
> I noticed an odd quirk: It only works when
> "net.ipv6.conf.all.forwarding" is 1. If this value is 0,
> "net.ipv6.conf.[wg0 | private0].forwarding" has no effect. In other
> words, I cannot seem to enable forwarding for *only* the interfaces
> that need it. It only works when forwarding is enabled for *all*
> interfaces. This is a problem because when the "all" value is set to 1,
> the machine will start to behave as a router on VLANs for which it is
> most definitely not a router.
> 
> For the servers, I am using the officially defined[0] subnet-router
> anycast address as the default IPv6 gateway, and it works well.
> However, when I flipped the switch on "net.ipv6.conf.all.forwarding",
> the VPN gateway started using NDP to announce that it was a router
> across *all* VLANs! Specifically, it was claiming ownership over our
> global subnet anycast address 2001:aaaa:bbbb:cccc::. This is neither
> true nor desired. The VPN gateway started to receive outbound non-VPN
> Internet traffic which broke Internet connectivity for all servers. 
> 
> This leads me to my first question: Why does
> "net.ipv6.conf.wg0.forwarding" have no effect?
> 
> In researching a solution, I decided to give eBPF a try. I attached an
> XDP program to "private0" (a layer 2 device, naturally) and
> successfully redirected packets to "wg0" (a layer 3 device) with
> bpf_redirect[1]. If the packets are unmodified, WireGuard will happily
> pass them to the remote hosts. This is an issue because the remote
> hosts are expecting to receive IP(v6) packets, not Ethernet frames. If
> I use bpf_xdp_adjust_head[1] to strip off the Ethernet frame, WireGuard
> will drop the packet[2] before it can be sent to the remote host.
> 
> My theory is that bpf_xdp_adjust_head is modifying the data pointer
> only and not any underlying structures that may be associated with the
> packet (sk_buff perhaps).
> 
> This leads me to my second question: Why can't I redirect traffic
> received on an L2 interface to an L3 interface, *even after stripping
> off the Ethernet frame?*
> 
> Finally, during this whole process I was using WireShark to inspect
> traffic received by the remote host (i.e. my work laptop). With the
> help of the extract-handshakes.sh script, I was able to decrypt traffic.
> I did discover a bug though. I believe "index_hashtable_insert" should
> actually be "wg_index_hashtable_insert"[3].
> 
> Does anyone have any insights as to what a proper solution would look
> like? Is there a way to achieve my goal without introducing eBPF? Is
> XDP completely unsuited for this particular purpose? Do I actually need
> to operate on the sk_buff level, as opposed to the xdp_buff level?
> 
> Thank you all for your time.
> 
> - Alex
> 
> [0] https://datatracker.ietf.org/doc/rfc1884/ (Section 2.5.1)
> [1] https://www.man7.org/linux/man-pages/man7/bpf-helpers.7.html
> [2] https://git.zx2c4.com/wireguard-linux/tree/drivers/net/wireguard/device.c#n131
> [3] https://git.zx2c4.com/wireguard-tools/tree/contrib/extract-handshakes/extract-handshakes.sh#n46


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: eBPF + IPv6 + WireGuard
  2021-12-18  0:06 eBPF + IPv6 + WireGuard Alex
  2021-12-20 21:49 ` Mike O'Connor
@ 2021-12-23 15:25 ` Toke Høiland-Jørgensen
  1 sibling, 0 replies; 3+ messages in thread
From: Toke Høiland-Jørgensen @ 2021-12-23 15:25 UTC (permalink / raw)
  To: Alex, wireguard

Alex <wireguard@centromere.net> writes:

> Hi all,
>
> I am championing WireGuard at work, and I have been granted permission
> to use it for establishing remote access to a private IPv6 VLAN for all
> employees. I have experimented with different approaches and ran in to
> an issue.
>
> With the help of a machine dedicated fully to the job of remote access
> (Ubuntu 20.04 / Linux 5.4), I've managed to establish end-to-end
> connectivity between my work laptop and the servers. The VPN gateway
> has a wg0 interface for employees and a "private0" interface for
> private VLAN connectivity. The goal is to link the two.
>
> I noticed an odd quirk: It only works when
> "net.ipv6.conf.all.forwarding" is 1. If this value is 0,
> "net.ipv6.conf.[wg0 | private0].forwarding" has no effect. In other
> words, I cannot seem to enable forwarding for *only* the interfaces
> that need it. It only works when forwarding is enabled for *all*
> interfaces. This is a problem because when the "all" value is set to 1,
> the machine will start to behave as a router on VLANs for which it is
> most definitely not a router.
>
> For the servers, I am using the officially defined[0] subnet-router
> anycast address as the default IPv6 gateway, and it works well.
> However, when I flipped the switch on "net.ipv6.conf.all.forwarding",
> the VPN gateway started using NDP to announce that it was a router
> across *all* VLANs! Specifically, it was claiming ownership over our
> global subnet anycast address 2001:aaaa:bbbb:cccc::. This is neither
> true nor desired. The VPN gateway started to receive outbound non-VPN
> Internet traffic which broke Internet connectivity for all servers. 
>
> This leads me to my first question: Why does
> "net.ipv6.conf.wg0.forwarding" have no effect?

That's how forwarding works in the kernel for IPv6:

https://elixir.bootlin.com/linux/latest/source/Documentation/networking/ip-sysctl.rst#L1981

I.e., there *is* no per-interface forwarding setting, you'll need to
setup netfilter rules or some other mechanism to control which
interfaces packets will be forwarded to/from. The per-interface
"forwarding" attribute only controls things like whether the kernel will
accept router advertisements on that interface. Yes, this is a bit
confusing (and also different from IPv4).

The isRouter flag in neighbour advertisements should be controllable via
the per-interface 'forwarding' setting; the default changes when you
enable the global setting, but I think it should be possible to re-set
it? See:

https://elixir.bootlin.com/linux/latest/source/Documentation/networking/ip-sysctl.rst#L2148

> In researching a solution, I decided to give eBPF a try. I attached an
> XDP program to "private0" (a layer 2 device, naturally) and
> successfully redirected packets to "wg0" (a layer 3 device) with
> bpf_redirect[1]. If the packets are unmodified, WireGuard will happily
> pass them to the remote hosts. This is an issue because the remote
> hosts are expecting to receive IP(v6) packets, not Ethernet frames. If
> I use bpf_xdp_adjust_head[1] to strip off the Ethernet frame, WireGuard
> will drop the packet[2] before it can be sent to the remote host.
>
> My theory is that bpf_xdp_adjust_head is modifying the data pointer
> only and not any underlying structures that may be associated with the
> packet (sk_buff perhaps).
>
> This leads me to my second question: Why can't I redirect traffic
> received on an L2 interface to an L3 interface, *even after stripping
> off the Ethernet frame?*

I assume you were using generic XDP here? Anyway, XDP is layer2-only by
definition, so there's not really any way to achieve what you're trying
to do with XDP.

> Finally, during this whole process I was using WireShark to inspect
> traffic received by the remote host (i.e. my work laptop). With the
> help of the extract-handshakes.sh script, I was able to decrypt traffic.
> I did discover a bug though. I believe "index_hashtable_insert" should
> actually be "wg_index_hashtable_insert"[3].
>
> Does anyone have any insights as to what a proper solution would look
> like? Is there a way to achieve my goal without introducing eBPF? Is
> XDP completely unsuited for this particular purpose? Do I actually need
> to operate on the sk_buff level, as opposed to the xdp_buff level?

You could maybe do something with TC-BPF, but why can't you just enable
net.ipv6.conf.all.forwarding and just use netfilter to control which
interfaces packets will be forwarded across? That seems like it would be
the simplest solution...

-Toke

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2022-01-04 18:24 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-12-18  0:06 eBPF + IPv6 + WireGuard Alex
2021-12-20 21:49 ` Mike O'Connor
2021-12-23 15:25 ` Toke Høiland-Jørgensen

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).