[RFC] Replace WireGuard AllowedIPs with IP route attribute

[RFC] Replace WireGuard AllowedIPs with IP route attribute

[Daniel Gröber]

Hi wireguard, birds, and babelers,

tl;dr I want to add a new Linux route attribute (think "via $wgpeer") to
supplement wireguard's internal AllowedIPs logic for both routing and
source address filtering.

I've been pondering how to better integrate wireguard into dynamic routing
daemons, particularly BIRD and babeld. Essentially we want to be able to
dynamically add/remove AllowedIPs depending on current reachability and/or
link quality stats.

Looking at the wg netlink API I see two major efficiency/scalability
problems: 1) there is no way to be notified of changes in AllowedIPs made
by other processes meaning we have to do periodic scans and 2) a peer's
AllowedIPs set can only be replaced wholesale, not modified
incrementally. This is problematic as "someone" might, in the worst case,
want to install an entire internet routing table's worth of AllowedIPs and
the set will likely change frequently. FYI: The IPv4 table has ~1M entries
at present, yikes.

Assuming external AllowedIPs changes are infrequent occationally dumping
them all to keep a consistent view of the state shouldn't be too much of an
issue as long as the netlink interface is performant enoug, so I'm going to
concentrate on the add/remove API for now.

Instead of doing the obvious thing and adding a more efficient incremental
AllowedIPs netlink interface I figure why not just add a route attribute to
select a target wg peer on a device. That way we could not only save memory
(no separate AllowedIPs trie) but also simplify routing daemon
implementation considerably.

This would mirror how on ethernet you can have `dev eth0 via $router_ip`.
I'm still reviewing the net/ code to find the best way to do this, but I'm
thinking either a new RTA_WGPEER, like: `default dev wg0 via-wgpeer
$peer_pubkey` or perhaps re-using RTA_VIA and keying off a statically
configured AllowedIP addresses.

To start I'd make this an opt-in replacement for our usual AllowedIPs
logic, making sure to only activate it if any via* RTAs are active on a
particular device, but if it proves to work well I don't see why we
couldn't adapt the netlink code to maintain AllowedIPs using this RTA (but
invisible to userspace) to re-use the same code and get rid of allowedips.c
altogether. That's assuming this ends up being less code overall or perhaps
more performant.

Happy to hear your thoughts,
--Daniel

Re: [RFC] Replace WireGuard AllowedIPs with IP route attribute

[Daniel Gröber]

Hi Bernd,

On Sat, Aug 19, 2023 at 07:50:38PM +0200, Bernd Naumann wrote:
> Chances are high I do miss something, but I've just set AllowedIPs to
> 0.0.0.0/0 and ::/0 and just used the routing protocol of my choice and
> filters to select which routes got exported and imported... :shrug:

Right, let me expand a bit. You are absolutely right, right now if you want
to use wg with dynamic routing daemons you essentially have to have one wg
tunnel per remote node with AllowedIPs=::/0 and that works just fine at
small scales.

The idea here is that we would like to go back to having just one tunnel
for all nodes involved in this particular network instead, due to general
operations scalability, mine is a mesh network so the number of tunnels
gets rather large quickly :)

Lots of tunnels suck for various reasons, monitoring if they're all up and
configured properly is one example but my understanding from previous
discussions is the performance is probably not ideal either.

--Daniel

Re: [Babel-users] [RFC] Replace WireGuard AllowedIPs with IP route attribute

[Steffen Vogel]

Hi Daniel,

Interesting ideas! I am wondering if this complexity is really necessary?
How many routes do you have per peer? In my personal setup I have maximum of 1-100 routes per peer which I can handle with the current API quite comfortably.

My biggest concern about the introduction of a route attribute is that this adds complexity for users.
WireGuard's simplicity (and portability) have been important factors for its success.

A route attribute would introduce another source for the crypto-routing peer selection process.
What happens if the two mechanisms select different peers? Which one would have precedence?
Similarly also for incoming packets. WireGuard's current principle is really easy to understand. If the source address in in the peers AllowedIP list, we will accept the packet. If not its discarded. This is a central part of WireGuard's crypto-key routing feature which would become more complex.

Also implementation wise I would have doubts: Should WireGuard itself perform route lookups to determine which packets will be accepted? Or does WireGuard needs to synchronize the kernel routing table with its internal data structures itself?

A second concern I have with the use of route attributes is limited portability. Not all platforms support them.
How do we handle WireGuard userspace implementations?

I've tackled this problem in a userspace daemon. The synchronisation of a kernel routing table with a WireGuard AllowedIPs settings can be done by cunicu's route synchronization feature: https://cunicu.li/docs/features/rtsync

The route synchronization feature keeps the kernel routing table in sync with WireGuard's AllowedIPs setting.
This synchronization is bi-directional:

- Networks which are found in a Peers AllowedIP list will be installed as a kernel route
- Kernel routes with the peers unique IP address as next-hop will be added to the Peers AllowedIPs list.

This rather simple feature allows user to pair cunicu with a software routing daemon like Bird2 while using a single WireGuard interface with multiple peer-to-peer links.

I am assigning each WireGuard interface a link-local address which is derived from the peers public key.
I am using the peers link-local address as the next-hop in my routing daemon to differentiate to which Peer the AllowedIP entry must be added.

I am keeping track of the kernel's routing table and AllowedIPs by regularly polling the kernel.
As the route synchronisation is just one of cunicu's features [1], I have a central "watcher" routine in cunicu which observes any modification the the WireGuard interfaces and dispatches events which the individual features then can hook into. These observations are not limited to the AllowedIPs but basically any state of the WireGuard interface. E.g. last handshake time or per/peer traffic counters.

In my setup a periodic synchronization worked fine. But I agree that it would be nice if we could have a Netlink multicast group for subscribing to changes like we also have for other parts of the Linux network stack like routing tables, or link states. This feature was already discussed on the WireGuard mailing list [7]. But unfortunately the patch was never accepted. Maybe we can revisit this patch?

I would also be a big supported of extending the netlink API for supporting incremental updates the AllowedIP lists. The netlink APIs are already different for each platform. So extending it for one platform wouldn't hurt here.

Unfortunately, I have far too many ideas for cunicu and limited time to realize them all. So I've recently moved the whole cunicu project into its own organization at GitHub/Codeberg [6] in attempt to find more contributors.

Best regards,
Steffen (stv0g)

[1] Others planned features are:
- Endpoint discovery via ICE/STUN/TURN
- Peer discovery
- IP-autoconfiguration by deriving link-local addresses from peers public keys

I have a lot more ideas here like integrating my
- Go babel routing implementation [2]
- or Rosenpass PQC key-exchange [3]
- or performing proper path-MTU discovery using DPLPMTUD [4]
- or using hardware tokens, TPMs, secure enclaves to rotate pre-shared keys backed by a hardware source-of-trust [5]

[2] https://github.com/cunicu/go-babel
[3] https://github.com/cunicu/go-rosenpass
[4] https://github.com/cunicu/go-pmtud
[5] https://github.com/cunicu/go-skes
[6] https://codeberg.org/cunicu
[7] https://lists.zx2c4.com/pipermail/wireguard/2021-January/006318.html

On Saturday, August 19, 2023 16:02 CEST, Daniel Gröber <dxld@darkboxed.org> wrote:

> Hi wireguard, birds, and babelers,
> 
> tl;dr I want to add a new Linux route attribute (think "via $wgpeer") to
> supplement wireguard's internal AllowedIPs logic for both routing and
> source address filtering.
> 
> I've been pondering how to better integrate wireguard into dynamic routing
> daemons, particularly BIRD and babeld. Essentially we want to be able to
> dynamically add/remove AllowedIPs depending on current reachability and/or
> link quality stats.
> 
> Looking at the wg netlink API I see two major efficiency/scalability
> problems: 1) there is no way to be notified of changes in AllowedIPs made
> by other processes meaning we have to do periodic scans and 2) a peer's
> AllowedIPs set can only be replaced wholesale, not modified
> incrementally. This is problematic as "someone" might, in the worst case,
> want to install an entire internet routing table's worth of AllowedIPs and
> the set will likely change frequently. FYI: The IPv4 table has ~1M entries
> at present, yikes.
> 
> Assuming external AllowedIPs changes are infrequent occationally dumping
> them all to keep a consistent view of the state shouldn't be too much of an
> issue as long as the netlink interface is performant enoug, so I'm going to
> concentrate on the add/remove API for now.
> 
> Instead of doing the obvious thing and adding a more efficient incremental
> AllowedIPs netlink interface I figure why not just add a route attribute to
> select a target wg peer on a device. That way we could not only save memory
> (no separate AllowedIPs trie) but also simplify routing daemon
> implementation considerably.
> 
> This would mirror how on ethernet you can have `dev eth0 via $router_ip`.
> I'm still reviewing the net/ code to find the best way to do this, but I'm
> thinking either a new RTA_WGPEER, like: `default dev wg0 via-wgpeer
> $peer_pubkey` or perhaps re-using RTA_VIA and keying off a statically
> configured AllowedIP addresses.
> 
> To start I'd make this an opt-in replacement for our usual AllowedIPs
> logic, making sure to only activate it if any via* RTAs are active on a
> particular device, but if it proves to work well I don't see why we
> couldn't adapt the netlink code to maintain AllowedIPs using this RTA (but
> invisible to userspace) to re-use the same code and get rid of allowedips.c
> altogether. That's assuming this ends up being less code overall or perhaps
> more performant.
> 
> Happy to hear your thoughts,
> --Daniel
> 
> _______________________________________________
> Babel-users mailing list
> Babel-users@alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/babel-users

Re: [RFC] Replace WireGuard AllowedIPs with IP route attribute

[Kyle Rose]

Daniel,

I attempted several times to send the following message to the mailing
list nearly two months ago, but they all landed in moderation for some
reason and were silently dropped after a few days. You are not the
only one who wants this functionality for mesh VPNs. I note the email
you subsequently replied to (by Bernd) also didn't show up on the
list, suggesting I am not the only one whose messages are being
inexplicably jailed.

Kyle

====
I really like the straightforward configurability of Wireguard
out-of-the-box. It was astonishingly easy to configure a mesh to
replace my previous hub-and-spoke OpenVPN setup. Thank you for making
this easy.

That said, I'd like the ability to use Linux's policy routing engine
to allow for more complex packet flows across the VPN that are
currently incompatible with Wireguard's internal packet handling. For
example, let's say I have 4 nodes and want to be able to use each of
the nodes as the default gateway for different types of flows.
Modifying the sender side to respect a route's gateway is
straightforward:

--- a/drivers/net/wireguard/allowe
dips.c
+++ b/drivers/net/wireguard/allowe
dips.c
@@ -6,6 +6,8 @@
 #include "allowedips.h"
 #include "peer.h"

+#include <net/route.h>
+
 enum { MAX_ALLOWEDIPS_BITS = 128 };

 static struct kmem_cache *node_cache;
@@ -356,10 +358,18 @@ int wg_allowedips_read_node(struct
allowedips_node *node, u8 ip[16], u8 *cidr)
 struct wg_peer *wg_allowedips_lookup_dst(stru
ct allowedips *table,
                                         struct sk_buff *skb)
 {
-       if (skb->protocol == htons(ETH_P_IP))
-               return lookup(table->root4, 32, &ip_hdr(skb)->daddr);
-       else if (skb->protocol == htons(ETH_P_IPV6))
-               return lookup(table->root6, 128, &ipv6_hdr(skb)->daddr);
+       struct rtable *rt = skb_rtable(skb);
+       if (rt->rt_uses_gateway) {
+               if (rt->rt_gw_family == AF_INET)
+                       return lookup(table->root4, 32, &rt->rt_gw4);
+               else if (rt->rt_gw_family == AF_INET6)
+                       return lookup(table->root6, 128, &rt->rt_gw6);
+       } else {
+               if (skb->protocol == htons(ETH_P_IP))
+                       return lookup(table->root4, 32, &ip_hdr(skb)->daddr);
+               else if (skb->protocol == htons(ETH_P_IPV6))
+                       return lookup(table->root6, 128, &ipv6_hdr(skb)->daddr);
+       }
        return NULL;
 }

The problem is that reply packets will be dropped via the source
address check for all but the peer with the default route listed in
AllowedIPs. The way the trie code works means a highly-invasive change
would be needed to allow for multiple peers to be associated with a
given prefix: I suspect any further complication (not to mention
possible additional data structure bloat) is undesirable, and anyway I
am looking to bypass most of the complexity created by Wireguard's
parallel packet routing infrastructure and instead leverage the far
more flexible Linux policy routing engine.

At this point, what I'd like to do is be able to skip the source
address check by configuration, instead relying on rp_filter and
firewall rules to reject bogus or unwanted packets. With such a config
knob, AllowedIPs would be used only for selecting the destination peer
based on the packet daddr or the route's gateway. For a mesh like I
described above, I would configure only a gateway IP for each peer in
AllowedIPs and use the policy routing engine for all other packet
routing behavior.

I appreciate that Wireguard works the way it does most likely because
routing is configured differently across the wide variety of devices
it supports (and in some cases may be unavailable to users), so I
don't think the AllowedIPs source address check should be removed by
default; but it would be nice if I could turn it off and rely on other
mechanisms in the kernel that would allow for more flexibility.


On Sat, Aug 19, 2023 at 10:05 AM Daniel Gröber <dxld@darkboxed.org> wrote:
>
> Hi wireguard, birds, and babelers,
>
> tl;dr I want to add a new Linux route attribute (think "via $wgpeer") to
> supplement wireguard's internal AllowedIPs logic for both routing and
> source address filtering.
>
> I've been pondering how to better integrate wireguard into dynamic routing
> daemons, particularly BIRD and babeld. Essentially we want to be able to
> dynamically add/remove AllowedIPs depending on current reachability and/or
> link quality stats.
>
> Looking at the wg netlink API I see two major efficiency/scalability
> problems: 1) there is no way to be notified of changes in AllowedIPs made
> by other processes meaning we have to do periodic scans and 2) a peer's
> AllowedIPs set can only be replaced wholesale, not modified
> incrementally. This is problematic as "someone" might, in the worst case,
> want to install an entire internet routing table's worth of AllowedIPs and
> the set will likely change frequently. FYI: The IPv4 table has ~1M entries
> at present, yikes.
>
> Assuming external AllowedIPs changes are infrequent occationally dumping
> them all to keep a consistent view of the state shouldn't be too much of an
> issue as long as the netlink interface is performant enoug, so I'm going to
> concentrate on the add/remove API for now.
>
> Instead of doing the obvious thing and adding a more efficient incremental
> AllowedIPs netlink interface I figure why not just add a route attribute to
> select a target wg peer on a device. That way we could not only save memory
> (no separate AllowedIPs trie) but also simplify routing daemon
> implementation considerably.
>
> This would mirror how on ethernet you can have `dev eth0 via $router_ip`.
> I'm still reviewing the net/ code to find the best way to do this, but I'm
> thinking either a new RTA_WGPEER, like: `default dev wg0 via-wgpeer
> $peer_pubkey` or perhaps re-using RTA_VIA and keying off a statically
> configured AllowedIP addresses.
>
> To start I'd make this an opt-in replacement for our usual AllowedIPs
> logic, making sure to only activate it if any via* RTAs are active on a
> particular device, but if it proves to work well I don't see why we
> couldn't adapt the netlink code to maintain AllowedIPs using this RTA (but
> invisible to userspace) to re-use the same code and get rid of allowedips.c
> altogether. That's assuming this ends up being less code overall or perhaps
> more performant.
>
> Happy to hear your thoughts,
> --Daniel

Re: [Babel-users] [RFC] Replace WireGuard AllowedIPs with IP route attribute

[Daniel Gröber]

Hi Steffen,

from the questions you ask I fear you've misunderstood my intention to
"replace" AllowedIPs. I'm strictly talking of the _implementation_
(allowedips.c) in Linux particular. The netlink interface would naturally
stay unchanged.

On Sat, Aug 19, 2023 at 10:00:17PM +0200, Steffen Vogel wrote:
> Interesting ideas! I am wondering if this complexity is really necessary?

I think so, right now wg's behaviour just prevents a number of advanced
use-cases which is a pitty.

> My biggest concern about the introduction of a route attribute is that
> this adds complexity for users.  WireGuard's simplicity (and portability)
> have been important factors for its success.

Nothing would change for users that don't use this feature.

> A route attribute would introduce another source for the crypto-routing
> peer selection process.  What happens if the two mechanisms select
> different peers? Which one would have precedence?

If you read my previous mail carefully you'll find I specified how this
would shake out. The rt approach will necessarily have to override
AllowedIPs to be useful.

> Similarly also for incoming packets. WireGuard's current principle is
> really easy to understand. If the source address in in the peers
> AllowedIP list, we will accept the packet. If not its discarded. This is
> a central part of WireGuard's crypto-key routing feature which would
> become more complex.

Having read Kyle's use-case I'm thinking my original plan to extend the wg
internal source-address filtering to use a rt lookup with our new attribute
would not be maximally useful so now my thinking is we should just have a
boolean toggle to disable it explicitly per device.

Then users can do whatever fancy rt (reverse-path) lookups they want with
nft. I'm particularly happy that nft rt lookups will actually do the right
thing with multipath/ECMP routes (any of the involved devices satisfies the
lookup) so this should actually work out fine in my case at least. Mmore
prototyping is required here though.

> A second concern I have with the use of route attributes is limited
> portability. Not all platforms support them.  How do we handle WireGuard
> userspace implementations?

No need. The Linux's rt table is Linux specific I have no plans to
introduce this on other platforms, that's for other intersted souls to
tackle. Again "replace" was referring to implementation concerns.

> - Networks which are found in a Peers AllowedIP list will be installed as a kernel route

That's configurable. I always turn this off when dealing with rt daemons.

> - Kernel routes with the peers unique IP address as next-hop will be added to the Peers AllowedIPs list.
> 
> This rather simple feature allows user to pair cunicu with a software
> routing daemon like Bird2 while using a single WireGuard interface with
> multiple peer-to-peer links.

Sounds like you do what I want to do at kernel level in userspace, then at
least we can agree this is a useful thing :)

> In my setup a periodic synchronization worked fine. But I agree that it
> would be nice if we could have a Netlink multicast group for subscribing
> to changes like we also have for other parts of the Linux network stack
> like routing tables, or link states. This feature was already discussed
> on the WireGuard mailing list [7]. But unfortunately the patch was never
> accepted. Maybe we can revisit this patch?

Sounds like a plan, I'll have a look at it.

> [1] Others planned features are:
> - IP-autoconfiguration by deriving link-local addresses from peers public keys

That's been discussed so many times before on the ML and someone always
realises Jason is right and there's no point to this in the end. Key
distribution is the crux of the problem.

--Daniel

Re: [Babel-users] [RFC] Replace WireGuard AllowedIPs with IP route attribute

[Kyle Rose]

On Sat, Aug 19, 2023 at 5:25 PM Daniel Gröber <dxld@darkboxed.org> wrote:
> Having read Kyle's use-case I'm thinking my original plan to extend the wg
> internal source-address filtering to use a rt lookup with our new attribute
> would not be maximally useful so now my thinking is we should just have a
> boolean toggle to disable it explicitly per device.

If there is interest among the maintainers in eventually merging a
change with a per-interface knob to turn off the source IP check, I
will go through the trouble of putting together an initial pass at
this. I don't want to spend the time if there is firm opposition to
the idea.

Thanks,
Kyle

Re: [Babel-users] [RFC] Replace WireGuard AllowedIPs with IP route attribute

[Daniel Gröber]

Hi Kyle,

On Mon, Aug 28, 2023 at 11:40:48AM -0400, Kyle Rose wrote:
> On Sat, Aug 19, 2023 at 5:25 PM Daniel Gröber <dxld@darkboxed.org> wrote:
> > Having read Kyle's use-case I'm thinking my original plan to extend the wg
> > internal source-address filtering to use a rt lookup with our new attribute
> > would not be maximally useful so now my thinking is we should just have a
> > boolean toggle to disable it explicitly per device.
> 
> If there is interest among the maintainers in eventually merging a
> change with a per-interface knob to turn off the source IP check, I
> will go through the trouble of putting together an initial pass at
> this. I don't want to spend the time if there is firm opposition to
> the idea.

I think just a patch to turn off the wg source IP check is not very useful
at the moment. It would encourage bad source IP filtering practices when
multiple peers are involved as no mechanism for identifying the sending
peer is available at the policy routing or netfilter level currently.

I think such a patch would have to get merged after some kind of mechanism
to identify and filter based on the sending wg peer is available.

So if you want to move this along I would suggest working on this
first. Since I'm also interested in having this feature I'm happy
collaborate.

It's just hard to find the motivation for writing more wg patches when my
pending ones have (mostly) been lying around for a year without a response,
but if you're also keen on this feature I'm sure it's easier to stay
motivated together :)

If my kernel patches go ignored for too long too I'll probably just resort
to getting a forked DKMS wireguard module into Debian with this
work. Perhaps that approach (or a package in a different distro) would work
for your use-case too?

--Daniel

Re: [Babel-users] [RFC] Replace WireGuard AllowedIPs with IP route attribute

[Juliusz Chroboczek]

Daniel, Kyle,

I've read the whole discussion, and I'm still not clear what advantages
the proposed route attribute has over having one interface per peer.  Is
it because interfaces are expensive in the Linux kernel?  Or is there some
other reason why it is better to run all WG tunnels over a single interface?

-- Juliusz

Re: [Babel-users] [RFC] Replace WireGuard AllowedIPs with IP route attribute

[Kyle Rose]

On Mon, Aug 28, 2023 at 1:41 PM Juliusz Chroboczek <jch@irif.fr> wrote:
> I've read the whole discussion, and I'm still not clear what advantages
> the proposed route attribute has over having one interface per peer.  Is
> it because interfaces are expensive in the Linux kernel?  Or is there some
> other reason why it is better to run all WG tunnels over a single interface?

Why manage n^2 tunnels and allocate n^2 /30 CIDRs when you can just
have one tunnel and a single subnet for a full mesh? IMO, the latter
should be a feature differentiating Wireguard from other solutions to
creating a mesh VPN. That is, in fact, the whole reason I dropped
OpenVPN for it.

Kyle

Re: [RFC] Replace WireGuard AllowedIPs with IP route attribute

[Daniel Gröber]

[-- Attachment #1: Type: text/plain, Size: 1727 bytes --]

Hi Juliusz,

On Mon, Aug 28, 2023 at 07:40:51PM +0200, Juliusz Chroboczek wrote:
> I've read the whole discussion, and I'm still not clear what advantages
> the proposed route attribute has over having one interface per peer.  Is
> it because interfaces are expensive in the Linux kernel?  Or is there some
> other reason why it is better to run all WG tunnels over a single interface?

Off the top of my head UDP port exhaustion is a scalability concern here,
just as an example, not that I'd actually ever need that many peers in my
network :)

One wg-device per-peer means we need one UDP port per-peer and since
currently binding to a specific IP is also not supported by wg (I have a
patch pending for this though) there's no good way to work around this.

Frankly having tons of interfaces is just an operational PITA in all sorts
of ways. Apart from the port exhaustion having more than one wg device also
means I have to _allocate_ a new port for each node in my managment system
somehow instead of just using a static port for the entire network. This
gets dicy fast as I want to move in the direction of dynamic peering as in
tinc.

Other than that my `ip -br a` output is getting unmanagably long and having
more than one device means I have to keep ACL lists in sync all over the
system. This is a problem for daemons that don't support automatic reload
(babeld for example :P). I also have to sync the set of interface to
nftables which is easy to get wrong as it's still manual in my setup.

All of that could be solved, but I would also like to get my wg+babel VPN
setup deployed more widely at some point and all that friction isn't going
to help with that so I'd rather have this supported properly.

--Daniel

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC] Replace WireGuard AllowedIPs with IP route attribute

[Ivan Labáth]

Hi,

IMO, a good tunnel solution may be if what is now called AllowedIPs,
were functionally split into:
 - AcceptIPS (to be different from AllowedIPs)
 - RouteIPs
Perhaps with a default shorthand of, say, IPs, setting both, as
AllowedIPs historically caused confusion wrt. it's meaning.

Wireguard API is a bit clunky, but I think one could dynamically manage
routes in reasonably efficient ways without extra interfaces and layers.

Not sure if it would fullfill all reasonably achievable goals.

Don't really have the time to implement anything, and I'm sure it would
not be easy, so just a possible tip to consider.

Regards,
Ivan Labáth

On Tue, Aug 29, 2023 at 12:13:12AM +0200, Daniel Gröber wrote:
> Hi Juliusz,
> 
> On Mon, Aug 28, 2023 at 07:40:51PM +0200, Juliusz Chroboczek wrote:
> > I've read the whole discussion, and I'm still not clear what advantages
> > the proposed route attribute has over having one interface per peer.  Is
> > it because interfaces are expensive in the Linux kernel?  Or is there some
> > other reason why it is better to run all WG tunnels over a single interface?
> 
> Off the top of my head UDP port exhaustion is a scalability concern here,
> just as an example, not that I'd actually ever need that many peers in my
> network :)
> 
> One wg-device per-peer means we need one UDP port per-peer and since
> currently binding to a specific IP is also not supported by wg (I have a
> patch pending for this though) there's no good way to work around this.
> 
> Frankly having tons of interfaces is just an operational PITA in all sorts
> of ways. Apart from the port exhaustion having more than one wg device also
> means I have to _allocate_ a new port for each node in my managment system
> somehow instead of just using a static port for the entire network. This
> gets dicy fast as I want to move in the direction of dynamic peering as in
> tinc.
> 
> Other than that my `ip -br a` output is getting unmanagably long and having
> more than one device means I have to keep ACL lists in sync all over the
> system. This is a problem for daemons that don't support automatic reload
> (babeld for example :P). I also have to sync the set of interface to
> nftables which is easy to get wrong as it's still manual in my setup.
> 
> All of that could be solved, but I would also like to get my wg+babel VPN
> setup deployed more widely at some point and all that friction isn't going
> to help with that so I'd rather have this supported properly.
> 
> --Daniel

Re: [RFC] Replace WireGuard AllowedIPs with IP route attribute

[Daniel Gröber]

Hi Ivan,

> IMO, a good tunnel solution may be if what is now called AllowedIPs,
> were functionally split into:
>  - AcceptIPS (to be different from AllowedIPs)
>  - RouteIPs
> Perhaps with a default shorthand of, say, IPs, setting both, as
> AllowedIPs historically caused confusion wrt. it's meaning.

That would be one way to paint the shed, yes.

This alone doesn't really address the crux of the problem though:
scalability.

> Wireguard API is a bit clunky, but I think one could dynamically manage
> routes in reasonably efficient ways without extra interfaces and layers.

The entire idea with the new route attribute is to put this functionality
into the right (pre-existing) layer and not invent a new way of expressing
this. We even get scalability for free. Win-Win.

--Daniel

PS: Your mail didn't reach my inbox for some reason, I randomly found it
while looking at the wg list archives. Consider configuring your mail
client to To/CC people you're replying to in order to better handle flaky
list servers.

Re: [RFC] Replace WireGuard AllowedIPs with IP route attribute

[Reto]

On Fri, Sep 29, 2023 at 03:12:57PM +0200, Daniel Gröber wrote:
> PS: Your mail didn't reach my inbox for some reason, I randomly found it
> while looking at the wg list archives. Consider configuring your mail
> client to To/CC people you're replying to in order to better handle flaky
> list servers.

Well...

Date: Sun, 3 Sep 2023 03:21:25 +0000
From: Ivan Labáth <labawi-wg@matrix-dream.net>
To: Daniel Gröber <dxld@darkboxed.org>
Cc: Juliusz Chroboczek <jch@irif.fr>, Kyle Rose <krose@krose.org>,
        bird-users@network.cz, babel-users@alioth-lists.debian.net, wireguard@lists.zx2c4.com
Subject: Re: [RFC] Replace WireGuard AllowedIPs with IP route attribute
Message-ID: <ZPP7tab5OqR39RcY@matrix-dream.net>

They did.

Cheers,
Reto

Re: [Babel-users] [RFC] Replace WireGuard AllowedIPs with IP route attribute

[Alexander Zubkov]

Hello all,

I heard recently about the lightweight tunnel infrastructure in Linux
kernel (ip route ... encap ...). And I think this might be helpful in
the context of this thread. Linux kernel allows already to add
encapsulation parameters to the route entry in its table. So you do
not need to create tunnel devices for that. And wireguard
encapsulation and destination might be added there too. But as I
understood the technology, it works only in one way (for outgoing
packets) and the decapsulation should be processed separately, for
example in case of VXLAN and MPLS they have their own tables.

Regards,
Alexander Zubkov
Qrator Labs


On Mon, Sep 11, 2023 at 5:46 PM Maria Matejka via Bird-users
<bird-users@network.cz> wrote:
>
> Hello!
>
> On 8/29/23 00:13, Daniel Gröber wrote:
>
> On Mon, Aug 28, 2023 at 07:40:51PM +0200, Juliusz Chroboczek wrote:
>
> I've read the whole discussion, and I'm still not clear what advantages
> the proposed route attribute has over having one interface per peer.  Is
> it because interfaces are expensive in the Linux kernel?  Or is there some
> other reason why it is better to run all WG tunnels over a single interface?
>
> Off the top of my head UDP port exhaustion is a scalability concern here,
>
> For enterprise setups, this very easily _can_ get a scalability concern fairly easily.
>
> One wg-device per-peer means we need one UDP port per-peer and since
> currently binding to a specific IP is also not supported by wg (I have a
> patch pending for this though) there's no good way to work around this.
>
> There is a theoretical frankenstein approach, running a virtual machine (maybe netns is enough) for each of the public IP address, and connect them by veth. You do not want to do this, but theoretically, it should work.
>
> Frankly having tons of interfaces is just an operational PITA in all sorts
> of ways. Apart from the port exhaustion having more than one wg device also
> means I have to _allocate_ a new port for each node in my managment system
> somehow instead of just using a static port for the entire network. This
> gets dicy fast as I want to move in the direction of dynamic peering as in
> tinc.
>
> Even with my 6 machines running in weird locations, it's a mess.
>
> All of that could be solved, but I would also like to get my wg+babel VPN
> setup deployed more widely at some point and all that friction isn't going
> to help with that so I'd rather have this supported properly.
>
> All in all, I would also like to see this setup deployed worldwide. If we could somehow help on the BIRD side, please let us know.
>
> Thank you for bringing this up.
>
> --
> Maria Matejka (she/her) | BIRD Team Leader | CZ.NIC, z.s.p.o.

Re: [Babel-users] [RFC] Replace WireGuard AllowedIPs with IP route attribute

[Daniel Gröber]

[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]

Hi Alexander,

On Thu, Nov 09, 2023 at 12:57:26PM +0100, Alexander Zubkov wrote:
> I heard recently about the lightweight tunnel infrastructure in Linux
> kernel (ip route ... encap ...). And I think this might be helpful in
> the context of this thread.

I hadn't seen that yet, thanks for pointing it out.

> Linux kernel allows already to add encapsulation parameters to the route
> entry in its table. So you do not need to create tunnel devices for
> that. And wireguard encapsulation and destination might be added there
> too.

Right, I think ultimately it's going to come down to either technical
constraints or in the absence of that, maintainer preference whether
via-wgpeer or "encap wg" is the way. The idea is very similar anyway.

> But as I understood the technology, it works only in one way (for
> outgoing packets) and the decapsulation should be processed separately,
> for example in case of VXLAN and MPLS they have their own tables.

That would be a problem as I specifically want to tie the source address
filtering to this too. I'll have a look at the internals (if and) when I
get around to starting work on this.

Thanks,
--Daniel

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Babel-users] [RFC] Replace WireGuard AllowedIPs with IP route attribute

[Juliusz Chroboczek]

> Is tying source address filtering to the routing table the right thing to do
> here? It seems to me that it would cause issues similar to those we see more
> generally with Unicast Reverse Path Filtering

Issues are caused by the kernel performing filtering that the routing
protocol is not aware of: it causes the routing daemon's routing table to
no longer match the effective forwarding table (the kernel's routing
table).  That's the reason why uRPF breaks most routing protocols, that's
the reason why we have trouble making Wireguard work with Babel, and also
the reason behind https://github.com/jech/babeld/issues/111.

Contrariwise, we can teach Babel to explicitly take into account the
kernel features that we're interested in using.  Thus, Babel could be
aware of the restrictions placed on a wireguard interface, and collaborate
with Wireguard so that the routing table and the forwarding table remain
congruent.  I haven't looked at the issue in detail, but I believe that
would be an interesting (short-term) research project, one that I would be
glad to collaborate with (but not necessarily lead, at least not right now).

For the specific case of source address filtering, Babel already has an
(implemented) extension to deal with source addresses, and I encourage you
to consider whether it can be used to deal with the issue at hand.  Please
see https://arxiv.org/pdf/1403.0445.pdf and RFC 9079.

-- Juliusz

Re: [Babel-users] [RFC] Replace WireGuard AllowedIPs with IP route attribute

[Daniel Gröber]

Hi Erin, Juliusz,

On Sat, Nov 18, 2023 at 11:21:57AM +0100, Erin Shepherd wrote:
> On Sat, 18 Nov 2023, at 03:19, Daniel Gröber wrote:
> > That would be a problem as I specifically want to tie the source address
> > filtering to this too. I'll have a look at the internals (if and) when I
> > get around to starting work on this.
> 
> Is tying source address filtering to the routing table the right thing to
> do here? It seems to me that it would cause issues similar to those we
> see more generally with Unicast Reverse Path Filtering

IMO not providing a way to do source address filtering at the routing level
was the original sin :)

There is certianly the multihoming challange to be overcome as traditional
BCP38 style filtering doesn't cut it in the general case. I have some ideas
on how to deal with this.

I've done some experiments and found that in Linux multi-nexthop routes
actually match reverse path lookups (using nftables "rt") for _any_ of the
source interfaces involved. I think this can be used to build RFC 3704
style Feasible Path Reverse Path Forwarding when the routing daemon
involved supports ECMP.

This experiment is what got me interested in having via-wgpeer in the
routing table in the first place, once we have that we can apply the above
idea not just at the interface level but at the wg peer level. Neat.

Can you think of a use-case where fpRPF isn't enough?

It's also noteworthy that once we have this support for via-wgpeer it'd be
possible to apply ip-rule policy to the filtering decision. Perhaps that
gives some additional power for more fun use-cases :)

On Sat, Nov 18, 2023 at 01:22:03PM +0100, Juliusz Chroboczek wrote:
> Issues are caused by the kernel performing filtering that the routing
> protocol is not aware of: it causes the routing daemon's routing table to
> no longer match the effective forwarding table (the kernel's routing
> table).  That's the reason why uRPF breaks most routing protocols, that's
> the reason why we have trouble making Wireguard work with Babel, and also
> the reason behind https://github.com/jech/babeld/issues/111.

Right on the money as always. This idea has been on my mind too.

> Contrariwise, we can teach Babel to explicitly take into account the
> kernel features that we're interested in using.  Thus, Babel could be
> aware of the restrictions placed on a wireguard interface, and collaborate
> with Wireguard so that the routing table and the forwarding table remain
> congruent.  I haven't looked at the issue in detail, but I believe that
> would be an interesting (short-term) research project, one that I would be
> glad to collaborate with (but not necessarily lead, at least not right now).

Sounds interesting do you have a funding source in mind?

> For the specific case of source address filtering, Babel already has an
> (implemented) extension to deal with source addresses, and I encourage you
> to consider whether it can be used to deal with the issue at hand.  Please
> see https://arxiv.org/pdf/1403.0445.pdf and RFC 9079.

I don't think I mentioned this to you yet, but I have another one of my
crazy ideas of doing something vaguely similar to BGP flowspec with
babel. Restricted to IP source/destination address, so no L4 stuff. I just
want to represent firewall policy using ipv6 subtrees and distribute it in
realtime using babel :)

Unfortunately this is currently stalled due to an apparent nft rt match
kernel bug preventing me from representing multiple possible outcomes since
I want to support dropping, accepting but also stateful firewalling of
matching flows.

--Daniel

Re: [Babel-users] [RFC] Replace WireGuard AllowedIPs with IP route attribute

[Daniel Gröber]

[-- Attachment #1: Type: text/plain, Size: 3621 bytes --]

Hi Alexander,

On Wed, Nov 22, 2023 at 12:17:49AM +0100, Alexander Zubkov wrote:
> > Can you think of a use-case where fpRPF isn't enough?
> 
> Yes. IMHO, the problem with RPF is that routing table doesn't reflect the
> network topology, but only a subset of it.

Right that is the fundamental problem, so my solution to that is: routing
should "just represent the full network topology" :)  As the routing
protocol sees it anyway, since the whole point of RFP is to only allow
paths that the routing system chooses.

Do note that while I implement the topology information using ECMP routes
there's no reason you actually have to use ECMP. You could still have
regular routes in your (main) routing table and use a separate table with
ECMP routes for RPF and this is very much something I want us to support.

> I mean in topologies where multiple pathes are possible, you can choose
> to use or even learn only a subset of those pathes.

If I undestand correctly you're talking about (local) routing daemon policy
here. Yes this is something you can do and my current approach of (abusing)
ECMP only works when your routing policy satisfies some symmetry criteria.

However as Juliusz pointed out integrating this idea into the routing
protocol proper could allow using arbitrary policy without ever breaking
RPF, but figuring out the details is (exciting) future work.

> In that sense might be yes, the original sin is that the routing table
> doesn't reflect all the topology, not only the pathes we choose for egress.
> Not sure though if it is a sin, in that case routing table would be too
> overcomplicated.

Right routing table (modification) performance and clutter is certainly a
reason to forgoe this approach but I find that for the kind of (small)
networks I want to run and that many people might run using wireguard this
is perfectly fine.

> If I understand correctly, such fpRPF approach works only if you both learn
> all possible pathes and use all of them in a multi-nexthop route. But for
> example in the Internet with its advanced BGP announcement policies it is
> not true at all.

Right to deply fpRPF on a large scale you really need some kind of support
from the routing protocol. AFAIK there's nothing like that for BGP yet?

I don't think it's completely inapplicable either though, might still work
for iBGP with appropriately designed routing policy. My interest lies
mostly with doing this using babel though.

> So from my point of view it is good to split the topology definition
> (ingress decapsulation) and the chosen pathes (egress routing). Because it
> is related, but still different processes. So the system can be more
> flexible. Although we need to repeat common things and keep ingress and
> egress consistent/synced.

To me flexibility is only desirable insofar as it doesn't conflict with
system security. Source address authenticity is an important property I
wouldn't want to give up here. If it's easier to ignore source address
filtering than it is to implement it nobody is going to do it (cf. the
internet) and I think that's the crux of the problem with "encap".

Wireguard gifted us this amazing state of source filtering being the easy
default and I want to keep it that way.

> my point is that RPF (with its variations too) has its bounds and cannot
> be a universal solution, there is no silver bullet here.

No, ofc. nothing we do can possibly "fix everything for everyone" but
that's no reason not to try a new approach for a particular problem in a
particular use-case :)

--Daniel

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]