PMTU Discovery Security Concerns

PMTU Discovery Security Concerns

[Jason A. Donenfeld]

Hi list,

[CC'ing Luis, who's been working on this with me.]

I've more or less figured out how to do PMTU discovery (something
along the lines of https://xn--4db.cc/WFHQzX2o/c inspired by the vti
driver). I wonder, however, if this is safe to do.

The basic idea is that if you're talking to a WireGuard peer via its
internal tunnel IP address, the kernel keeps some notion of what that
internal IP address's PMTU is. Meanwhile, WireGuard itself is talking
to that peer via its external endpoint IP address, and the kernel
keeps some notion of what that external IP address's PMTU is. If the
encrypted packets are larger than the external PMTU, well behaved
networks will send ICMP messages, indicating that packets sent to that
external endpoint IP should be smaller. This, however, doesn't have
anything to do with what the user is trying to send internally, and so
there will continue to be overly large packets sent.

The way to fix it would be to relay the external endpoint PMTU to the
internal tunnel PMTU. Then, when an external encrypted packet gets
dropped due to being overly large, the ICMP message for that winds up
affecting the internal tunneled IP address's PMTU, so that the next
message it sends is smaller. All is well, packets flow, and TCP
sessions no longer stall. This is generally how PMTU discovery works
with network tunnels.

The security problem is that those ICMP messages indicating we should
send smaller packets are unauthenticated, since they're triggered by
things external to the tunnel, rather than inside the tunnel. By
propagating the information from those unauthenticated ICMP messages
to state that concerns the inside of the tunnel, we're essentially
enabling an unauthenticated state injection. This could enable some
mischief. On the benign end of the spectrum, an attacker could launch
a DoS attack by causing the sending end to use smaller and smaller
packets. On the scarier end of the spectrum, an attacker could
intelligently do this to change the size of packets and observe the
way in which a data flow is rechunked, in order to infer something
about the actual data being encrypted.

These security concerns make me inclined to just simplistically
declare, "PMTU discovery in tunnels can't be done securely with the
existing Internet, so WireGuard doesn't support it." However,
undoubtedly smart people have thought about this before, and perhaps
they've come up with real solutions for this. Indeed I've come across
various discussions of the matter in the IPsec RFCs. But at the
present moment, I'm unsure what the most reasonable way forward is.

So, I thought if anyone on the list has thought about this extensively
before and would like to chime in with some wisdom, I'm all ears.

Regards,
Jason

Re: PMTU Discovery Security Concerns

[Ryan Whelan]

[-- Attachment #1: Type: text/plain, Size: 3360 bytes --]

I don't have an actual fix; but is something that could have a switch that
could be configured per interface?  I know knobs and controls aren't really
desirable, but if Off by default, it would encourage those turning it on to
understand what they're exposing.

$0.02

On Sun, Apr 15, 2018 at 10:08 AM, Jason A. Donenfeld <Jason@zx2c4.com>
wrote:

> Hi list,
>
> [CC'ing Luis, who's been working on this with me.]
>
> I've more or less figured out how to do PMTU discovery (something
> along the lines of https://xn--4db.cc/WFHQzX2o/c inspired by the vti
> driver). I wonder, however, if this is safe to do.
>
> The basic idea is that if you're talking to a WireGuard peer via its
> internal tunnel IP address, the kernel keeps some notion of what that
> internal IP address's PMTU is. Meanwhile, WireGuard itself is talking
> to that peer via its external endpoint IP address, and the kernel
> keeps some notion of what that external IP address's PMTU is. If the
> encrypted packets are larger than the external PMTU, well behaved
> networks will send ICMP messages, indicating that packets sent to that
> external endpoint IP should be smaller. This, however, doesn't have
> anything to do with what the user is trying to send internally, and so
> there will continue to be overly large packets sent.
>
> The way to fix it would be to relay the external endpoint PMTU to the
> internal tunnel PMTU. Then, when an external encrypted packet gets
> dropped due to being overly large, the ICMP message for that winds up
> affecting the internal tunneled IP address's PMTU, so that the next
> message it sends is smaller. All is well, packets flow, and TCP
> sessions no longer stall. This is generally how PMTU discovery works
> with network tunnels.
>
> The security problem is that those ICMP messages indicating we should
> send smaller packets are unauthenticated, since they're triggered by
> things external to the tunnel, rather than inside the tunnel. By
> propagating the information from those unauthenticated ICMP messages
> to state that concerns the inside of the tunnel, we're essentially
> enabling an unauthenticated state injection. This could enable some
> mischief. On the benign end of the spectrum, an attacker could launch
> a DoS attack by causing the sending end to use smaller and smaller
> packets. On the scarier end of the spectrum, an attacker could
> intelligently do this to change the size of packets and observe the
> way in which a data flow is rechunked, in order to infer something
> about the actual data being encrypted.
>
> These security concerns make me inclined to just simplistically
> declare, "PMTU discovery in tunnels can't be done securely with the
> existing Internet, so WireGuard doesn't support it." However,
> undoubtedly smart people have thought about this before, and perhaps
> they've come up with real solutions for this. Indeed I've come across
> various discussions of the matter in the IPsec RFCs. But at the
> present moment, I'm unsure what the most reasonable way forward is.
>
> So, I thought if anyone on the list has thought about this extensively
> before and would like to chime in with some wisdom, I'm all ears.
>
> Regards,
> Jason
> _______________________________________________
> WireGuard mailing list
> WireGuard@lists.zx2c4.com
> https://lists.zx2c4.com/mailman/listinfo/wireguard
>

[-- Attachment #2: Type: text/html, Size: 4172 bytes --]

Re: PMTU Discovery Security Concerns

[Jason A. Donenfeld]

On Sun, Apr 15, 2018 at 5:45 PM, Ryan Whelan <rcwhelan@gmail.com> wrote:
> I don't have an actual fix; but is something that could have a switch that
> could be configured per interface?  I know knobs and controls aren't really
> desirable, but if Off by default, it would encourage those turning it on to
> understand what they're exposing.

Knobs with a security impact? Sounds like a foot-gun disaster, and
something we've always tried to avoid here. I'm interested more in
seeing whether there's a secure way to do the right thing all of the
time.

Re: PMTU Discovery Security Concerns

[Tim Sedlmeyer]

PMTUD on the Internet is often broken and increasingly becoming more
broken, so in my opinion introducing any level of potential security
concern to support it would be unwise.

If MTU issues are regularly presenting a significant issue to
successful deployment of wireguard than in the short term I would
suggest doing what many ipsec implementations do, give up some
performance in order to increase the likelihood of successful
transmission by assigning a default MTU significantly small enough to
avoid issues in the vast majority of circumstances. For instance by
default the OS X ipsec vpn implementation assigns an MTU of 1280, the
minimum IPv6 datagram size, to the tunnel by default. Cisco assigns an
MTU of 1300 by default.

In the long term some form of packetization layer path mtu discovery
probably should be added to the wireguard protocol itself. Perhaps by
padding the first message of the handshake to utilize it as a mtu
discovery probe.

On Sun, Apr 15, 2018 at 10:08 AM, Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> Hi list,
>
> [CC'ing Luis, who's been working on this with me.]
>
> I've more or less figured out how to do PMTU discovery (something
> along the lines of https://xn--4db.cc/WFHQzX2o/c inspired by the vti
> driver). I wonder, however, if this is safe to do.
>
> The basic idea is that if you're talking to a WireGuard peer via its
> internal tunnel IP address, the kernel keeps some notion of what that
> internal IP address's PMTU is. Meanwhile, WireGuard itself is talking
> to that peer via its external endpoint IP address, and the kernel
> keeps some notion of what that external IP address's PMTU is. If the
> encrypted packets are larger than the external PMTU, well behaved
> networks will send ICMP messages, indicating that packets sent to that
> external endpoint IP should be smaller. This, however, doesn't have
> anything to do with what the user is trying to send internally, and so
> there will continue to be overly large packets sent.
>
> The way to fix it would be to relay the external endpoint PMTU to the
> internal tunnel PMTU. Then, when an external encrypted packet gets
> dropped due to being overly large, the ICMP message for that winds up
> affecting the internal tunneled IP address's PMTU, so that the next
> message it sends is smaller. All is well, packets flow, and TCP
> sessions no longer stall. This is generally how PMTU discovery works
> with network tunnels.
>
> The security problem is that those ICMP messages indicating we should
> send smaller packets are unauthenticated, since they're triggered by
> things external to the tunnel, rather than inside the tunnel. By
> propagating the information from those unauthenticated ICMP messages
> to state that concerns the inside of the tunnel, we're essentially
> enabling an unauthenticated state injection. This could enable some
> mischief. On the benign end of the spectrum, an attacker could launch
> a DoS attack by causing the sending end to use smaller and smaller
> packets. On the scarier end of the spectrum, an attacker could
> intelligently do this to change the size of packets and observe the
> way in which a data flow is rechunked, in order to infer something
> about the actual data being encrypted.
>
> These security concerns make me inclined to just simplistically
> declare, "PMTU discovery in tunnels can't be done securely with the
> existing Internet, so WireGuard doesn't support it." However,
> undoubtedly smart people have thought about this before, and perhaps
> they've come up with real solutions for this. Indeed I've come across
> various discussions of the matter in the IPsec RFCs. But at the
> present moment, I'm unsure what the most reasonable way forward is.
>
> So, I thought if anyone on the list has thought about this extensively
> before and would like to chime in with some wisdom, I'm all ears.
>
> Regards,
> Jason
> _______________________________________________
> WireGuard mailing list
> WireGuard@lists.zx2c4.com
> https://lists.zx2c4.com/mailman/listinfo/wireguard

Re: PMTU Discovery Security Concerns

[Jason A. Donenfeld]

On Sun, Apr 15, 2018 at 6:06 PM, Tim Sedlmeyer <tim@sedlmeyer.org> wrote:
> PMTUD on the Internet is often broken and increasingly becoming more
> broken, so in my opinion introducing any level of potential security
> concern to support it would be unwise.

I was wondering if there's actually an appropriate use case for PMTUD
within networks that are well behaved. For example, within various
intranets, or between physical sites within a campus. Perhaps these
aren't relevant, since they're centrally managed anyway, and so we
should just give up with PMTUD all together?

> If MTU issues are regularly presenting a significant issue to
> successful deployment of wireguard than in the short term I would
> suggest doing what many ipsec implementations do, give up some
> performance in order to increase the likelihood of successful
> transmission by assigning a default MTU significantly small enough to
> avoid issues in the vast majority of circumstances. For instance by
> default the OS X ipsec vpn implementation assigns an MTU of 1280, the
> minimum IPv6 datagram size, to the tunnel by default. Cisco assigns an
> MTU of 1300 by default.

Not a bad idea for end user clients. Ugly, but maybe nobody would be
too perturbed. wg-quick(8) has an MTU= parameter, after all, which
could be set to 1280.

> In the long term some form of packetization layer path mtu discovery
> probably should be added to the wireguard protocol itself. Perhaps by
> padding the first message of the handshake to utilize it as a mtu
> discovery probe.

I was thinking about implementing something like this on top of
WireGuard -- a basic ping probe tool that walks through each peer and
tries to ping one of its allowed IPs within the tunnel. Maybe this
would take care of most peoples' use cases. It is explicit, however,
rather than the nice on-demand automatic aspect of traditional PMTUD.

But anyway, all of this falls into the category of, "let's just not do
PTMUD." I'm still not convinced it's impossible to do securely, mostly
because I haven't heard anybody explicitly say, "we thought about this
for 25 years and it can't be done."

Re: PMTU Discovery Security Concerns

[Tim Sedlmeyer]

On Sun, Apr 15, 2018 at 12:13 PM, Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> On Sun, Apr 15, 2018 at 6:06 PM, Tim Sedlmeyer <tim@sedlmeyer.org> wrote:
>> PMTUD on the Internet is often broken and increasingly becoming more
>> broken, so in my opinion introducing any level of potential security
>> concern to support it would be unwise.
>
> I was wondering if there's actually an appropriate use case for PMTUD
> within networks that are well behaved. For example, within various
> intranets, or between physical sites within a campus. Perhaps these
> aren't relevant, since they're centrally managed anyway, and so we
> should just give up with PMTUD all together?
>

If you are in the network operations side of a intranet/campus network,
getting wireguard going probably isn't your first time wrangling with MTU.

>> If MTU issues are regularly presenting a significant issue to
>> successful deployment of wireguard than in the short term I would
>> suggest doing what many ipsec implementations do, give up some
>> performance in order to increase the likelihood of successful
>> transmission by assigning a default MTU significantly small enough to
>> avoid issues in the vast majority of circumstances. For instance by
>> default the OS X ipsec vpn implementation assigns an MTU of 1280, the
>> minimum IPv6 datagram size, to the tunnel by default. Cisco assigns an
>> MTU of 1300 by default.
>
> Not a bad idea for end user clients. Ugly, but maybe nobody would be
> too perturbed. wg-quick(8) has an MTU= parameter, after all, which
> could be set to 1280.
>
>> In the long term some form of packetization layer path mtu discovery
>> probably should be added to the wireguard protocol itself. Perhaps by
>> padding the first message of the handshake to utilize it as a mtu
>> discovery probe.
>
> I was thinking about implementing something like this on top of
> WireGuard -- a basic ping probe tool that walks through each peer and
> tries to ping one of its allowed IPs within the tunnel. Maybe this
> would take care of most peoples' use cases. It is explicit, however,
> rather than the nice on-demand automatic aspect of traditional PMTUD.

Not sure how well a ping probe would work.

- Which allowed-ip do you use?
- If the allowed-ip is a network, which ip within it do you choose to ping?
- If you are connected to a single peer with an allowed-ip of 0.0.0.0/0 what
  ip do you ping?

The allowed-ip isn't guaranteed to be on the same device as the peer so,
in the end you aren't measuring the mtu over the connection between peers
but the complete path to that allowed-ip which could involve more devices and
connections with smaller MTUs than between the peers themselves.

See RFC4821, RFC8085 and
https://tools.ietf.org/html/draft-ietf-tsvwg-datagram-plpmtud-01
for more info about PLMTUD.

https://datatracker.ietf.org/meeting/101/materials/slides-101-ipsecme-packetization-layer-path-mtu-discovery-01
has a quick overview of where IPsec stands with implementing it.

>
> But anyway, all of this falls into the category of, "let's just not do
> PTMUD." I'm still not convinced it's impossible to do securely, mostly
> because I haven't heard anybody explicitly say, "we thought about this
> for 25 years and it can't be done."

Re: PMTU Discovery Security Concerns

[Jason A. Donenfeld]

On Sun, Apr 15, 2018 at 7:51 PM, Tim Sedlmeyer <tim@sedlmeyer.org> wrote:
> - Which allowed-ip do you use?
> - If the allowed-ip is a network, which ip within it do you choose to ping?
> - If you are connected to a single peer with an allowed-ip of 0.0.0.0/0 what
>   ip do you ping?

Yea, the actual IP discovery is a hurdle to figure out.

> The allowed-ip isn't guaranteed to be on the same device as the peer so,
> in the end you aren't measuring the mtu over the connection between peers
> but the complete path to that allowed-ip which could involve more devices and
> connections with smaller MTUs than between the peers themselves.

That's probably fine and even desirable, since we're looking for the
PMTU along a certain route.

> See RFC4821, RFC8085 and
> https://tools.ietf.org/html/draft-ietf-tsvwg-datagram-plpmtud-01
> for more info about PLMTUD.
>
> https://datatracker.ietf.org/meeting/101/materials/slides-101-ipsecme-packetization-layer-path-mtu-discovery-01
> has a quick overview of where IPsec stands with implementing it.

Thanks for these. I followed the rabbit hole, and found [1], which
seems to be the current latest and greatest from the IPsec people.
It's probes inside the control plane. Reading the references, such as
[2], it seems pretty unanimous that going anywhere near out-of-tunnel
ICMP messages is a disaster, as I suggested in the original post here.
That's useful confirmation, and I guess we'll indeed have to look at
creative non-ICMP solutions for PMTUD to happen.

[1] https://tools.ietf.org/html/draft-spiriyath-ipsecme-dynamic-ipsec-pmtu-01.html
[2] https://tools.ietf.org/html/draft-roca-ipsecme-ptb-pts-attack-00

Re: PMTU Discovery Security Concerns

[Derek Fawcus]

On Mon, Apr 16, 2018 at 07:23:29AM +0200, Jason A. Donenfeld wrote:
> > See RFC4821, RFC8085 and
> > https://tools.ietf.org/html/draft-ietf-tsvwg-datagram-plpmtud-01
> > for more info about PLMTUD.
> >
> > https://datatracker.ietf.org/meeting/101/materials/slides-101-ipsecme-packetization-layer-path-mtu-discovery-01
> > has a quick overview of where IPsec stands with implementing it.
> 
> Thanks for these. I followed the rabbit hole, and found [1], which
> seems to be the current latest and greatest from the IPsec people.
> It's probes inside the control plane.

Using something based upon that UDP version of PLMTUD would seem to
be the correct approach, sending probe packets protected by the
encryption layer.

Rather than have it totally based upon the control layer, one could
use a data layer notification of decrypted packets arriving to
indicate that certain sizes have been received, then reflect those
back to the sender in a minimal encrypted ack packet.  i.e. something
similar to how solicited probe responses operate for IPv6 NDP.

Keep in mind that the PMTU may be different in the two directions.

DF