Development discussion of WireGuard
 help / color / mirror / Atom feed
* Wintun NeighborDiscoverySupported
@ 2021-09-09 14:32 Brad Spencer
  2021-09-09 17:42 ` Jason A. Donenfeld
  0 siblings, 1 reply; 6+ messages in thread
From: Brad Spencer @ 2021-09-09 14:32 UTC (permalink / raw)
  To: WireGuard mailing list

We have noticed that Windows seems to try to send ARP requests over 
Wintun interfaces.  In our configurations, these don't go anywhere and 
get no responses, but the ARP table fills up with addresses.  For example:

$  arp -a -N 10.0.0.100 |wc -l
387

Our Wintun interfaces are created with these properties:

PS C:\> Get-NetIPInterface -InterfaceIndex 69 |Select-Object -Property 
AddressFamily,NeighborDiscoverySupported,NeighborUnreachabilityDetection |fl

AddressFamily                   : IPv6
NeighborDiscoverySupported      : Yes
NeighborUnreachabilityDetection : Enabled

AddressFamily                   : IPv4
NeighborDiscoverySupported      : Yes
NeighborUnreachabilityDetection : Enabled

We _think_ that the NeighborDiscoverySupported property being Yes means 
that Windows issues ARP requests for addresses on the Wintun interface.  
(The second property controls another neighbour behaviour that perhaps 
could also be disabled for Wintun interfaces.)

Microsoft's documentation for SetIpInterfaceEntry() claims that these 
properties are set "by the network stack", but offers no hint as to what 
part of the network stack does this.  It does not seem possible to 
change these properties on an existing interface.

Are these properties that either the Wintun driver or user-space API is 
able to control?  Should they be set to No and Disabled?  I don't see 
any use of the driver variant of SetIpInterfaceEntry() or 
MIB_IPINTERFACE_ROW in Wintun's driver or API code, so I'm not sure when 
these properties get determined.

-- 
Brad Spencer


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Wintun NeighborDiscoverySupported
  2021-09-09 14:32 Wintun NeighborDiscoverySupported Brad Spencer
@ 2021-09-09 17:42 ` Jason A. Donenfeld
  2021-09-09 18:15   ` Brad Spencer
  0 siblings, 1 reply; 6+ messages in thread
From: Jason A. Donenfeld @ 2021-09-09 17:42 UTC (permalink / raw)
  To: Brad Spencer; +Cc: WireGuard mailing list

Hi Brad,

That sure is interesting. Indeed UseNeighborUnreachabilityDetection
and SupportsNeighborDiscovery can't be set with SetIpInterfaceEntry. I
haven't (yet?) found any way for the driver itself to indicate that
these should be false, either, via OIDs or similar. This might require
some frustrating reverse engineering. I remember noticing this a long
time ago, but I deemed it "annoying but harmless". However, you now
mention:

On Thu, Sep 9, 2021 at 4:33 PM Brad Spencer <bspencer@blackberry.com> wrote:
> but the ARP table fills up with addresses.  For example:
>
> $  arp -a -N 10.0.0.100 |wc -l
> 387

If that grows indefinitely, that sounds... bad. So this might be worth
looking into again. One thing about your message, though, raised a
question in my mind, but I'm not sure whether its an artifact of your
wording or a real thing you observed:

> We have noticed that Windows seems to try to send ARP requests over
> Wintun interfaces.  In our configurations, these don't go anywhere and
> get no responses,

Indeed the ARP table fills up, as shown above, but I'm wondering how
you're observing ARP requests exactly. ARP is a layer 2 protocol, and
Wintun (and WireGuardNT) are layer 3 devices that should, in theory at
least, not have anything to do with ARP packets. So how exactly were
you "seeing" the ARP requests on the Wintun interface? Did wireshark
show it? Or did you read from the Wintun ring and actually see an ARP
frame? Or something else? Or was this just a manner of speaking and
you didn't actually observe ARP frames themselves?

Another small question:

> We _think_ that the NeighborDiscoverySupported property being Yes means
> that Windows issues ARP requests for addresses on the Wintun interface.

That seems like a good intuition. I'm wondering whether that's
something you're assuming or something you read on a Microsoft
website. I ask because this might provide a good entry point for
whatever reverse engineering I wind up doing to fix this.

Regards,
Jason

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Wintun NeighborDiscoverySupported
  2021-09-09 17:42 ` Jason A. Donenfeld
@ 2021-09-09 18:15   ` Brad Spencer
  2021-09-09 20:23     ` Alan Graham
  0 siblings, 1 reply; 6+ messages in thread
From: Brad Spencer @ 2021-09-09 18:15 UTC (permalink / raw)
  To: Jason A. Donenfeld; +Cc: WireGuard mailing list

On 2021-09-09 2:42 p.m., Jason A. Donenfeld wrote:
> So how exactly were
> you "seeing" the ARP requests on the Wintun interface? Did wireshark
> show it? Or did you read from the Wintun ring and actually see an ARP
> frame? Or something else? Or was this just a manner of speaking and
> you didn't actually observe ARP frames themselves?
You're right to suspect that I was speaking imprecisely here.  We have 
never seen an ARP request appear on the Wintun interface!  I meant to 
say that we have noticed the ARP table for the Wintun interface 
accumulating entries.

>> We _think_ that the NeighborDiscoverySupported property being Yes means
>> that Windows issues ARP requests for addresses on the Wintun interface.
> That seems like a good intuition. I'm wondering whether that's
> something you're assuming or something you read on a Microsoft
> website. I ask because this might provide a good entry point for
> whatever reverse engineering I wind up doing to fix this.
I pieced this together from a few scraps.

On the MIB_IPINTERFACE_ROW page[1], the docs only tersely say:

"A value that specifies if the IP interface support neighbor discovery."

Microsoft seems to use the same terminology when documenting 
SetIpNetEntry2()[2]:

"The SetIpNetEntry2 function sets the physical address of an existing 
neighbor IP address entry on the local computer."

And then, most importantly, MIB_IPNET_ROW2, the structure used by that 
function says this in its Remarks section[3]:

"For IPv4, this includes addresses determined used the Address 
Resolution Protocol (ARP). For IPv6, this includes addresses determined 
using the Neighbor Discovery (ND) protocol for IPv6 as specified in RFC 
2461. "

So, it seems that the "NetEntry" APIs are those that deal with ARP (and 
ND) entries, and the term Microsoft uses for that is "neighbor discovery".

I don't know if the SupportsNeighborDiscovery field of MIB_INTERFACE_ROW 
is implied by other properties of the network interface (such as "all 
Ethernet interfaces support ARP") or whether it can be individually set 
at all.

One other detail is that we have the gateway for the tunnel's routes set 
to 0.0.0.0 (or "::").  I presume that also influences how Windows 
decides which addresses might be on-link neighbours.

1. 
https://docs.microsoft.com/en-us/windows/win32/api/netioapi/ns-netioapi-mib_ipinterface_row
2. 
https://docs.microsoft.com/en-us/windows/win32/api/netioapi/nf-netioapi-setipnetentry2
3. 
https://docs.microsoft.com/en-us/windows/win32/api/netioapi/ns-netioapi-mib_ipnet_table2

-- 
Brad Spencer


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Wintun NeighborDiscoverySupported
  2021-09-09 18:15   ` Brad Spencer
@ 2021-09-09 20:23     ` Alan Graham
  2021-09-10 17:19       ` Brad Spencer
  0 siblings, 1 reply; 6+ messages in thread
From: Alan Graham @ 2021-09-09 20:23 UTC (permalink / raw)
  To: Brad Spencer; +Cc: Jason A. Donenfeld, WireGuard mailing list

Hi Brad, Jason,

Adding the 0.0.0.0/0 (or ":::") to the config is what is causing some
growth of the arp table, but it is not growing indefinitely.  After
looking around for SupportsNeighborDiscovery and finding nothing, I
decided to check the repro.  When not routing all traffic through an
interface the arp cache is basically static:

PS C:\>  arp -a -N 100.0.0.2

Interface: 100.0.0.2 --- 0x38
  Internet Address      Physical Address      Type
  100.0.0.4                                   dynamic
  224.0.0.22                                  static
  224.0.0.251                                 static
  224.0.0.252                                 static
  239.255.255.250                             static

When routing all traffic through it grows more or less like an arp cache should:

PS C:\>  arp -a -N 100.0.0.2

Interface: 100.0.0.2 --- 0x38
  Internet Address      Physical Address      Type
  0.0.0.0                                     static
  8.8.4.4                                     dynamic
  8.8.8.8                                     dynamic
  13.64.180.106                               dynamic
  18.211.21.156                               dynamic
  20.98.103.204                               dynamic
  23.6.164.43                                 dynamic
  23.40.197.163                               dynamic
  23.204.103.147                              dynamic
  35.185.199.42                               dynamic
  40.83.247.108                               dynamic
  51.143.14.218                               dynamic
  54.214.97.217                               dynamic
  65.8.17.107                                 dynamic
  67.160.11.228                               dynamic
  69.1.20.242                                 dynamic
  73.118.184.4                                dynamic
  74.125.142.188                              dynamic
  74.125.197.188                              dynamic
  75.75.75.75                                 dynamic
  100.0.0.2                                   dynamic
  100.0.0.4                                   dynamic
  142.250.69.202                              dynamic
  142.250.217.99                              dynamic
  142.251.33.74                               dynamic
  142.251.33.106                              dynamic
  173.194.202.189                             dynamic
  192.168.200.2                               dynamic
  192.228.79.201                              dynamic
  224.0.0.22                                  static
  224.0.0.251                                 static
  224.0.0.252                                 static
  239.255.255.250                             static

So I tend to agree with Jason that this is "harmless" and shouldn't
cause any serious problems.  It would be nice for Microsoft to fix
Set-NetIPInterface, it looks like a bug that SupportsNeighborDiscovery
can't be set.

Best regards,
Alan Graham


On Thu, Sep 9, 2021 at 11:17 AM Brad Spencer <bspencer@blackberry.com> wrote:
>
> On 2021-09-09 2:42 p.m., Jason A. Donenfeld wrote:
> > So how exactly were
> > you "seeing" the ARP requests on the Wintun interface? Did wireshark
> > show it? Or did you read from the Wintun ring and actually see an ARP
> > frame? Or something else? Or was this just a manner of speaking and
> > you didn't actually observe ARP frames themselves?
> You're right to suspect that I was speaking imprecisely here.  We have
> never seen an ARP request appear on the Wintun interface!  I meant to
> say that we have noticed the ARP table for the Wintun interface
> accumulating entries.
>
> >> We _think_ that the NeighborDiscoverySupported property being Yes means
> >> that Windows issues ARP requests for addresses on the Wintun interface.
> > That seems like a good intuition. I'm wondering whether that's
> > something you're assuming or something you read on a Microsoft
> > website. I ask because this might provide a good entry point for
> > whatever reverse engineering I wind up doing to fix this.
> I pieced this together from a few scraps.
>
> On the MIB_IPINTERFACE_ROW page[1], the docs only tersely say:
>
> "A value that specifies if the IP interface support neighbor discovery."
>
> Microsoft seems to use the same terminology when documenting
> SetIpNetEntry2()[2]:
>
> "The SetIpNetEntry2 function sets the physical address of an existing
> neighbor IP address entry on the local computer."
>
> And then, most importantly, MIB_IPNET_ROW2, the structure used by that
> function says this in its Remarks section[3]:
>
> "For IPv4, this includes addresses determined used the Address
> Resolution Protocol (ARP). For IPv6, this includes addresses determined
> using the Neighbor Discovery (ND) protocol for IPv6 as specified in RFC
> 2461. "
>
> So, it seems that the "NetEntry" APIs are those that deal with ARP (and
> ND) entries, and the term Microsoft uses for that is "neighbor discovery".
>
> I don't know if the SupportsNeighborDiscovery field of MIB_INTERFACE_ROW
> is implied by other properties of the network interface (such as "all
> Ethernet interfaces support ARP") or whether it can be individually set
> at all.
>
> One other detail is that we have the gateway for the tunnel's routes set
> to 0.0.0.0 (or "::").  I presume that also influences how Windows
> decides which addresses might be on-link neighbours.
>
> 1.
> https://docs.microsoft.com/en-us/windows/win32/api/netioapi/ns-netioapi-mib_ipinterface_row
> 2.
> https://docs.microsoft.com/en-us/windows/win32/api/netioapi/nf-netioapi-setipnetentry2
> 3.
> https://docs.microsoft.com/en-us/windows/win32/api/netioapi/ns-netioapi-mib_ipnet_table2
>
> --
> Brad Spencer
>

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Wintun NeighborDiscoverySupported
  2021-09-09 20:23     ` Alan Graham
@ 2021-09-10 17:19       ` Brad Spencer
  2021-09-10 20:56         ` Alan Graham
  0 siblings, 1 reply; 6+ messages in thread
From: Brad Spencer @ 2021-09-10 17:19 UTC (permalink / raw)
  To: Alan Graham; +Cc: Jason A. Donenfeld, WireGuard mailing list

On 2021-09-09 5:23 p.m., Alan Graham wrote:
> Adding the 0.0.0.0/0 (or ":::") to the config is what is causing some
> growth of the arp table, but it is not growing indefinitely.  After
> looking around for SupportsNeighborDiscovery and finding nothing, I
> decided to check the repro.  When not routing all traffic through an
> interface the arp cache is basically static:

Thanks for looking into this, too.

I also suspect having the gateway set like this is probably necessary 
for Windows to start adding ARP entries.  How were you able to determine 
that it is also sufficient?  My guess is that if it is possible to 
indicate to Windows that the interface does not support neighbour 
discovery in general, doing so likely prevents ARP entries regardless of 
the gateway values.

BTW, how did you determine that it does not grow indefinitely?

> So I tend to agree with Jason that this is "harmless" and shouldn't
> cause any serious problems.  It would be nice for Microsoft to fix
> Set-NetIPInterface, it looks like a bug that SupportsNeighborDiscovery
> can't be set.

One "harm" might be that the OS keeping an easy-to-query list of all 
(recent?) destinations in the ARP table, which could be undesirable.

I'm not sure it's a bug, per se.  It seems by design that you cannot 
change the value of SupportsNeighborDiscovery after the interface is 
created.  The documentation for SetIpInterfaceEntry()[1] says:

> The MaxReassemblySize, MinRouterAdvertisementInterval, 
> MaxRouterAdvertisementInterval , Connected, SupportsWakeUpPatterns, 
> SupportsNeighborDiscovery, SupportsRouterDiscovery, ReachableTime, 
> TransmitOffload, and ReceiveOffload members of the MIB_IPINTERFACE_ROW 
> structure pointed to by the Row are ignored when the 
> SetIpInterfaceEntry function is called. These members are set by the 
> network stack and cannot be changed using the SetIpInterfaceEntry 
> function.

I noticed that the Cmdlet in PowerShell seems to treat the 
-NeighborDiscoverySupported option as an input filter vs. a value that 
you can set.  While this surprised me, it is at least consistent with 
the Win32 API docs.

1. 
https://docs.microsoft.com/en-us/windows/win32/api/netioapi/nf-netioapi-setipinterfaceentry#remarks 


-- 
Brad Spencer


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Wintun NeighborDiscoverySupported
  2021-09-10 17:19       ` Brad Spencer
@ 2021-09-10 20:56         ` Alan Graham
  0 siblings, 0 replies; 6+ messages in thread
From: Alan Graham @ 2021-09-10 20:56 UTC (permalink / raw)
  To: Brad Spencer; +Cc: Jason A. Donenfeld, WireGuard mailing list

I mainly reached the conclusion that it's not growing indefinitely by
looking at the results in the arp table.  There weren't duplicates or
other errors which might suggest the arp table wasn't working
properly.  I believe it is, in that it should be aging out entries,
etc.  I also stopped and restarted the wireguard service and it
cleared the cache as it should.

I also noticed the Powershell Cmdlet used the wrong filter parameters
when trying to set NeighborDiscoverySupported, as you did.  My
expectation would be to get an error invalid parameter like with
NeighborUnreachabilityDetection, not that the filter returned zero
results because it used the wrong query.

Best regards,
Alan Graham


On Fri, Sep 10, 2021 at 10:19 AM Brad Spencer <bspencer@blackberry.com> wrote:
>
> On 2021-09-09 5:23 p.m., Alan Graham wrote:
> > Adding the 0.0.0.0/0 (or ":::") to the config is what is causing some
> > growth of the arp table, but it is not growing indefinitely.  After
> > looking around for SupportsNeighborDiscovery and finding nothing, I
> > decided to check the repro.  When not routing all traffic through an
> > interface the arp cache is basically static:
>
> Thanks for looking into this, too.
>
> I also suspect having the gateway set like this is probably necessary
> for Windows to start adding ARP entries.  How were you able to determine
> that it is also sufficient?  My guess is that if it is possible to
> indicate to Windows that the interface does not support neighbour
> discovery in general, doing so likely prevents ARP entries regardless of
> the gateway values.
>
> BTW, how did you determine that it does not grow indefinitely?
>
> > So I tend to agree with Jason that this is "harmless" and shouldn't
> > cause any serious problems.  It would be nice for Microsoft to fix
> > Set-NetIPInterface, it looks like a bug that SupportsNeighborDiscovery
> > can't be set.
>
> One "harm" might be that the OS keeping an easy-to-query list of all
> (recent?) destinations in the ARP table, which could be undesirable.
>
> I'm not sure it's a bug, per se.  It seems by design that you cannot
> change the value of SupportsNeighborDiscovery after the interface is
> created.  The documentation for SetIpInterfaceEntry()[1] says:
>
> > The MaxReassemblySize, MinRouterAdvertisementInterval,
> > MaxRouterAdvertisementInterval , Connected, SupportsWakeUpPatterns,
> > SupportsNeighborDiscovery, SupportsRouterDiscovery, ReachableTime,
> > TransmitOffload, and ReceiveOffload members of the MIB_IPINTERFACE_ROW
> > structure pointed to by the Row are ignored when the
> > SetIpInterfaceEntry function is called. These members are set by the
> > network stack and cannot be changed using the SetIpInterfaceEntry
> > function.
>
> I noticed that the Cmdlet in PowerShell seems to treat the
> -NeighborDiscoverySupported option as an input filter vs. a value that
> you can set.  While this surprised me, it is at least consistent with
> the Win32 API docs.
>
> 1.
> https://docs.microsoft.com/en-us/windows/win32/api/netioapi/nf-netioapi-setipinterfaceentry#remarks
>
>
> --
> Brad Spencer
>

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2021-09-10 20:56 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-09 14:32 Wintun NeighborDiscoverySupported Brad Spencer
2021-09-09 17:42 ` Jason A. Donenfeld
2021-09-09 18:15   ` Brad Spencer
2021-09-09 20:23     ` Alan Graham
2021-09-10 17:19       ` Brad Spencer
2021-09-10 20:56         ` Alan Graham

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).