Windows tunnel shows established but traffic sometimes does not move after recycling tunnel

Windows tunnel shows established but traffic sometimes does not move after recycling tunnel

[Sander Saares]

Hi!

I submit a report on a problem encountered attempting to use WireGuard in a Windows-to-Windows "VPN gateway/proxy" deployment. I have a test deployment available in case I can provide further data for ease of debugging.

Scenario:
* Server A set up as WireGuard server, accepting connections from server B.
* Traffic from WireGuard network is forwarded and NATed by server A using built-in Windows networking features.
* Server B connects through WireGuard tunnel to access the internet.

For purpose of experimentation, the internet is defined as 8.8.8.8/32.

Expected result: tunnel is successfully established, internet traffic of server B is forwarded through server A.
Actual result: tunnel is successfully established (at least as shown in WireGuard GUI) but sometimes the expected traffic flows do not occur.
Occasionally, actual result matches expected result.

Method of observation: mutual ping on private IP address; ping from server B (WG client) to 8.8.8.8.
In failure case:
* both pings time out (server A and server B cannot ping each other on private IP)
* ping to 8.8.8.8 times out, EXCEPT for the first ping after tunnel is re-established (server B always seems to get 1 response before connectivity vanishes; possibly this is a ping not routed through the VPN, so it just goes directly out from server B to the internet?)
In success case, all pings work fine and get expected responses.

I suspect some startup/lifecycle/timing issue disrupting proper operation of the tunnel and/or associated routing configuration. If I can provide more data that may prove useful, I am happy to do so when instructed on how to collect it.

Configuration and experiment log follows.

Both systems are Windows 2019 (17763.652) running in clean Azure VMs, fully patched. WireGuard 0.0.19.

WireGuard server (server A)
=======================
[Interface]
PrivateKey = 
ListenPort = 9000
Address = 172.16.16.1/24

[Peer]
PublicKey = 
AllowedIPs = 172.16.16.0/24

WireGuard client (server B)
=======================
[Interface]
PrivateKey = 
Address = 172.16.16.2/24

[Peer]
PublicKey = 
AllowedIPs = 172.16.16.0/24, 8.8.8.8/32
Endpoint = xxx:9000

Forward+NAT setup (server A)
=================
PS C:\Users\saares> $interfaces = Get-NetIPInterface
PS C:\Users\saares> $interfaces[4]

ifIndex InterfaceAlias                  AddressFamily NlMtu(Bytes) InterfaceMetric Dhcp     ConnectionState PolicyStore
------- --------------                  ------------- ------------ --------------- ----     --------------- -----------
3       wg-test                         IPv4                  1420               5 Disabled Connected       ActiveStore

PS C:\Users\saares> $interfaces[4] | Set-NetIPInterface -Forwarding Enabled
PS C:\Users\saares> New-NetNat -Name NAT -InternalIPInterfaceAddressPrefix "172.16.16.0/24"

Name                             : NAT
ExternalIPInterfaceAddressPrefix :
InternalIPInterfaceAddressPrefix : 172.16.16.0/24
IcmpQueryTimeout                 : 30
TcpEstablishedConnectionTimeout  : 1800
TcpTransientConnectionTimeout    : 120
TcpFilteringBehavior             : AddressDependentFiltering
UdpFilteringBehavior             : AddressDependentFiltering
UdpIdleSessionTimeout            : 120
UdpInboundRefresh                : False
Store                            : Local
Active                           : True

Experiment log
===============
Immediate after setup -> all OK
Recycle tunnel on server -> all OK
Restart server PC -> tunnel reestablished but traffic does not move
Recycle tunnel on server -> all OK
Recycle tunnel on server -> all OK
Recycle tunnel on server -> all OK
Recycle tunnel on server -> all OK
Recycle tunnel on server -> all OK
Recycle tunnel on server -> all OK
Restart server PC -> all OK
Restart server PC -> tunnel reestablished but traffic does not move
Recycle tunnel on server -> tunnel reestablished but traffic does not move
Recycle tunnel on server -> tunnel reestablished but traffic does not move
Recycle tunnel on server -> tunnel reestablished but traffic does not move
Recycle tunnel on server -> tunnel reestablished but traffic does not move
Recycle tunnel on server -> all OK
Recycle tunnel on server -> tunnel reestablished but traffic does not move
Recycle tunnel on server -> tunnel reestablished but traffic does not move
Recycle tunnel on server -> tunnel reestablished but traffic does not move



Cheers,

Sander Saares
Advisor 
 
Axinom | Soola 8 | 51004 Tartu | Estonia
phone: +49 911 80109-54
saares@axinom.com | www.axinom.com 
 
Managing Directors: Sergei Gussev, Oleg Knut
Tartu Circuit Court, Reg. 11046287

_______________________________________________
WireGuard mailing list
WireGuard@lists.zx2c4.com
https://lists.zx2c4.com/mailman/listinfo/wireguard

RE: Windows tunnel shows established but traffic sometimes does not move after recycling tunnel

[Sander Saares]

Hi!

I tested again with latest Wireguard (.35) and Windows 2019 (.914) and the problem remains easy to reproduce.

In short, after restarting a Windows that is doing NAT for a peer behind a Wireguard tunnel, the tunnel stops correctly moving packets. I can see the Wiregurad "sent" and "received" counts increment to some degree but the end result is that pings do not work after some restarts of the tunnel/server).

I remain ready to assist in further diagnosis if a more informed person can guide me to what data might be useful.


Cheers,

Sander Saares, Advisor, Axinom
phone: +49 911 80109-54 | saares@axinom.com

-----Original Message-----
From: Sander Saares 
Sent: esmaspäev, 5. august 2019 10:55
To: 'wireguard@lists.zx2c4.com' <wireguard@lists.zx2c4.com>
Subject: Windows tunnel shows established but traffic sometimes does not move after recycling tunnel

Hi!

I submit a report on a problem encountered attempting to use WireGuard in a Windows-to-Windows "VPN gateway/proxy" deployment. I have a test deployment available in case I can provide further data for ease of debugging.

Scenario:
* Server A set up as WireGuard server, accepting connections from server B.
* Traffic from WireGuard network is forwarded and NATed by server A using built-in Windows networking features.
* Server B connects through WireGuard tunnel to access the internet.

For purpose of experimentation, the internet is defined as 8.8.8.8/32.

Expected result: tunnel is successfully established, internet traffic of server B is forwarded through server A.
Actual result: tunnel is successfully established (at least as shown in WireGuard GUI) but sometimes the expected traffic flows do not occur.
Occasionally, actual result matches expected result.

Method of observation: mutual ping on private IP address; ping from server B (WG client) to 8.8.8.8.
In failure case:
* both pings time out (server A and server B cannot ping each other on private IP)
* ping to 8.8.8.8 times out, EXCEPT for the first ping after tunnel is re-established (server B always seems to get 1 response before connectivity vanishes; possibly this is a ping not routed through the VPN, so it just goes directly out from server B to the internet?)
In success case, all pings work fine and get expected responses.

I suspect some startup/lifecycle/timing issue disrupting proper operation of the tunnel and/or associated routing configuration. If I can provide more data that may prove useful, I am happy to do so when instructed on how to collect it.

Configuration and experiment log follows.

Both systems are Windows 2019 (17763.652) running in clean Azure VMs, fully patched. WireGuard 0.0.19.

WireGuard server (server A)
=======================
[Interface]
PrivateKey = 
ListenPort = 9000
Address = 172.16.16.1/24

[Peer]
PublicKey = 
AllowedIPs = 172.16.16.0/24

WireGuard client (server B)
=======================
[Interface]
PrivateKey = 
Address = 172.16.16.2/24

[Peer]
PublicKey = 
AllowedIPs = 172.16.16.0/24, 8.8.8.8/32
Endpoint = xxx:9000

Forward+NAT setup (server A)
=================
PS C:\Users\saares> $interfaces = Get-NetIPInterface
PS C:\Users\saares> $interfaces[4]

ifIndex InterfaceAlias                  AddressFamily NlMtu(Bytes) InterfaceMetric Dhcp     ConnectionState PolicyStore
------- --------------                  ------------- ------------ --------------- ----     --------------- -----------
3       wg-test                         IPv4                  1420               5 Disabled Connected       ActiveStore

PS C:\Users\saares> $interfaces[4] | Set-NetIPInterface -Forwarding Enabled
PS C:\Users\saares> New-NetNat -Name NAT -InternalIPInterfaceAddressPrefix "172.16.16.0/24"

Name                             : NAT
ExternalIPInterfaceAddressPrefix :
InternalIPInterfaceAddressPrefix : 172.16.16.0/24
IcmpQueryTimeout                 : 30
TcpEstablishedConnectionTimeout  : 1800
TcpTransientConnectionTimeout    : 120
TcpFilteringBehavior             : AddressDependentFiltering
UdpFilteringBehavior             : AddressDependentFiltering
UdpIdleSessionTimeout            : 120
UdpInboundRefresh                : False
Store                            : Local
Active                           : True

Experiment log
===============
Immediate after setup -> all OK
Recycle tunnel on server -> all OK
Restart server PC -> tunnel reestablished but traffic does not move
Recycle tunnel on server -> all OK
Recycle tunnel on server -> all OK
Recycle tunnel on server -> all OK
Recycle tunnel on server -> all OK
Recycle tunnel on server -> all OK
Recycle tunnel on server -> all OK
Restart server PC -> all OK
Restart server PC -> tunnel reestablished but traffic does not move
Recycle tunnel on server -> tunnel reestablished but traffic does not move
Recycle tunnel on server -> tunnel reestablished but traffic does not move
Recycle tunnel on server -> tunnel reestablished but traffic does not move
Recycle tunnel on server -> tunnel reestablished but traffic does not move
Recycle tunnel on server -> all OK
Recycle tunnel on server -> tunnel reestablished but traffic does not move
Recycle tunnel on server -> tunnel reestablished but traffic does not move
Recycle tunnel on server -> tunnel reestablished but traffic does not move



Cheers,

Sander Saares
Advisor 
 
Axinom | Soola 8 | 51004 Tartu | Estonia
phone: +49 911 80109-54
saares@axinom.com | www.axinom.com 
 
Managing Directors: Sergei Gussev, Oleg Knut
Tartu Circuit Court, Reg. 11046287

_______________________________________________
WireGuard mailing list
WireGuard@lists.zx2c4.com
https://lists.zx2c4.com/mailman/listinfo/wireguard

Re: Windows tunnel shows established but traffic sometimes does not move after recycling tunnel

[Jason A. Donenfeld]

I suspect the reason is because WireGuard uses a fresh interface each
time, so you have to reenable forwarding after the interface comes up.
_______________________________________________
WireGuard mailing list
WireGuard@lists.zx2c4.com
https://lists.zx2c4.com/mailman/listinfo/wireguard

RE: Windows tunnel shows established but traffic sometimes does not move after recycling tunnel

[Sander Saares]

That does not seem to be the case - the configuration shows that forwarding remains enabled on the interface. 

Furthermore, the issue not only affects routed traffic to external destinations but also traffic on the WireGuard network itself - pings between the internal WireGuard peer IP addresses stop working.

The primary connection with routing/NAT functionality is that this seems to be a precondition for the fault to appear - but the disrupted traffic is not limited to routed/NATed traffic.

The theory that creating new interfaces causes this effect also does not reconcile with the fact that the behavior is inconsistent between machine and tunnel restarts.


I will illustrate what the data tells me. After tunnel setup, I enable forwarding and NAT as follows:

PS C:\Users\saares> Get-NetIPInterface wg

ifIndex InterfaceAlias                  AddressFamily NlMtu(Bytes) InterfaceMetric Dhcp     ConnectionState PolicyStore
------- --------------                  ------------- ------------ --------------- ----     --------------- -----------
10      wg                              IPv6                 65535               5 Disabled Connected       ActiveStore
10      wg                              IPv4                  1420               5 Disabled Connected       ActiveStore


PS C:\Users\saares> Get-NetIPInterface wg | Set-NetIPInterface -Forwarding Enabled
PS C:\Users\saares> New-NetNat -name nat -InternalIPInterfaceAddressPrefix "192.168.90.0/24"


Name                             : nat
ExternalIPInterfaceAddressPrefix :
InternalIPInterfaceAddressPrefix : 192.168.90.0/24
IcmpQueryTimeout                 : 30
TcpEstablishedConnectionTimeout  : 1800
TcpTransientConnectionTimeout    : 120
TcpFilteringBehavior             : AddressDependentFiltering
UdpFilteringBehavior             : AddressDependentFiltering
UdpIdleSessionTimeout            : 120
UdpInboundRefresh                : False
Store                            : Local
Active                           : True

After this, data starts being routed correctly and NAT is performed. Recycling the tunnel does not break anything so far - traffic continues to move shortly after the tunnel comes up again.

Now I restart the machine. From here on, it is a game of chance. Today, after the first restart everything worked fine. Windows tells me forwarding is still enabled on the WireGuard interface.

PS C:\Users\saares> Get-NetIPInterface -Forwarding Enabled

ifIndex InterfaceAlias                  AddressFamily NlMtu(Bytes) InterfaceMetric Dhcp     ConnectionState PolicyStore
------- --------------                  ------------- ------------ --------------- ----     --------------- -----------
7       wg                              IPv6                 65535               5 Disabled Connected       ActiveStore
7       wg                              IPv4                  1420               5 Disabled Connected       ActiveStore

Then I restart the computer again. Now pings no longer work. Not between the peers on the private address range (192.168.90.0/24) nor to the internet.

Windows tells me the configuration has not changed - forwarding remains enabled and NAT remains active

I can see the following in the "server" (routing) side route table (.2 is the router, .1 is the client, both Windows): 

     192.168.90.0    255.255.255.0         On-link      192.168.90.2    261
     192.168.90.1  255.255.255.255         On-link      192.168.90.2      5
     192.168.90.2  255.255.255.255         On-link      192.168.90.2    261
   192.168.90.255  255.255.255.255         On-link      192.168.90.2    261


On the client side there is the equivalent:

     192.168.90.0    255.255.255.0         On-link      192.168.90.1      5
     192.168.90.1  255.255.255.255         On-link      192.168.90.1    261
   192.168.90.255  255.255.255.255         On-link      192.168.90.1    261


On the data transfer size counters in WireGuard, I can see that the pings do travel from client to router ("sent" on client increases and "received" on router increases by same amount) but no responses appear to be going back toward the peer acting as client.

When I ping the peer acting as client from the router side, I see the expected amount of traffic in both directions on the tunnel. However, the router side shows the ping as timeout.

Based on the transfer size counters, I would say that the packets coming into the router side through the WireGuard tunnel are not being processed and are being discarded for some reason.

I lack the knowledge to better analyze why/how they are being discarded but if you can tell me what I can do to investigate, I will do so.

Sometimes (20% of the time?) recycling the tunnel itself will fix the problem. Feels very much like some sort of timing/caching/initialization-order issue to my untrained eye. Note also that sometimes 1 ping from "client" peet to internet (via "router" peer) will actually get through before traffic flow disappears.

Toggling IP forwarding and re-creating the NAT configuration does not appear to change anything.

Starting the router side with the tunnel deactivated and only later activating it does not appear to change anything.




Cheers,

Sander Saares, Advisor, Axinom
phone: +49 911 80109-54 | saares@axinom.com

-----Original Message-----
From: Jason A. Donenfeld <Jason@zx2c4.com> 
Sent: neljapäev, 19. detsember 2019 03:24
To: Sander Saares <saares@axinom.com>
Cc: wireguard@lists.zx2c4.com
Subject: Re: Windows tunnel shows established but traffic sometimes does not move after recycling tunnel

I suspect the reason is because WireGuard uses a fresh interface each time, so you have to reenable forwarding after the interface comes up.
_______________________________________________
WireGuard mailing list
WireGuard@lists.zx2c4.com
https://lists.zx2c4.com/mailman/listinfo/wireguard