"roaming" between source ports does not work

"roaming" between source ports does not work

[Matthias May]

Hi

== Premise
* I've recently implemented support for wireguard in our LTE-router.

== Source Environment
* The basis is OpenWRT.
* Used versions:
* On the client/initiator:
 * wg
  * 1.0.20200908
  * ad33b2d2267a37e0f65c97e65e7d4d926d5aef7d530c251b63fbf919048eead9
 * wg-tools
  * 1.0.20200827
  * 51bc85e33a5b3cf353786ae64b0f1216d7a871447f058b6137f793eb0f53b7fd
* On the server/responder:
 * Debian stretch (9.13), installed from repository
 * deb http://deb.debian.org/debian/ unstable main
 * # wg --version
 * wireguard-tools v1.0.20200827
 * I don't really know what the version of the build dkms is

== Issue
* We've implemented an automated test that seems to have a problem.
 * Each night, the device is configured to connect to the debian box.
  * This works fine the first time.
  * However it doesn't work anymore after this first time.

== Observerion
When the "client" connects the first time, wg-output on the "server"
looks like this:
> interface: wg1
>   public key: 7GxCG4m+6Kf4wjJ9vbQaGFASLGXLB5ddPWgBYw4gOk8=
>   private key: (hidden)
>   listening port: 51821
>
> peer: fizBdi/YkdzFLaq6Hnq+OZaGmbJBYC15QSP1Mik/EFU=
>   endpoint: 172.29.42.230:38442
>   allowed ips: 10.0.41.3/32
>   latest handshake: 44 seconds ago
>   transfer: 8.01 MiB received, 7.96 MiB sent

and on the "client:
> interface: wg1
>   public key: fizBdi/YkdzFLaq6Hnq+OZaGmbJBYC15QSP1Mik/EFU=
>   private key: (hidden)
>   listening port: 38442
>
> peer: 7GxCG4m+6Kf4wjJ9vbQaGFASLGXLB5ddPWgBYw4gOk8=
>   endpoint: 172.29.60.13:51821
>   allowed ips: 10.0.41.0/24
>   latest handshake: 1 minute, 3 seconds ago
>   transfer: 187.06 KiB received, 189.96 KiB sent

Ports and IPs match, everything works.

However on the second run of the test:
On the "server" still:
> peer: fizBdi/YkdzFLaq6Hnq+OZaGmbJBYC15QSP1Mik/EFU=
>   endpoint: 172.29.42.230:38442
>   allowed ips: 10.0.41.3/32
>   latest handshake: 4 minutes, 52 seconds ago
>   transfer: 8.05 MiB received, 7.99 MiB sent

But the "client" shows:
> interface: wg1
>   public key: fizBdi/YkdzFLaq6Hnq+OZaGmbJBYC15QSP1Mik/EFU=
>   private key: (hidden)
>   listening port: 47858

The client device has been restarted in between.

Since the listen-port is set to 0, it obviously has now a new,
different, source-port.
The server doesn't pick this up.
Since peers may roam between IPs, i was under the impression, that it
would also roam between ports.


Is this working as intended?
If yes: How should the configuration look like to support clients doing
a power-cycle?


I'm aware, that i could set a static port on the client, but this won't
work when going through NAT with port-scrambling.
So i don't really have control over the source-port of the connection
anyway.
I suppose this would also apply when a router/firewall inbetween has
some aggressive killing of states where the keepalive is not fast
enough, and source-port scrambling is done.

But the main usecase i'm looking at here is: restart of a device.

BR
Matthias

Re: "roaming" between source ports does not work

[Ivan Labáth]

Hello,

are you sure changing of source port is the issue?
Seems like something that would be reported a long
time ago.

Wireguard handshake fails, if your timestamps aren't
monotonically increasing - maybe this is the issue?

For confirmation - does connection fail on wg restart without
a device power cycle, or if you change the source port
while the tunnel is running?

If your device is power cycling on a schedule, without a RTC,
you should arrange an increasing nonce/time, if you can save
data, maybe use NTP or a workaround may be to remove and
re-add the peer on the server on a compatible schedule, 

Regards,
Ivan


On Sun, Nov 08, 2020 at 11:00:30PM +0100, Matthias May wrote:
> Hi
> 
> == Premise
> * I've recently implemented support for wireguard in our LTE-router.
> 
> == Source Environment
> * The basis is OpenWRT.
> * Used versions:
> * On the client/initiator:
>  * wg
>   * 1.0.20200908
>   * ad33b2d2267a37e0f65c97e65e7d4d926d5aef7d530c251b63fbf919048eead9
>  * wg-tools
>   * 1.0.20200827
>   * 51bc85e33a5b3cf353786ae64b0f1216d7a871447f058b6137f793eb0f53b7fd
> * On the server/responder:
>  * Debian stretch (9.13), installed from repository
>  * deb http://deb.debian.org/debian/ unstable main
>  * # wg --version
>  * wireguard-tools v1.0.20200827
>  * I don't really know what the version of the build dkms is
> 
> == Issue
> * We've implemented an automated test that seems to have a problem.
>  * Each night, the device is configured to connect to the debian box.
>   * This works fine the first time.
>   * However it doesn't work anymore after this first time.
> 
> == Observerion
> When the "client" connects the first time, wg-output on the "server"
> looks like this:
> > interface: wg1
> >   public key: 7GxCG4m+6Kf4wjJ9vbQaGFASLGXLB5ddPWgBYw4gOk8=
> >   private key: (hidden)
> >   listening port: 51821
> >
> > peer: fizBdi/YkdzFLaq6Hnq+OZaGmbJBYC15QSP1Mik/EFU=
> >   endpoint: 172.29.42.230:38442
> >   allowed ips: 10.0.41.3/32
> >   latest handshake: 44 seconds ago
> >   transfer: 8.01 MiB received, 7.96 MiB sent
> 
> and on the "client:
> > interface: wg1
> >   public key: fizBdi/YkdzFLaq6Hnq+OZaGmbJBYC15QSP1Mik/EFU=
> >   private key: (hidden)
> >   listening port: 38442
> >
> > peer: 7GxCG4m+6Kf4wjJ9vbQaGFASLGXLB5ddPWgBYw4gOk8=
> >   endpoint: 172.29.60.13:51821
> >   allowed ips: 10.0.41.0/24
> >   latest handshake: 1 minute, 3 seconds ago
> >   transfer: 187.06 KiB received, 189.96 KiB sent
> 
> Ports and IPs match, everything works.
> 
> However on the second run of the test:
> On the "server" still:
> > peer: fizBdi/YkdzFLaq6Hnq+OZaGmbJBYC15QSP1Mik/EFU=
> >   endpoint: 172.29.42.230:38442
> >   allowed ips: 10.0.41.3/32
> >   latest handshake: 4 minutes, 52 seconds ago
> >   transfer: 8.05 MiB received, 7.99 MiB sent
> 
> But the "client" shows:
> > interface: wg1
> >   public key: fizBdi/YkdzFLaq6Hnq+OZaGmbJBYC15QSP1Mik/EFU=
> >   private key: (hidden)
> >   listening port: 47858
> 
> The client device has been restarted in between.
> 
> Since the listen-port is set to 0, it obviously has now a new,
> different, source-port.
> The server doesn't pick this up.
> Since peers may roam between IPs, i was under the impression, that it
> would also roam between ports.
> 
> 
> Is this working as intended?
> If yes: How should the configuration look like to support clients doing
> a power-cycle?
> 
> 
> I'm aware, that i could set a static port on the client, but this won't
> work when going through NAT with port-scrambling.
> So i don't really have control over the source-port of the connection
> anyway.
> I suppose this would also apply when a router/firewall inbetween has
> some aggressive killing of states where the keepalive is not fast
> enough, and source-port scrambling is done.
> 
> But the main usecase i'm looking at here is: restart of a device.
> 
> BR
> Matthias

Re: "roaming" between source ports does not work

[Eicke Herbertz]

[-- Attachment #1.1: Type: text/plain, Size: 4856 bytes --]

Hi,

I agree that something simple as changing the source port would have
been reported. However, I have very similar issues occasionally with
some of my systems that are all behind NAT. They are remote, so I can't
access their status information on error. Rebooting the devices usually
works and iirc, just disconnecting network for some time works, too.

I don't know if that's far fetched, but I actually suspect the cause of
this issue not in WireGuard, but in some buggy NAT-implementation out
there. This would only hold true though, if you are also able to fix
your issue by replacing related networking hardware.

My central point being however is, that I've probably experienced the
same issue.

Regards,
Eicke

Am 24.11.20 um 08:57 schrieb Ivan Labáth:
> Hello,
> 
> are you sure changing of source port is the issue?
> Seems like something that would be reported a long
> time ago.
> 
> Wireguard handshake fails, if your timestamps aren't
> monotonically increasing - maybe this is the issue?
> 
> For confirmation - does connection fail on wg restart without
> a device power cycle, or if you change the source port
> while the tunnel is running?
> 
> If your device is power cycling on a schedule, without a RTC,
> you should arrange an increasing nonce/time, if you can save
> data, maybe use NTP or a workaround may be to remove and
> re-add the peer on the server on a compatible schedule, 
> 
> Regards,
> Ivan
> 
> 
> On Sun, Nov 08, 2020 at 11:00:30PM +0100, Matthias May wrote:
>> Hi
>>
>> == Premise
>> * I've recently implemented support for wireguard in our LTE-router.
>>
>> == Source Environment
>> * The basis is OpenWRT.
>> * Used versions:
>> * On the client/initiator:
>>  * wg
>>   * 1.0.20200908
>>   * ad33b2d2267a37e0f65c97e65e7d4d926d5aef7d530c251b63fbf919048eead9
>>  * wg-tools
>>   * 1.0.20200827
>>   * 51bc85e33a5b3cf353786ae64b0f1216d7a871447f058b6137f793eb0f53b7fd
>> * On the server/responder:
>>  * Debian stretch (9.13), installed from repository
>>  * deb http://deb.debian.org/debian/ unstable main
>>  * # wg --version
>>  * wireguard-tools v1.0.20200827
>>  * I don't really know what the version of the build dkms is
>>
>> == Issue
>> * We've implemented an automated test that seems to have a problem.
>>  * Each night, the device is configured to connect to the debian box.
>>   * This works fine the first time.
>>   * However it doesn't work anymore after this first time.
>>
>> == Observerion
>> When the "client" connects the first time, wg-output on the "server"
>> looks like this:
>>> interface: wg1
>>>   public key: 7GxCG4m+6Kf4wjJ9vbQaGFASLGXLB5ddPWgBYw4gOk8=
>>>   private key: (hidden)
>>>   listening port: 51821
>>>
>>> peer: fizBdi/YkdzFLaq6Hnq+OZaGmbJBYC15QSP1Mik/EFU=
>>>   endpoint: 172.29.42.230:38442
>>>   allowed ips: 10.0.41.3/32
>>>   latest handshake: 44 seconds ago
>>>   transfer: 8.01 MiB received, 7.96 MiB sent
>>
>> and on the "client:
>>> interface: wg1
>>>   public key: fizBdi/YkdzFLaq6Hnq+OZaGmbJBYC15QSP1Mik/EFU=
>>>   private key: (hidden)
>>>   listening port: 38442
>>>
>>> peer: 7GxCG4m+6Kf4wjJ9vbQaGFASLGXLB5ddPWgBYw4gOk8=
>>>   endpoint: 172.29.60.13:51821
>>>   allowed ips: 10.0.41.0/24
>>>   latest handshake: 1 minute, 3 seconds ago
>>>   transfer: 187.06 KiB received, 189.96 KiB sent
>>
>> Ports and IPs match, everything works.
>>
>> However on the second run of the test:
>> On the "server" still:
>>> peer: fizBdi/YkdzFLaq6Hnq+OZaGmbJBYC15QSP1Mik/EFU=
>>>   endpoint: 172.29.42.230:38442
>>>   allowed ips: 10.0.41.3/32
>>>   latest handshake: 4 minutes, 52 seconds ago
>>>   transfer: 8.05 MiB received, 7.99 MiB sent
>>
>> But the "client" shows:
>>> interface: wg1
>>>   public key: fizBdi/YkdzFLaq6Hnq+OZaGmbJBYC15QSP1Mik/EFU=
>>>   private key: (hidden)
>>>   listening port: 47858
>>
>> The client device has been restarted in between.
>>
>> Since the listen-port is set to 0, it obviously has now a new,
>> different, source-port.
>> The server doesn't pick this up.
>> Since peers may roam between IPs, i was under the impression, that it
>> would also roam between ports.
>>
>>
>> Is this working as intended?
>> If yes: How should the configuration look like to support clients doing
>> a power-cycle?
>>
>>
>> I'm aware, that i could set a static port on the client, but this won't
>> work when going through NAT with port-scrambling.
>> So i don't really have control over the source-port of the connection
>> anyway.
>> I suppose this would also apply when a router/firewall inbetween has
>> some aggressive killing of states where the keepalive is not fast
>> enough, and source-port scrambling is done.
>>
>> But the main usecase i'm looking at here is: restart of a device.
>>
>> BR
>> Matthias


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: "roaming" between source ports does not work

[Matthias May]

On 24/11/2020 08:57, Ivan Labáth wrote:
> Hello,
> 
> are you sure changing of source port is the issue?
> Seems like something that would be reported a long
> time ago.
> 
> Wireguard handshake fails, if your timestamps aren't
> monotonically increasing - maybe this is the issue?
> 
> For confirmation - does connection fail on wg restart without
> a device power cycle, or if you change the source port
> while the tunnel is running?
> 
> If your device is power cycling on a schedule, without a RTC,
> you should arrange an increasing nonce/time, if you can save
> data, maybe use NTP or a workaround may be to remove and
> re-add the peer on the server on a compatible schedule, 
> 
> Regards,
> Ivan
> 
> 
> On Sun, Nov 08, 2020 at 11:00:30PM +0100, Matthias May wrote:
>> Hi
>>
>> == Premise
>> * I've recently implemented support for wireguard in our LTE-router.
>>
>> == Source Environment
>> * The basis is OpenWRT.
>> * Used versions:
>> * On the client/initiator:
>>  * wg
>>   * 1.0.20200908
>>   * ad33b2d2267a37e0f65c97e65e7d4d926d5aef7d530c251b63fbf919048eead9
>>  * wg-tools
>>   * 1.0.20200827
>>   * 51bc85e33a5b3cf353786ae64b0f1216d7a871447f058b6137f793eb0f53b7fd
>> * On the server/responder:
>>  * Debian stretch (9.13), installed from repository
>>  * deb https://urldefense.com/v3/__http://deb.debian.org/debian/__;!!I9LPvj3b!RT42f9KbvkRAUCotWoe9WbvdGg0pfsEckxDFl3iujPxZcNW5KHCoRkhTfxHA91cvFlQ$  unstable main
>>  * # wg --version
>>  * wireguard-tools v1.0.20200827
>>  * I don't really know what the version of the build dkms is
>>
>> == Issue
>> * We've implemented an automated test that seems to have a problem.
>>  * Each night, the device is configured to connect to the debian box.
>>   * This works fine the first time.
>>   * However it doesn't work anymore after this first time.
>>
>> == Observerion
>> When the "client" connects the first time, wg-output on the "server"
>> looks like this:
>>> interface: wg1
>>>   public key: 7GxCG4m+6Kf4wjJ9vbQaGFASLGXLB5ddPWgBYw4gOk8=
>>>   private key: (hidden)
>>>   listening port: 51821
>>>
>>> peer: fizBdi/YkdzFLaq6Hnq+OZaGmbJBYC15QSP1Mik/EFU=
>>>   endpoint: 172.29.42.230:38442
>>>   allowed ips: 10.0.41.3/32
>>>   latest handshake: 44 seconds ago
>>>   transfer: 8.01 MiB received, 7.96 MiB sent
>>
>> and on the "client:
>>> interface: wg1
>>>   public key: fizBdi/YkdzFLaq6Hnq+OZaGmbJBYC15QSP1Mik/EFU=
>>>   private key: (hidden)
>>>   listening port: 38442
>>>
>>> peer: 7GxCG4m+6Kf4wjJ9vbQaGFASLGXLB5ddPWgBYw4gOk8=
>>>   endpoint: 172.29.60.13:51821
>>>   allowed ips: 10.0.41.0/24
>>>   latest handshake: 1 minute, 3 seconds ago
>>>   transfer: 187.06 KiB received, 189.96 KiB sent
>>
>> Ports and IPs match, everything works.
>>
>> However on the second run of the test:
>> On the "server" still:
>>> peer: fizBdi/YkdzFLaq6Hnq+OZaGmbJBYC15QSP1Mik/EFU=
>>>   endpoint: 172.29.42.230:38442
>>>   allowed ips: 10.0.41.3/32
>>>   latest handshake: 4 minutes, 52 seconds ago
>>>   transfer: 8.05 MiB received, 7.99 MiB sent
>>
>> But the "client" shows:
>>> interface: wg1
>>>   public key: fizBdi/YkdzFLaq6Hnq+OZaGmbJBYC15QSP1Mik/EFU=
>>>   private key: (hidden)
>>>   listening port: 47858
>>
>> The client device has been restarted in between.
>>
>> Since the listen-port is set to 0, it obviously has now a new,
>> different, source-port.
>> The server doesn't pick this up.
>> Since peers may roam between IPs, i was under the impression, that it
>> would also roam between ports.
>>
>>
>> Is this working as intended?
>> If yes: How should the configuration look like to support clients doing
>> a power-cycle?
>>
>>
>> I'm aware, that i could set a static port on the client, but this won't
>> work when going through NAT with port-scrambling.
>> So i don't really have control over the source-port of the connection
>> anyway.
>> I suppose this would also apply when a router/firewall inbetween has
>> some aggressive killing of states where the keepalive is not fast
>> enough, and source-port scrambling is done.
>>
>> But the main usecase i'm looking at here is: restart of a device.
>>
>> BR
>> Matthias


Hi Ivan
Thank you for response.
It seems my message was hanging somewhere, at least i didn't see it show up on the list until just now, thus i gave up
on it.

Yes your suggestion to use NTP is/was correct.
I found some similar reports.
Once NTP was configured, roaming between ports started working.

BR
Matthias