FR: interface ListenAddress (Aka:Multihomed server issue)

FR: interface ListenAddress (Aka:Multihomed server issue)

[Jan De Landtsheer]

[-- Attachment #1: Type: text/plain, Size: 1826 bytes --]

Jason,
To elaborate on
https://lists.zx2c4.com/pipermail/wireguard/2017-August/001598.html, there
is something that can be clarified …

I have a multihomed server (our router for everything) attached to a core
switch with vlans, and the router runs openvswitch (but that’s besides the
point).

We run a bunch of wg peers, interconnected to each other (30 or so), but
most connect directly to our router.

The router has an Uplink interface with a /30 and I use that interface
solely to forward packets to our (bgp routed) default gw (Provider).

On the same router, I have a Public Interface, also with a public IP (/24)
and have on the router itself some IP addresses used for DNAT, and here
specifically one for Wireguard. (so NOT the Uplink IP address)

When wireguard clients connect, their config shows their peer to be the
Uplink IP address instead of the IP on the Public interface that was
specifically assigned for wireguard (wgsrv), and as such packets sent to
the Uplink IP address were dropped by the firewall.

You might say: open up the port for wireguard on the Uplink and off you go.
Which I did, to solve my immediate problem. (still find it ugly)

But no we’re getting a second provider in da house, that will be connected
the same way as the other, with that link being Uplink2. So now I *really*
need my bgp routed Public IP address to be the sole answering wireguard IP
packets, so that I can be sure that if one of my bgp peers dies, the same
Public ip address is used by the clients, not the one wireguard deduces
from the subnet with the default route.

Now, wireguard will use the incoming UplinkX ip as source and advertise it
to the clients connected through either one that has the same metric and
routing policy

Voila… in a nutshell ;-)

Jan
​

[-- Attachment #2: Type: text/html, Size: 8497 bytes --]

Re: FR: interface ListenAddress (Aka:Multihomed server issue)

[Jason A. Donenfeld]

Hey Jan,

> When wireguard clients connect, their config shows their peer
> to be the Uplink IP address instead of the IP on the Public
> interface that was specifically assigned for wireguard (wgsrv)

Do you mean to say that the _endpoint_ IP address of the WireGuard
peer is an IP associated with Uplink instead of with Public? If this
is the case, it might be some odd DNAT situation causing this to
happen for you? The peer's endpoint IP address is simply the src IP of
the most recently authenticated packet from the peer. It sounds like
there's something odd in place causing the src IP to be wrong? But I
can't think of how this would be WireGuard related. Unless I've
misunderstood something?

Jason

Re: FR: interface ListenAddress (Aka:Multihomed server issue)

[Jan De Landtsheer]

[-- Attachment #1: Type: text/plain, Size: 2678 bytes --]

no, very simple …
I have (for the sake of brevity) 2 interfaces:

one is eth0 with ip 123.45.67.1/30 and I have on the box 123.45.67.2 as
default gateway.
on that link I bgp peer with 123.45.67.2 and announce my own /24, let’s say
134.56.78.0/24

another eth interface (eth1) hosts several ip addresses and one of these is
134.56.78.5/24

for that interface I allow port 443 to accept packets for

[Interface]
ListenPort = 443

but I do not allow packets to connect to 123.45.67.1/30 on port 443 (as
this iface is just my Provider’s /30

when a client connects to 134.56.78.5/24, the wg server tells the client
that it’s destination is 123.45.67.1/30 for this link , and that gets of
course firewalled.
So reluctantly I opened up port 443 on the uplink interface to accomodate
this, erm, inconvenience.

on client side I have a config :

[Peer]
PublicKey = (hidden)
EndPoint = 134.56.78.5:443
AllowedIPs =  0.0.0.0/0

but when connection is established
wg show says :

peer: (hidden)
  endpoint: 123.34.56.1:443
  allowed ips: 10.0.0.0/8
  latest handshake: 36 seconds ago
  transfer: 468.40 MiB received, 17.88 MiB sent

but now of course, when the third interface eth2 will arrive, with another
subnet to another provider, my announced IP 134.56.78.5/24 may not be
altered by the path taken, otherwise the clients need to reconnect…

but I don’t know for sure… it seems to be a regression somewhere as I don’t
recall to have that problem before… I had to add this accept rule last
week, suddenly, as some peers could connect, but not transfer packets any
more.

Now I understand that wg finds it’s IP by following the shortest path, but
that is, in my case, counterproductive.
It should reply with the IP it was spoken to (here 134.56.78.5)

I think ;-)

Jan

On Thu, Aug 10, 2017 at 5:51 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:

Hey Jan,
>
> > When wireguard clients connect, their config shows their peer
> > to be the Uplink IP address instead of the IP on the Public
> > interface that was specifically assigned for wireguard (wgsrv)
>
> Do you mean to say that the _endpoint_ IP address of the WireGuard
> peer is an IP associated with Uplink instead of with Public? If this
> is the case, it might be some odd DNAT situation causing this to
> happen for you? The peer's endpoint IP address is simply the src IP of
> the most recently authenticated packet from the peer. It sounds like
> there's something odd in place causing the src IP to be wrong? But I
> can't think of how this would be WireGuard related. Unless I've
> misunderstood something?
>
> Jason
>
​

[-- Attachment #2: Type: text/html, Size: 13123 bytes --]

Re: FR: interface ListenAddress (Aka:Multihomed server issue)

[Jason A. Donenfeld]

[-- Attachment #1.1: Type: text/plain, Size: 585 bytes --]

Hi Jan,

So it looks like this is happening to you:


This should not be the behavior, and if it is, you've either found a bug in
WireGuard or a bug in your own setup.

1) Are you running the latest snapshot of WireGuard? Which one?
2) "but I don’t know for sure… it seems to be a regression somewhere as I
don’t recall to have that problem before…" Can you be more precise?
3) If you are running the latest version, does this patch fix it?
http://ix.io/z3d
4) Can you confirm that there exists a route from the server back to the
client?

Thanks,
Jason
​

[-- Attachment #1.2: Type: text/html, Size: 1099 bytes --]

[-- Attachment #2: rts.jpg --]
[-- Type: image/jpeg, Size: 81200 bytes --]

Re: FR: interface ListenAddress (Aka:Multihomed server issue)

[Jan De Landtsheer]

[-- Attachment #1.1: Type: text/plain, Size: 2666 bytes --]

basically this is what happens:
client connects to 134.56.78.5:443
wg show gives:
peer 111.22.33.25:443


 +----------------------+
                                                       |
   |
                                      +----------------+      client
   |
                                      |                |   81.82.222.111/18
(fixed IP)
                                  XXXX++XX             |
   |
                            XXXXXX        XXX
 +----------------------+
                          XX  X              XX
                          X     internet      X          client config:
                          X X                XX            [interface]
                               XX           XX             peer
134.56.78.5:443
               111.22.33.26/30  |XXX      XX
                default gateway |   XXXXXXX
                                |
                                |                        after connect:
                     UPLINK     |                        wg show
                                |                          peer
111.22.33.25:443
                                |
            +-------------------+-------------------+
            |    eth1:111.22.33.25/30               |
            |                                       |
            |       router  = wg server             |
            |                                       |
            |    eth0: 134.56.78.1/24 deft gw for 134.56.78.0/24
            |    eth0: 134.56.78.5/24 a free ip with open port for 443 (wg
destination)
            +----------------+----------------------+
                             |
                             |
                             |
                             |
                             |
+----------------------------+--------------------------------+ also
134.56.78.0/24

no nat at all in this setup only some firewall filtering

Jan

On Thu, Aug 10, 2017 at 8:40 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:

> Hi Jan,
>
> So it looks like this is happening to you:
>
>
> This should not be the behavior, and if it is, you've either found a bug
> in WireGuard or a bug in your own setup.
>
> 1) Are you running the latest snapshot of WireGuard? Which one?
> 2) "but I don’t know for sure… it seems to be a regression somewhere as I
> don’t recall to have that problem before…" Can you be more precise?
> 3) If you are running the latest version, does this patch fix it?
> http://ix.io/z3d
> 4) Can you confirm that there exists a route from the server back to the
> client?
>
> Thanks,
> Jason
> ​
>

[-- Attachment #1.2: Type: text/html, Size: 6321 bytes --]

[-- Attachment #2: rts.jpg --]
[-- Type: image/jpeg, Size: 81200 bytes --]

Re: FR: interface ListenAddress (Aka:Multihomed server issue)

[Jan De Landtsheer]

[-- Attachment #1.1: Type: text/plain, Size: 2955 bytes --]

Forgot to tell :
both versions (server/client) 0.0.20170726

Jan

On Thu, Aug 10, 2017 at 9:10 PM Jan De Landtsheer <
jan.delandtsheer@gmail.com> wrote:

>
> basically this is what happens:
> client connects to 134.56.78.5:443
> wg show gives:
> peer 111.22.33.25:443
>
>
>  +----------------------+
>                                                        |
>    |
>                                       +----------------+      client
>    |
>                                       |                |
> 81.82.222.111/18 (fixed IP)
>                                   XXXX++XX             |
>    |
>                             XXXXXX        XXX
>  +----------------------+
>                           XX  X              XX
>                           X     internet      X          client config:
>                           X X                XX            [interface]
>                                XX           XX             peer
> 134.56.78.5:443
>                111.22.33.26/30  |XXX      XX
>                 default gateway |   XXXXXXX
>                                 |
>                                 |                        after connect:
>                      UPLINK     |                        wg show
>                                 |                          peer
> 111.22.33.25:443
>                                 |
>             +-------------------+-------------------+
>             |    eth1:111.22.33.25/30               |
>             |                                       |
>             |       router  = wg server             |
>             |                                       |
>             |    eth0: 134.56.78.1/24 deft gw for 134.56.78.0/24
>             |    eth0: 134.56.78.5/24 a free ip with open port for 443
> (wg destination)
>             +----------------+----------------------+
>                              |
>                              |
>                              |
>                              |
>                              |
> +----------------------------+--------------------------------+ also
> 134.56.78.0/24
>
> no nat at all in this setup only some firewall filtering
>
> Jan
>
> On Thu, Aug 10, 2017 at 8:40 PM Jason A. Donenfeld <Jason@zx2c4.com>
> wrote:
>
>> Hi Jan,
>>
>> So it looks like this is happening to you:
>>
>>
>> This should not be the behavior, and if it is, you've either found a bug
>> in WireGuard or a bug in your own setup.
>>
>> 1) Are you running the latest snapshot of WireGuard? Which one?
>> 2) "but I don’t know for sure… it seems to be a regression somewhere as
>> I don’t recall to have that problem before…" Can you be more precise?
>> 3) If you are running the latest version, does this patch fix it?
>> http://ix.io/z3d
>> 4) Can you confirm that there exists a route from the server back to the
>> client?
>>
>> Thanks,
>> Jason
>> ​
>>
>

[-- Attachment #1.2: Type: text/html, Size: 6970 bytes --]

[-- Attachment #2: rts.jpg --]
[-- Type: image/jpeg, Size: 81200 bytes --]

Re: FR: interface ListenAddress (Aka:Multihomed server issue)

[Jason A. Donenfeld]

Hi Jan,

Thanks for the drawing. So the issue is that you want packets to exit
through eth1 using the addresses of eth0. I believe applying this
patch should enable that: http://ix.io/z3d Can you apply that and let
me know if it works?

I'm curious: do TCP connections generally work correctly with your
configuration?

Jason

Re: FR: interface ListenAddress (Aka:Multihomed server issue)

[Jan De Landtsheer]

[-- Attachment #1: Type: text/plain, Size: 1365 bytes --]

TCP connections work all right, as they’re established sockets, where the
kernel does the routing… I assumed you would search for the route yourself
;-)
rcu_dereference_bh(rt->dst.dev->ip_ptr) indeed does , as the packet
effectively comes in through the uplink.

In the firewall config I need to specify both interfaces (Uplink and Public
(eth1 and eth0 in the drawing) to filter

nft add rule ip filter input iif {Uplink,Public} jump public and define my
rules in the public chain
nft add rule ip filter public ip daddr 134.56.78.5 udp dport 443 accept so
a packet coming in on Uplink for the wg gets accepted only if the dst ip
matches.

nftables FTW ;-)

That in se is not very important if you have only one uplink, but if you
have multiple routes (default gw’s) you really need the ip behind the
uplinks.

But anyway, tested and confirmed to work now,

Many thanks for the quick reply
​

On Thu, Aug 10, 2017 at 9:46 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:

> Hi Jan,
>
> Thanks for the drawing. So the issue is that you want packets to exit
> through eth1 using the addresses of eth0. I believe applying this
> patch should enable that: http://ix.io/z3d Can you apply that and let
> me know if it works?
>
> I'm curious: do TCP connections generally work correctly with your
> configuration?
>
> Jason
>

[-- Attachment #2: Type: text/html, Size: 4136 bytes --]

Re: FR: interface ListenAddress (Aka:Multihomed server issue)

[Jan De Landtsheer]

[-- Attachment #1: Type: text/plain, Size: 1675 bytes --]

seeing the latest & greatest, this patch will not apply cleanly, so I don't
know ... is this train of thought going to be kept for later releases ?

On Thu, Aug 10, 2017 at 10:50 PM Jan De Landtsheer <
jan.delandtsheer@gmail.com> wrote:

> TCP connections work all right, as they’re established sockets, where the
> kernel does the routing… I assumed you would search for the route yourself
> ;-)
> rcu_dereference_bh(rt->dst.dev->ip_ptr) indeed does , as the packet
> effectively comes in through the uplink.
>
> In the firewall config I need to specify both interfaces (Uplink and
> Public (eth1 and eth0 in the drawing) to filter
>
> nft add rule ip filter input iif {Uplink,Public} jump public and define
> my rules in the public chain
> nft add rule ip filter public ip daddr 134.56.78.5 udp dport 443 accept
> so a packet coming in on Uplink for the wg gets accepted only if the dst ip
> matches.
>
> nftables FTW ;-)
>
> That in se is not very important if you have only one uplink, but if you
> have multiple routes (default gw’s) you really need the ip behind the
> uplinks.
>
> But anyway, tested and confirmed to work now,
>
> Many thanks for the quick reply
> ​
>
> On Thu, Aug 10, 2017 at 9:46 PM Jason A. Donenfeld <Jason@zx2c4.com>
> wrote:
>
>> Hi Jan,
>>
>> Thanks for the drawing. So the issue is that you want packets to exit
>> through eth1 using the addresses of eth0. I believe applying this
>> patch should enable that: http://ix.io/z3d Can you apply that and let
>> me know if it works?
>>
>> I'm curious: do TCP connections generally work correctly with your
>> configuration?
>>
>> Jason
>>
>

[-- Attachment #2: Type: text/html, Size: 4655 bytes --]

Re: FR: interface ListenAddress (Aka:Multihomed server issue)

[Jason A. Donenfeld]

On Thu, Aug 10, 2017 at 11:00 PM, Jan De Landtsheer
<jan.delandtsheer@gmail.com> wrote:
> seeing the latest & greatest, this patch will not apply cleanly, so I don't
> know ... is this train of thought going to be kept for later releases ?

If anything changes, it would be for whatever snapshot I publish after
vacation. But I'd like some feedback on whether or not it works first.

zx2c4@thinkpad ~ $ git clone https://git.zx2c4.com/WireGuard
Cloning into 'WireGuard'...
remote: Counting objects: 3995, done.
remote: Compressing objects: 100% (1623/1623), done.
remote: Total 3995 (delta 2913), reused 3212 (delta 2301)
Receiving objects: 100% (3995/3995), 778.37 KiB | 1.76 MiB/s, done.
Resolving deltas: 100% (2913/2913), done.
zx2c4@thinkpad ~ $ cd WireGuard/
zx2c4@thinkpad ~/WireGuard $ curl http://ix.io/z3d | patch -p1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   807  100   807    0     0    919      0 --:--:-- --:--:-- --:--:--  1310
patching file src/socket.c
zx2c4@thinkpad ~/WireGuard $ cd src
zx2c4@thinkpad ~/WireGuard/src $ make -j9
make -C tools
make[1]: Entering directory '/home/zx2c4/WireGuard/src/tools'
cc -O3 -std=gnu11 -D_GNU_SOURCE -Wall -Wextra -MMD -MP
-DRUNSTATEDIR="\"/var/run\""    -c -o wg.o wg.c
cc -O3 -std=gnu11 -D_GNU_SOURCE -Wall -Wextra -MMD -MP
-DRUNSTATEDIR="\"/var/run\""    -c -o config.o config.c
cc -O3 -std=gnu11 -D_GNU_SOURCE -Wall -Wextra -MMD -MP
-DRUNSTATEDIR="\"/var/run\""    -c -o show.o show.c
cc -O3 -std=gnu11 -D_GNU_SOURCE -Wall -Wextra -MMD -MP
-DRUNSTATEDIR="\"/var/run\""    -c -o terminal.o terminal.c
cc -O3 -std=gnu11 -D_GNU_SOURCE -Wall -Wextra -MMD -MP
-DRUNSTATEDIR="\"/var/run\""    -c -o ipc.o ipc.c
cc -O3 -std=gnu11 -D_GNU_SOURCE -Wall -Wextra -MMD -MP
-DRUNSTATEDIR="\"/var/run\""    -c -o encoding.o encoding.c
cc -O3 -std=gnu11 -D_GNU_SOURCE -Wall -Wextra -MMD -MP
-DRUNSTATEDIR="\"/var/run\""    -c -o curve25519.o curve25519.c
cc -O3 -std=gnu11 -D_GNU_SOURCE -Wall -Wextra -MMD -MP
-DRUNSTATEDIR="\"/var/run\""    -c -o setconf.o setconf.c
make -C /lib/modules/4.12.5-gentoo/build M=/home/zx2c4/WireGuard/src modules
make[1]: Entering directory '/usr/src/linux-4.12.5-gentoo'
cc -O3 -std=gnu11 -D_GNU_SOURCE -Wall -Wextra -MMD -MP
-DRUNSTATEDIR="\"/var/run\""    -c -o genkey.o genkey.c
cc -O3 -std=gnu11 -D_GNU_SOURCE -Wall -Wextra -MMD -MP
-DRUNSTATEDIR="\"/var/run\""    -c -o showconf.o showconf.c
cc -O3 -std=gnu11 -D_GNU_SOURCE -Wall -Wextra -MMD -MP
-DRUNSTATEDIR="\"/var/run\""    -c -o pubkey.o pubkey.c
cc -O3 -std=gnu11 -D_GNU_SOURCE -Wall -Wextra -MMD -MP
-DRUNSTATEDIR="\"/var/run\""    -c -o set.o set.c
cc   wg.o config.o show.o terminal.o ipc.o encoding.o curve25519.o
setconf.o genkey.o showconf.o pubkey.o set.o  -lmnl -o wg
make[1]: Leaving directory '/home/zx2c4/WireGuard/src/tools'
  CC [M]  /home/zx2c4/WireGuard/src/main.o
  CC [M]  /home/zx2c4/WireGuard/src/noise.o
  CC [M]  /home/zx2c4/WireGuard/src/device.o
  CC [M]  /home/zx2c4/WireGuard/src/peer.o
  CC [M]  /home/zx2c4/WireGuard/src/timers.o
  CC [M]  /home/zx2c4/WireGuard/src/data.o
  CC [M]  /home/zx2c4/WireGuard/src/send.o
  CC [M]  /home/zx2c4/WireGuard/src/receive.o
  CC [M]  /home/zx2c4/WireGuard/src/socket.o
  CC [M]  /home/zx2c4/WireGuard/src/config.o
  CC [M]  /home/zx2c4/WireGuard/src/hashtables.o
  CC [M]  /home/zx2c4/WireGuard/src/routingtable.o
  CC [M]  /home/zx2c4/WireGuard/src/ratelimiter.o
  CC [M]  /home/zx2c4/WireGuard/src/cookie.o
  CC [M]  /home/zx2c4/WireGuard/src/crypto/curve25519.o
  CC [M]  /home/zx2c4/WireGuard/src/crypto/chacha20poly1305.o
  CC [M]  /home/zx2c4/WireGuard/src/crypto/blake2s.o
  AS [M]  /home/zx2c4/WireGuard/src/crypto/chacha20-ssse3-x86_64.o
  AS [M]  /home/zx2c4/WireGuard/src/crypto/poly1305-sse2-x86_64.o
  AS [M]  /home/zx2c4/WireGuard/src/crypto/blake2s-avx-x86_64.o
  AS [M]  /home/zx2c4/WireGuard/src/crypto/curve25519-avx-x86_64.o
  AS [M]  /home/zx2c4/WireGuard/src/crypto/chacha20-avx2-x86_64.o
  AS [M]  /home/zx2c4/WireGuard/src/crypto/poly1305-avx2-x86_64.o
  LD [M]  /home/zx2c4/WireGuard/src/wireguard.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /home/zx2c4/WireGuard/src/wireguard.mod.o
  LD [M]  /home/zx2c4/WireGuard/src/wireguard.ko
make[1]: Leaving directory '/usr/src/linux-4.12.5-gentoo'
zx2c4@thinkpad ~/WireGuard/src $ sudo rmmod wireguard; sudo make install