Development discussion of WireGuard
 help / color / mirror / Atom feed
* Wireguard source ip selection issue with multi interfaces
@ 2021-11-19 13:10 曹煜
  2021-11-23 13:52 ` Christoph Loesch
  0 siblings, 1 reply; 2+ messages in thread
From: 曹煜 @ 2021-11-19 13:10 UTC (permalink / raw)
  To: wireguard

Hi,
As Jason said in mailing list
(https://lists.zx2c4.com/pipermail/wireguard/2017-November/002018.html)
years ago, if wireguard reply the client with wrong src ip, then it is
a bug. And now I'm trying to set up a minimal configuration.

Server info:

Ubuntu 20.04 in VMware with two Host-only network adapters:

wacke@Ubuntu:/etc$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal

wacke@Ubuntu:/etc$ uname -a
Linux Ubuntu 5.14.8-051408-generic #202109280455 SMP Tue Sep 28
04:57:31 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

wacke@Ubuntu:/etc$ sudo ip route show
169.254.0.0/16 dev ens33 scope link metric 1000
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.187.0/24 dev ens33 proto kernel scope link src 192.168.187.129
metric 100
192.168.187.0/24 dev ens35 proto kernel scope link src 192.168.187.128
metric 101
192.168.199.0/24 dev wg0 proto kernel scope link src 192.168.199.1

wacke@Ubuntu:/etc$ sudo cat /etc/wireguard/wg0.conf
[Interface]
PrivateKey = oLLeTlvwWsfGoyQpDwZo0k7AsMf0LkkfycGPwFjamEA=
ListenPort = 5999

[Peer]
PublicKey = JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
AllowedIPs = 192.168.199.2/32
PersistentKeepalive = 25

[Peer]
PublicKey = keIPXZFgvB79biV74kGc5R9vAHpzyVyQHci8KBSDIUw=
AllowedIPs = 192.168.199.3/32
PersistentKeepalive = 25

wacke@Ubuntu:/etc$ sudo wg setconf wg0 /etc/wireguard/wg0.conf
wacke@Ubuntu:/etc$ sudo ip address add dev wg0 192.168.199.1/24
wacke@Ubuntu:/etc$ sudo ip link set up dev wg0
wacke@Ubuntu:/etc$ sudo wg show
interface: wg0
  public key: Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
  private key: (hidden)
  listening port: 5999

peer: JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
  endpoint: 192.168.187.1:35560
  allowed ips: 192.168.199.2/32
  transfer: 2.89 KiB received, 1.80 KiB sent
  persistent keepalive: every 25 seconds

peer: keIPXZFgvB79biV74kGc5R9vAHpzyVyQHci8KBSDIUw=
  allowed ips: 192.168.199.3/32
  persistent keepalive: every 25 seconds

Client info:

HOME-Server:/NAS/Software/Software # uname -a
Linux HOME-Server 5.14.14-2-default #1 SMP Thu Oct 21 05:05:03 UTC
2021 (2b5383f) x86_64 x86_64 x86_64 GNU/Linux

HOME-Server:/NAS/Software/Software # cat wg0.conf
[Interface]
PrivateKey = 4Pw1cetxd9TdfH3TSab+9UcRBlHwZ1o/vmrUgkerZls=

[Peer]
PublicKey = Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
Endpoint = 192.168.187.128:5999
AllowedIPs = 192.168.199.0/24
PersistentKeepalive = 25

HOME-Server:/NAS/Software/Software # ip link add wg0 type wireguard
HOME-Server:/NAS/Software/Software # ip address add dev wg0 192.168.199.2/24
HOME-Server:/NAS/Software/Software # wg setconf wg0 wg0.conf
HOME-Server:/NAS/Software/Software # ip link set wg0 up
HOME-Server:/NAS/Software/Software # wg show
interface: wg0
  public key: JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
  private key: (hidden)
  listening port: 53058

peer: Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
  endpoint: 192.168.187.128:5999
  allowed ips: 192.168.199.0/24
  transfer: 0 B received, 59.26 KiB sent
  persistent keepalive: every 25 seconds


tcpdump of client side:

HOME-Server:/NAS/Software/Software # tcpdump -i any -vn "(dst port 5999)"
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2),
snapshot length 262144 bytes
22:10:37.794301 vmnet1 Out IP (tos 0x88, ttl 64, id 54724, offset 0,
flags [none], proto UDP (17), length 176)
    192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:10:43.170306 vmnet1 Out IP (tos 0x88, ttl 64, id 55327, offset 0,
flags [none], proto UDP (17), length 176)
    192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:10:48.546281 vmnet1 Out IP (tos 0x88, ttl 64, id 56142, offset 0,
flags [none], proto UDP (17), length 176)
    192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:10:53.670343 vmnet1 Out IP (tos 0x88, ttl 64, id 57053, offset 0,
flags [none], proto UDP (17), length 176)
    192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:11:18.754308 vmnet1 Out IP (tos 0x88, ttl 64, id 58505, offset 0,
flags [none], proto UDP (17), length 176)
    192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:11:24.130286 vmnet1 Out IP (tos 0x88, ttl 64, id 58621, offset 0,
flags [none], proto UDP (17), length 176)
    192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:11:29.506305 vmnet1 Out IP (tos 0x88, ttl 64, id 58810, offset 0,
flags [none], proto UDP (17), length 176)
    192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:11:34.882326 vmnet1 Out IP (tos 0x88, ttl 64, id 59444, offset 0,
flags [none], proto UDP (17), length 176)
    192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
^C
8 packets captured
8 packets received by filter
0 packets dropped by kernel

tcpdump of server side:

wacke@Ubuntu:~$ sudo tcpdump -i any -vn "(src port 5999)"
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1),
capture size 262144 bytes
22:10:11.754862 IP (tos 0x88, ttl 64, id 22024, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:16.874400 IP (tos 0x88, ttl 64, id 22399, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:21.994433 IP (tos 0x88, ttl 64, id 23057, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:27.370558 IP (tos 0x88, ttl 64, id 23912, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:32.494605 IP (tos 0x88, ttl 64, id 24944, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:37.610779 IP (tos 0x88, ttl 64, id 25327, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:42.986834 IP (tos 0x88, ttl 64, id 25498, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:48.362908 IP (tos 0x88, ttl 64, id 26521, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:53.486942 IP (tos 0x88, ttl 64, id 27214, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:11:18.571437 IP (tos 0x88, ttl 64, id 31112, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:11:23.947460 IP (tos 0x88, ttl 64, id 31897, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:11:29.323608 IP (tos 0x88, ttl 64, id 32715, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:11:34.699676 IP (tos 0x88, ttl 64, id 33840, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
^C
13 packets captured
13 packets received by filter
0 packets dropped by kernel

As the tcpdump log showed, the wireguard server always chose the ip
with lowest metric as source ip (192.168.187.129) to reply to the
client (while the client tried to connect 192.168.187.128).

Hope the info above can help to debug. Many thanks.

^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: Wireguard source ip selection issue with multi interfaces
  2021-11-19 13:10 Wireguard source ip selection issue with multi interfaces 曹煜
@ 2021-11-23 13:52 ` Christoph Loesch
  0 siblings, 0 replies; 2+ messages in thread
From: Christoph Loesch @ 2021-11-23 13:52 UTC (permalink / raw)
  To: wireguard

Hi,

Is this related to my issue with using wrong source ip on outgoing connections?
(https://lists.zx2c4.com/pipermail/wireguard/2021-November/007307.html)

Which bug would that be? Would be interesting in more detail to understand the issue.
I'm having a hard time to reproduce (or somehow "fix") the issue I am facing, since this issue is on only 5 devices from over 20 devices where all are configured same.

The configuration or better all routes do not even have any metric defined so I am unsure what to add/remove to change the behaviour.
Also in my configuration there are multiple interfaces (br0,br1,eth0,eth1,etc..) as it is a router but there are no multiple default routes.
Comparing perfectly fine working devices to one of those five devices which have this issue I am not able to find a difference in IP-config or routes.
Even kernel and firmware versions are all same.

The only difference I see is while looking at tcpdump where a simple ping to the wireguard server's internal IP (172.27.0.1/24 in my case) originates with the public IP of the device.

If it is a known bug, it would be nice to know what (and if) I could try to fix and possibly confirm the bug. Or even better beeing able to reproduce the issue.

Thanks and kind regards,
Christoph

Am 19.11.2021 um 14:10 schrieb 曹煜:
> Hi,
> As Jason said in mailing list
> (https://lists.zx2c4.com/pipermail/wireguard/2017-November/002018.html)
> years ago, if wireguard reply the client with wrong src ip, then it is
> a bug. And now I'm trying to set up a minimal configuration.
>
> Server info:
>
> Ubuntu 20.04 in VMware with two Host-only network adapters:
>
> wacke@Ubuntu:/etc$ lsb_release -a
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description: Ubuntu 20.04.3 LTS
> Release: 20.04
> Codename: focal
>
> wacke@Ubuntu:/etc$ uname -a
> Linux Ubuntu 5.14.8-051408-generic #202109280455 SMP Tue Sep 28
> 04:57:31 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>
> wacke@Ubuntu:/etc$ sudo ip route show
> 169.254.0.0/16 dev ens33 scope link metric 1000
> 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
> 192.168.187.0/24 dev ens33 proto kernel scope link src 192.168.187.129
> metric 100
> 192.168.187.0/24 dev ens35 proto kernel scope link src 192.168.187.128
> metric 101
> 192.168.199.0/24 dev wg0 proto kernel scope link src 192.168.199.1
>
> wacke@Ubuntu:/etc$ sudo cat /etc/wireguard/wg0.conf
> [Interface]
> PrivateKey = oLLeTlvwWsfGoyQpDwZo0k7AsMf0LkkfycGPwFjamEA=
> ListenPort = 5999
>
> [Peer]
> PublicKey = JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
> AllowedIPs = 192.168.199.2/32
> PersistentKeepalive = 25
>
> [Peer]
> PublicKey = keIPXZFgvB79biV74kGc5R9vAHpzyVyQHci8KBSDIUw=
> AllowedIPs = 192.168.199.3/32
> PersistentKeepalive = 25
>
> wacke@Ubuntu:/etc$ sudo wg setconf wg0 /etc/wireguard/wg0.conf
> wacke@Ubuntu:/etc$ sudo ip address add dev wg0 192.168.199.1/24
> wacke@Ubuntu:/etc$ sudo ip link set up dev wg0
> wacke@Ubuntu:/etc$ sudo wg show
> interface: wg0
>    public key: Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
>    private key: (hidden)
>    listening port: 5999
>
> peer: JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
>    endpoint: 192.168.187.1:35560
>    allowed ips: 192.168.199.2/32
>    transfer: 2.89 KiB received, 1.80 KiB sent
>    persistent keepalive: every 25 seconds
>
> peer: keIPXZFgvB79biV74kGc5R9vAHpzyVyQHci8KBSDIUw=
>    allowed ips: 192.168.199.3/32
>    persistent keepalive: every 25 seconds
>
> Client info:
>
> HOME-Server:/NAS/Software/Software # uname -a
> Linux HOME-Server 5.14.14-2-default #1 SMP Thu Oct 21 05:05:03 UTC
> 2021 (2b5383f) x86_64 x86_64 x86_64 GNU/Linux
>
> HOME-Server:/NAS/Software/Software # cat wg0.conf
> [Interface]
> PrivateKey = 4Pw1cetxd9TdfH3TSab+9UcRBlHwZ1o/vmrUgkerZls=
>
> [Peer]
> PublicKey = Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
> Endpoint = 192.168.187.128:5999
> AllowedIPs = 192.168.199.0/24
> PersistentKeepalive = 25
>
> HOME-Server:/NAS/Software/Software # ip link add wg0 type wireguard
> HOME-Server:/NAS/Software/Software # ip address add dev wg0 192.168.199.2/24
> HOME-Server:/NAS/Software/Software # wg setconf wg0 wg0.conf
> HOME-Server:/NAS/Software/Software # ip link set wg0 up
> HOME-Server:/NAS/Software/Software # wg show
> interface: wg0
>    public key: JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
>    private key: (hidden)
>    listening port: 53058
>
> peer: Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
>    endpoint: 192.168.187.128:5999
>    allowed ips: 192.168.199.0/24
>    transfer: 0 B received, 59.26 KiB sent
>    persistent keepalive: every 25 seconds
>
>
> tcpdump of client side:
>
> HOME-Server:/NAS/Software/Software # tcpdump -i any -vn "(dst port 5999)"
> tcpdump: data link type LINUX_SLL2
> tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2),
> snapshot length 262144 bytes
> 22:10:37.794301 vmnet1 Out IP (tos 0x88, ttl 64, id 54724, offset 0,
> flags [none], proto UDP (17), length 176)
>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
> 22:10:43.170306 vmnet1 Out IP (tos 0x88, ttl 64, id 55327, offset 0,
> flags [none], proto UDP (17), length 176)
>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
> 22:10:48.546281 vmnet1 Out IP (tos 0x88, ttl 64, id 56142, offset 0,
> flags [none], proto UDP (17), length 176)
>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
> 22:10:53.670343 vmnet1 Out IP (tos 0x88, ttl 64, id 57053, offset 0,
> flags [none], proto UDP (17), length 176)
>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
> 22:11:18.754308 vmnet1 Out IP (tos 0x88, ttl 64, id 58505, offset 0,
> flags [none], proto UDP (17), length 176)
>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
> 22:11:24.130286 vmnet1 Out IP (tos 0x88, ttl 64, id 58621, offset 0,
> flags [none], proto UDP (17), length 176)
>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
> 22:11:29.506305 vmnet1 Out IP (tos 0x88, ttl 64, id 58810, offset 0,
> flags [none], proto UDP (17), length 176)
>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
> 22:11:34.882326 vmnet1 Out IP (tos 0x88, ttl 64, id 59444, offset 0,
> flags [none], proto UDP (17), length 176)
>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
> ^C
> 8 packets captured
> 8 packets received by filter
> 0 packets dropped by kernel
>
> tcpdump of server side:
>
> wacke@Ubuntu:~$ sudo tcpdump -i any -vn "(src port 5999)"
> tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1),
> capture size 262144 bytes
> 22:10:11.754862 IP (tos 0x88, ttl 64, id 22024, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:10:16.874400 IP (tos 0x88, ttl 64, id 22399, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:10:21.994433 IP (tos 0x88, ttl 64, id 23057, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:10:27.370558 IP (tos 0x88, ttl 64, id 23912, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:10:32.494605 IP (tos 0x88, ttl 64, id 24944, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:10:37.610779 IP (tos 0x88, ttl 64, id 25327, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:10:42.986834 IP (tos 0x88, ttl 64, id 25498, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:10:48.362908 IP (tos 0x88, ttl 64, id 26521, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:10:53.486942 IP (tos 0x88, ttl 64, id 27214, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:11:18.571437 IP (tos 0x88, ttl 64, id 31112, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:11:23.947460 IP (tos 0x88, ttl 64, id 31897, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:11:29.323608 IP (tos 0x88, ttl 64, id 32715, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:11:34.699676 IP (tos 0x88, ttl 64, id 33840, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> ^C
> 13 packets captured
> 13 packets received by filter
> 0 packets dropped by kernel
>
> As the tcpdump log showed, the wireguard server always chose the ip
> with lowest metric as source ip (192.168.187.129) to reply to the
> client (while the client tried to connect 192.168.187.128).
>
> Hope the info above can help to debug. Many thanks.

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2021-11-23 13:52 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-19 13:10 Wireguard source ip selection issue with multi interfaces 曹煜
2021-11-23 13:52 ` Christoph Loesch

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).