Development discussion of WireGuard
 help / color / mirror / Atom feed
* Wireguard source ip selection issue with multi interfaces
@ 2021-11-19 13:10 曹煜
  2021-11-23 13:52 ` Christoph Loesch
  0 siblings, 1 reply; 7+ messages in thread
From: 曹煜 @ 2021-11-19 13:10 UTC (permalink / raw)
  To: wireguard

Hi,
As Jason said in mailing list
(https://lists.zx2c4.com/pipermail/wireguard/2017-November/002018.html)
years ago, if wireguard reply the client with wrong src ip, then it is
a bug. And now I'm trying to set up a minimal configuration.

Server info:

Ubuntu 20.04 in VMware with two Host-only network adapters:

wacke@Ubuntu:/etc$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal

wacke@Ubuntu:/etc$ uname -a
Linux Ubuntu 5.14.8-051408-generic #202109280455 SMP Tue Sep 28
04:57:31 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

wacke@Ubuntu:/etc$ sudo ip route show
169.254.0.0/16 dev ens33 scope link metric 1000
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.187.0/24 dev ens33 proto kernel scope link src 192.168.187.129
metric 100
192.168.187.0/24 dev ens35 proto kernel scope link src 192.168.187.128
metric 101
192.168.199.0/24 dev wg0 proto kernel scope link src 192.168.199.1

wacke@Ubuntu:/etc$ sudo cat /etc/wireguard/wg0.conf
[Interface]
PrivateKey = oLLeTlvwWsfGoyQpDwZo0k7AsMf0LkkfycGPwFjamEA=
ListenPort = 5999

[Peer]
PublicKey = JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
AllowedIPs = 192.168.199.2/32
PersistentKeepalive = 25

[Peer]
PublicKey = keIPXZFgvB79biV74kGc5R9vAHpzyVyQHci8KBSDIUw=
AllowedIPs = 192.168.199.3/32
PersistentKeepalive = 25

wacke@Ubuntu:/etc$ sudo wg setconf wg0 /etc/wireguard/wg0.conf
wacke@Ubuntu:/etc$ sudo ip address add dev wg0 192.168.199.1/24
wacke@Ubuntu:/etc$ sudo ip link set up dev wg0
wacke@Ubuntu:/etc$ sudo wg show
interface: wg0
  public key: Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
  private key: (hidden)
  listening port: 5999

peer: JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
  endpoint: 192.168.187.1:35560
  allowed ips: 192.168.199.2/32
  transfer: 2.89 KiB received, 1.80 KiB sent
  persistent keepalive: every 25 seconds

peer: keIPXZFgvB79biV74kGc5R9vAHpzyVyQHci8KBSDIUw=
  allowed ips: 192.168.199.3/32
  persistent keepalive: every 25 seconds

Client info:

HOME-Server:/NAS/Software/Software # uname -a
Linux HOME-Server 5.14.14-2-default #1 SMP Thu Oct 21 05:05:03 UTC
2021 (2b5383f) x86_64 x86_64 x86_64 GNU/Linux

HOME-Server:/NAS/Software/Software # cat wg0.conf
[Interface]
PrivateKey = 4Pw1cetxd9TdfH3TSab+9UcRBlHwZ1o/vmrUgkerZls=

[Peer]
PublicKey = Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
Endpoint = 192.168.187.128:5999
AllowedIPs = 192.168.199.0/24
PersistentKeepalive = 25

HOME-Server:/NAS/Software/Software # ip link add wg0 type wireguard
HOME-Server:/NAS/Software/Software # ip address add dev wg0 192.168.199.2/24
HOME-Server:/NAS/Software/Software # wg setconf wg0 wg0.conf
HOME-Server:/NAS/Software/Software # ip link set wg0 up
HOME-Server:/NAS/Software/Software # wg show
interface: wg0
  public key: JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
  private key: (hidden)
  listening port: 53058

peer: Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
  endpoint: 192.168.187.128:5999
  allowed ips: 192.168.199.0/24
  transfer: 0 B received, 59.26 KiB sent
  persistent keepalive: every 25 seconds


tcpdump of client side:

HOME-Server:/NAS/Software/Software # tcpdump -i any -vn "(dst port 5999)"
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2),
snapshot length 262144 bytes
22:10:37.794301 vmnet1 Out IP (tos 0x88, ttl 64, id 54724, offset 0,
flags [none], proto UDP (17), length 176)
    192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:10:43.170306 vmnet1 Out IP (tos 0x88, ttl 64, id 55327, offset 0,
flags [none], proto UDP (17), length 176)
    192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:10:48.546281 vmnet1 Out IP (tos 0x88, ttl 64, id 56142, offset 0,
flags [none], proto UDP (17), length 176)
    192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:10:53.670343 vmnet1 Out IP (tos 0x88, ttl 64, id 57053, offset 0,
flags [none], proto UDP (17), length 176)
    192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:11:18.754308 vmnet1 Out IP (tos 0x88, ttl 64, id 58505, offset 0,
flags [none], proto UDP (17), length 176)
    192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:11:24.130286 vmnet1 Out IP (tos 0x88, ttl 64, id 58621, offset 0,
flags [none], proto UDP (17), length 176)
    192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:11:29.506305 vmnet1 Out IP (tos 0x88, ttl 64, id 58810, offset 0,
flags [none], proto UDP (17), length 176)
    192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
22:11:34.882326 vmnet1 Out IP (tos 0x88, ttl 64, id 59444, offset 0,
flags [none], proto UDP (17), length 176)
    192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
^C
8 packets captured
8 packets received by filter
0 packets dropped by kernel

tcpdump of server side:

wacke@Ubuntu:~$ sudo tcpdump -i any -vn "(src port 5999)"
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1),
capture size 262144 bytes
22:10:11.754862 IP (tos 0x88, ttl 64, id 22024, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:16.874400 IP (tos 0x88, ttl 64, id 22399, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:21.994433 IP (tos 0x88, ttl 64, id 23057, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:27.370558 IP (tos 0x88, ttl 64, id 23912, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:32.494605 IP (tos 0x88, ttl 64, id 24944, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:37.610779 IP (tos 0x88, ttl 64, id 25327, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:42.986834 IP (tos 0x88, ttl 64, id 25498, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:48.362908 IP (tos 0x88, ttl 64, id 26521, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:10:53.486942 IP (tos 0x88, ttl 64, id 27214, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:11:18.571437 IP (tos 0x88, ttl 64, id 31112, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:11:23.947460 IP (tos 0x88, ttl 64, id 31897, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:11:29.323608 IP (tos 0x88, ttl 64, id 32715, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
22:11:34.699676 IP (tos 0x88, ttl 64, id 33840, offset 0, flags
[none], proto UDP (17), length 120)
    192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
^C
13 packets captured
13 packets received by filter
0 packets dropped by kernel

As the tcpdump log showed, the wireguard server always chose the ip
with lowest metric as source ip (192.168.187.129) to reply to the
client (while the client tried to connect 192.168.187.128).

Hope the info above can help to debug. Many thanks.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Wireguard source ip selection issue with multi interfaces
  2021-11-19 13:10 Wireguard source ip selection issue with multi interfaces 曹煜
@ 2021-11-23 13:52 ` Christoph Loesch
  2022-10-19 22:47   ` Source IP selection or multi-home or multi-interface issue - solved with source-based route - was: " Christoph Loesch
  2022-10-19 22:56   ` Christoph Loesch
  0 siblings, 2 replies; 7+ messages in thread
From: Christoph Loesch @ 2021-11-23 13:52 UTC (permalink / raw)
  To: wireguard

Hi,

Is this related to my issue with using wrong source ip on outgoing connections?
(https://lists.zx2c4.com/pipermail/wireguard/2021-November/007307.html)

Which bug would that be? Would be interesting in more detail to understand the issue.
I'm having a hard time to reproduce (or somehow "fix") the issue I am facing, since this issue is on only 5 devices from over 20 devices where all are configured same.

The configuration or better all routes do not even have any metric defined so I am unsure what to add/remove to change the behaviour.
Also in my configuration there are multiple interfaces (br0,br1,eth0,eth1,etc..) as it is a router but there are no multiple default routes.
Comparing perfectly fine working devices to one of those five devices which have this issue I am not able to find a difference in IP-config or routes.
Even kernel and firmware versions are all same.

The only difference I see is while looking at tcpdump where a simple ping to the wireguard server's internal IP (172.27.0.1/24 in my case) originates with the public IP of the device.

If it is a known bug, it would be nice to know what (and if) I could try to fix and possibly confirm the bug. Or even better beeing able to reproduce the issue.

Thanks and kind regards,
Christoph

Am 19.11.2021 um 14:10 schrieb 曹煜:
> Hi,
> As Jason said in mailing list
> (https://lists.zx2c4.com/pipermail/wireguard/2017-November/002018.html)
> years ago, if wireguard reply the client with wrong src ip, then it is
> a bug. And now I'm trying to set up a minimal configuration.
>
> Server info:
>
> Ubuntu 20.04 in VMware with two Host-only network adapters:
>
> wacke@Ubuntu:/etc$ lsb_release -a
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description: Ubuntu 20.04.3 LTS
> Release: 20.04
> Codename: focal
>
> wacke@Ubuntu:/etc$ uname -a
> Linux Ubuntu 5.14.8-051408-generic #202109280455 SMP Tue Sep 28
> 04:57:31 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>
> wacke@Ubuntu:/etc$ sudo ip route show
> 169.254.0.0/16 dev ens33 scope link metric 1000
> 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
> 192.168.187.0/24 dev ens33 proto kernel scope link src 192.168.187.129
> metric 100
> 192.168.187.0/24 dev ens35 proto kernel scope link src 192.168.187.128
> metric 101
> 192.168.199.0/24 dev wg0 proto kernel scope link src 192.168.199.1
>
> wacke@Ubuntu:/etc$ sudo cat /etc/wireguard/wg0.conf
> [Interface]
> PrivateKey = oLLeTlvwWsfGoyQpDwZo0k7AsMf0LkkfycGPwFjamEA=
> ListenPort = 5999
>
> [Peer]
> PublicKey = JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
> AllowedIPs = 192.168.199.2/32
> PersistentKeepalive = 25
>
> [Peer]
> PublicKey = keIPXZFgvB79biV74kGc5R9vAHpzyVyQHci8KBSDIUw=
> AllowedIPs = 192.168.199.3/32
> PersistentKeepalive = 25
>
> wacke@Ubuntu:/etc$ sudo wg setconf wg0 /etc/wireguard/wg0.conf
> wacke@Ubuntu:/etc$ sudo ip address add dev wg0 192.168.199.1/24
> wacke@Ubuntu:/etc$ sudo ip link set up dev wg0
> wacke@Ubuntu:/etc$ sudo wg show
> interface: wg0
>    public key: Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
>    private key: (hidden)
>    listening port: 5999
>
> peer: JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
>    endpoint: 192.168.187.1:35560
>    allowed ips: 192.168.199.2/32
>    transfer: 2.89 KiB received, 1.80 KiB sent
>    persistent keepalive: every 25 seconds
>
> peer: keIPXZFgvB79biV74kGc5R9vAHpzyVyQHci8KBSDIUw=
>    allowed ips: 192.168.199.3/32
>    persistent keepalive: every 25 seconds
>
> Client info:
>
> HOME-Server:/NAS/Software/Software # uname -a
> Linux HOME-Server 5.14.14-2-default #1 SMP Thu Oct 21 05:05:03 UTC
> 2021 (2b5383f) x86_64 x86_64 x86_64 GNU/Linux
>
> HOME-Server:/NAS/Software/Software # cat wg0.conf
> [Interface]
> PrivateKey = 4Pw1cetxd9TdfH3TSab+9UcRBlHwZ1o/vmrUgkerZls=
>
> [Peer]
> PublicKey = Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
> Endpoint = 192.168.187.128:5999
> AllowedIPs = 192.168.199.0/24
> PersistentKeepalive = 25
>
> HOME-Server:/NAS/Software/Software # ip link add wg0 type wireguard
> HOME-Server:/NAS/Software/Software # ip address add dev wg0 192.168.199.2/24
> HOME-Server:/NAS/Software/Software # wg setconf wg0 wg0.conf
> HOME-Server:/NAS/Software/Software # ip link set wg0 up
> HOME-Server:/NAS/Software/Software # wg show
> interface: wg0
>    public key: JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
>    private key: (hidden)
>    listening port: 53058
>
> peer: Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
>    endpoint: 192.168.187.128:5999
>    allowed ips: 192.168.199.0/24
>    transfer: 0 B received, 59.26 KiB sent
>    persistent keepalive: every 25 seconds
>
>
> tcpdump of client side:
>
> HOME-Server:/NAS/Software/Software # tcpdump -i any -vn "(dst port 5999)"
> tcpdump: data link type LINUX_SLL2
> tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2),
> snapshot length 262144 bytes
> 22:10:37.794301 vmnet1 Out IP (tos 0x88, ttl 64, id 54724, offset 0,
> flags [none], proto UDP (17), length 176)
>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
> 22:10:43.170306 vmnet1 Out IP (tos 0x88, ttl 64, id 55327, offset 0,
> flags [none], proto UDP (17), length 176)
>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
> 22:10:48.546281 vmnet1 Out IP (tos 0x88, ttl 64, id 56142, offset 0,
> flags [none], proto UDP (17), length 176)
>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
> 22:10:53.670343 vmnet1 Out IP (tos 0x88, ttl 64, id 57053, offset 0,
> flags [none], proto UDP (17), length 176)
>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
> 22:11:18.754308 vmnet1 Out IP (tos 0x88, ttl 64, id 58505, offset 0,
> flags [none], proto UDP (17), length 176)
>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
> 22:11:24.130286 vmnet1 Out IP (tos 0x88, ttl 64, id 58621, offset 0,
> flags [none], proto UDP (17), length 176)
>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
> 22:11:29.506305 vmnet1 Out IP (tos 0x88, ttl 64, id 58810, offset 0,
> flags [none], proto UDP (17), length 176)
>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
> 22:11:34.882326 vmnet1 Out IP (tos 0x88, ttl 64, id 59444, offset 0,
> flags [none], proto UDP (17), length 176)
>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
> ^C
> 8 packets captured
> 8 packets received by filter
> 0 packets dropped by kernel
>
> tcpdump of server side:
>
> wacke@Ubuntu:~$ sudo tcpdump -i any -vn "(src port 5999)"
> tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1),
> capture size 262144 bytes
> 22:10:11.754862 IP (tos 0x88, ttl 64, id 22024, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:10:16.874400 IP (tos 0x88, ttl 64, id 22399, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:10:21.994433 IP (tos 0x88, ttl 64, id 23057, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:10:27.370558 IP (tos 0x88, ttl 64, id 23912, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:10:32.494605 IP (tos 0x88, ttl 64, id 24944, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:10:37.610779 IP (tos 0x88, ttl 64, id 25327, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:10:42.986834 IP (tos 0x88, ttl 64, id 25498, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:10:48.362908 IP (tos 0x88, ttl 64, id 26521, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:10:53.486942 IP (tos 0x88, ttl 64, id 27214, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:11:18.571437 IP (tos 0x88, ttl 64, id 31112, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:11:23.947460 IP (tos 0x88, ttl 64, id 31897, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:11:29.323608 IP (tos 0x88, ttl 64, id 32715, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> 22:11:34.699676 IP (tos 0x88, ttl 64, id 33840, offset 0, flags
> [none], proto UDP (17), length 120)
>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
> ^C
> 13 packets captured
> 13 packets received by filter
> 0 packets dropped by kernel
>
> As the tcpdump log showed, the wireguard server always chose the ip
> with lowest metric as source ip (192.168.187.129) to reply to the
> client (while the client tried to connect 192.168.187.128).
>
> Hope the info above can help to debug. Many thanks.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Source IP selection or multi-home or multi-interface issue - solved with source-based route - was: Re: Wireguard source ip selection issue with multi interfaces
  2021-11-23 13:52 ` Christoph Loesch
@ 2022-10-19 22:47   ` Christoph Loesch
  2022-10-19 22:56   ` Christoph Loesch
  1 sibling, 0 replies; 7+ messages in thread
From: Christoph Loesch @ 2022-10-19 22:47 UTC (permalink / raw)
  To: wireguard

Now I've read enough (rather old and also pretty recent) posts about that topic and I want to help finally fix this issue.

Not just because it bugs our network too after we optimized/simplified our router-setup but also because it seems over the years many users seem to be affected. Until now every thread I've read ends with an open question rather than a solution.

In Jason's post here from 2017 I found his script that should reproduce the issue, I tried it and could not reproduce the issue with that script..
To be honest I do not fully understand it to modify it to reproduce the issue in my environment.

Here I posted one example and the (temporary) solution by deleting the route that wireguard had set and re-creating the same route but defining the (correct) IP as src.
https://lists.zx2c4.com/pipermail/wireguard/2021-November/007324.html

How could I help to fix this issue?
If it helps, I could give full access to a test-router which has the same setup as all other routers in our network.

Kind regards


Am 23.11.2021 um 14:52 schrieb Christoph Loesch:
> Hi,
>
> Is this related to my issue with using wrong source ip on outgoing connections?
> (https://lists.zx2c4.com/pipermail/wireguard/2021-November/007307.html)
>
> Which bug would that be? Would be interesting in more detail to understand the issue.
> I'm having a hard time to reproduce (or somehow "fix") the issue I am facing, since this issue is on only 5 devices from over 20 devices where all are configured same.
>
> The configuration or better all routes do not even have any metric defined so I am unsure what to add/remove to change the behaviour.
> Also in my configuration there are multiple interfaces (br0,br1,eth0,eth1,etc..) as it is a router but there are no multiple default routes.
> Comparing perfectly fine working devices to one of those five devices which have this issue I am not able to find a difference in IP-config or routes.
> Even kernel and firmware versions are all same.
>
> The only difference I see is while looking at tcpdump where a simple ping to the wireguard server's internal IP (172.27.0.1/24 in my case) originates with the public IP of the device.
>
> If it is a known bug, it would be nice to know what (and if) I could try to fix and possibly confirm the bug. Or even better beeing able to reproduce the issue.
>
> Thanks and kind regards,
> Christoph
>
> Am 19.11.2021 um 14:10 schrieb 曹煜:
>> Hi,
>> As Jason said in mailing list
>> (https://lists.zx2c4.com/pipermail/wireguard/2017-November/002018.html)
>> years ago, if wireguard reply the client with wrong src ip, then it is
>> a bug. And now I'm trying to set up a minimal configuration.
>>
>> Server info:
>>
>> Ubuntu 20.04 in VMware with two Host-only network adapters:
>>
>> wacke@Ubuntu:/etc$ lsb_release -a
>> No LSB modules are available.
>> Distributor ID: Ubuntu
>> Description: Ubuntu 20.04.3 LTS
>> Release: 20.04
>> Codename: focal
>>
>> wacke@Ubuntu:/etc$ uname -a
>> Linux Ubuntu 5.14.8-051408-generic #202109280455 SMP Tue Sep 28
>> 04:57:31 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>>
>> wacke@Ubuntu:/etc$ sudo ip route show
>> 169.254.0.0/16 dev ens33 scope link metric 1000
>> 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
>> 192.168.187.0/24 dev ens33 proto kernel scope link src 192.168.187.129
>> metric 100
>> 192.168.187.0/24 dev ens35 proto kernel scope link src 192.168.187.128
>> metric 101
>> 192.168.199.0/24 dev wg0 proto kernel scope link src 192.168.199.1
>>
>> wacke@Ubuntu:/etc$ sudo cat /etc/wireguard/wg0.conf
>> [Interface]
>> PrivateKey = oLLeTlvwWsfGoyQpDwZo0k7AsMf0LkkfycGPwFjamEA=
>> ListenPort = 5999
>>
>> [Peer]
>> PublicKey = JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
>> AllowedIPs = 192.168.199.2/32
>> PersistentKeepalive = 25
>>
>> [Peer]
>> PublicKey = keIPXZFgvB79biV74kGc5R9vAHpzyVyQHci8KBSDIUw=
>> AllowedIPs = 192.168.199.3/32
>> PersistentKeepalive = 25
>>
>> wacke@Ubuntu:/etc$ sudo wg setconf wg0 /etc/wireguard/wg0.conf
>> wacke@Ubuntu:/etc$ sudo ip address add dev wg0 192.168.199.1/24
>> wacke@Ubuntu:/etc$ sudo ip link set up dev wg0
>> wacke@Ubuntu:/etc$ sudo wg show
>> interface: wg0
>>    public key: Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
>>    private key: (hidden)
>>    listening port: 5999
>>
>> peer: JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
>>    endpoint: 192.168.187.1:35560
>>    allowed ips: 192.168.199.2/32
>>    transfer: 2.89 KiB received, 1.80 KiB sent
>>    persistent keepalive: every 25 seconds
>>
>> peer: keIPXZFgvB79biV74kGc5R9vAHpzyVyQHci8KBSDIUw=
>>    allowed ips: 192.168.199.3/32
>>    persistent keepalive: every 25 seconds
>>
>> Client info:
>>
>> HOME-Server:/NAS/Software/Software # uname -a
>> Linux HOME-Server 5.14.14-2-default #1 SMP Thu Oct 21 05:05:03 UTC
>> 2021 (2b5383f) x86_64 x86_64 x86_64 GNU/Linux
>>
>> HOME-Server:/NAS/Software/Software # cat wg0.conf
>> [Interface]
>> PrivateKey = 4Pw1cetxd9TdfH3TSab+9UcRBlHwZ1o/vmrUgkerZls=
>>
>> [Peer]
>> PublicKey = Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
>> Endpoint = 192.168.187.128:5999
>> AllowedIPs = 192.168.199.0/24
>> PersistentKeepalive = 25
>>
>> HOME-Server:/NAS/Software/Software # ip link add wg0 type wireguard
>> HOME-Server:/NAS/Software/Software # ip address add dev wg0 192.168.199.2/24
>> HOME-Server:/NAS/Software/Software # wg setconf wg0 wg0.conf
>> HOME-Server:/NAS/Software/Software # ip link set wg0 up
>> HOME-Server:/NAS/Software/Software # wg show
>> interface: wg0
>>    public key: JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
>>    private key: (hidden)
>>    listening port: 53058
>>
>> peer: Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
>>    endpoint: 192.168.187.128:5999
>>    allowed ips: 192.168.199.0/24
>>    transfer: 0 B received, 59.26 KiB sent
>>    persistent keepalive: every 25 seconds
>>
>>
>> tcpdump of client side:
>>
>> HOME-Server:/NAS/Software/Software # tcpdump -i any -vn "(dst port 5999)"
>> tcpdump: data link type LINUX_SLL2
>> tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2),
>> snapshot length 262144 bytes
>> 22:10:37.794301 vmnet1 Out IP (tos 0x88, ttl 64, id 54724, offset 0,
>> flags [none], proto UDP (17), length 176)
>>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>> 22:10:43.170306 vmnet1 Out IP (tos 0x88, ttl 64, id 55327, offset 0,
>> flags [none], proto UDP (17), length 176)
>>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>> 22:10:48.546281 vmnet1 Out IP (tos 0x88, ttl 64, id 56142, offset 0,
>> flags [none], proto UDP (17), length 176)
>>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>> 22:10:53.670343 vmnet1 Out IP (tos 0x88, ttl 64, id 57053, offset 0,
>> flags [none], proto UDP (17), length 176)
>>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>> 22:11:18.754308 vmnet1 Out IP (tos 0x88, ttl 64, id 58505, offset 0,
>> flags [none], proto UDP (17), length 176)
>>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>> 22:11:24.130286 vmnet1 Out IP (tos 0x88, ttl 64, id 58621, offset 0,
>> flags [none], proto UDP (17), length 176)
>>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>> 22:11:29.506305 vmnet1 Out IP (tos 0x88, ttl 64, id 58810, offset 0,
>> flags [none], proto UDP (17), length 176)
>>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>> 22:11:34.882326 vmnet1 Out IP (tos 0x88, ttl 64, id 59444, offset 0,
>> flags [none], proto UDP (17), length 176)
>>      192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>> ^C
>> 8 packets captured
>> 8 packets received by filter
>> 0 packets dropped by kernel
>>
>> tcpdump of server side:
>>
>> wacke@Ubuntu:~$ sudo tcpdump -i any -vn "(src port 5999)"
>> tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1),
>> capture size 262144 bytes
>> 22:10:11.754862 IP (tos 0x88, ttl 64, id 22024, offset 0, flags
>> [none], proto UDP (17), length 120)
>>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:10:16.874400 IP (tos 0x88, ttl 64, id 22399, offset 0, flags
>> [none], proto UDP (17), length 120)
>>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:10:21.994433 IP (tos 0x88, ttl 64, id 23057, offset 0, flags
>> [none], proto UDP (17), length 120)
>>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:10:27.370558 IP (tos 0x88, ttl 64, id 23912, offset 0, flags
>> [none], proto UDP (17), length 120)
>>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:10:32.494605 IP (tos 0x88, ttl 64, id 24944, offset 0, flags
>> [none], proto UDP (17), length 120)
>>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:10:37.610779 IP (tos 0x88, ttl 64, id 25327, offset 0, flags
>> [none], proto UDP (17), length 120)
>>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:10:42.986834 IP (tos 0x88, ttl 64, id 25498, offset 0, flags
>> [none], proto UDP (17), length 120)
>>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:10:48.362908 IP (tos 0x88, ttl 64, id 26521, offset 0, flags
>> [none], proto UDP (17), length 120)
>>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:10:53.486942 IP (tos 0x88, ttl 64, id 27214, offset 0, flags
>> [none], proto UDP (17), length 120)
>>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:11:18.571437 IP (tos 0x88, ttl 64, id 31112, offset 0, flags
>> [none], proto UDP (17), length 120)
>>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:11:23.947460 IP (tos 0x88, ttl 64, id 31897, offset 0, flags
>> [none], proto UDP (17), length 120)
>>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:11:29.323608 IP (tos 0x88, ttl 64, id 32715, offset 0, flags
>> [none], proto UDP (17), length 120)
>>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:11:34.699676 IP (tos 0x88, ttl 64, id 33840, offset 0, flags
>> [none], proto UDP (17), length 120)
>>      192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> ^C
>> 13 packets captured
>> 13 packets received by filter
>> 0 packets dropped by kernel
>>
>> As the tcpdump log showed, the wireguard server always chose the ip
>> with lowest metric as source ip (192.168.187.129) to reply to the
>> client (while the client tried to connect 192.168.187.128).
>>
>> Hope the info above can help to debug. Many thanks.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Source IP selection or multi-home or multi-interface issue - solved with source-based route - was: Re: Wireguard source ip selection issue with multi interfaces
  2021-11-23 13:52 ` Christoph Loesch
  2022-10-19 22:47   ` Source IP selection or multi-home or multi-interface issue - solved with source-based route - was: " Christoph Loesch
@ 2022-10-19 22:56   ` Christoph Loesch
  2022-10-21 13:11     ` Christoph Loesch
  1 sibling, 1 reply; 7+ messages in thread
From: Christoph Loesch @ 2022-10-19 22:56 UTC (permalink / raw)
  To: wireguard

Now I've read enough (rather old and also pretty recent) posts about that topic and I want to help finally fix this issue.

Not just because it bugs our network too after we optimized/simplified our router-setup but also because it seems over the years many users seem to beaffected. Until now every thread I've read ends with an open question rather than a solution.

In Jason's post here from 2017 I found his script that should reproduce theissue, I tried it and could not reproduce the issue with that script..
https://lists.zx2c4.com/pipermail/wireguard/2017-November/002092.html
To be honest I do not fully understand it to modify it to reproduce the issue in my environment.

Here I posted one example and the (temporary) solution by deleting the route that wireguard had set and re-creating the same route but defining the (correct) IP as src.
https://lists.zx2c4.com/pipermail/wireguard/2021-November/007324.html

How could I help to fix this issue?
If it helps, I could give full access to a test-router which has the same setup as all other routers in our network.

Kind regards


Am 23.11.2021 um 14:52 schrieb Christoph Loesch:
> Hi,
>
> Is this related to my issue with using wrong source ip on outgoing connections?
> (https://lists.zx2c4.com/pipermail/wireguard/2021-November/007307.html)
>
> Which bug would that be? Would be interesting in more detail to understand the issue.
> I'm having a hard time to reproduce (or somehow "fix") the issue I am facing, since this issue is on only 5 devices from over 20 devices where all are configured same.
>
> The configuration or better all routes do not even have any metric defined so I am unsure what to add/remove to change the behaviour.
> Also in my configuration there are multiple interfaces (br0,br1,eth0,eth1,etc..) as it is a router but there are no multiple default routes.
> Comparing perfectly fine working devices to one of those five devices which have this issue I am not able to find a difference in IP-config or routes.
> Even kernel and firmware versions are all same.
>
> The only difference I see is while looking at tcpdump where a simple pingto the wireguard server's internal IP (172.27.0.1/24 in my case) originates with the public IP of the device.
>
> If it is a known bug, it would be nice to know what (and if) I could try to fix and possibly confirm the bug. Or even better beeing able to reproduce the issue.
>
> Thanks and kind regards,
> Christoph
>
> Am 19.11.2021 um 14:10 schrieb 曹煜:
>> Hi,
>> As Jason said in mailing list
>> (https://lists.zx2c4.com/pipermail/wireguard/2017-November/002018.html)
>> years ago, if wireguard reply the client with wrong src ip, then it is
>> a bug. And now I'm trying to set up a minimal configuration.
>>
>> Server info:
>>
>> Ubuntu 20.04 in VMware with two Host-only network adapters:
>>
>> wacke@Ubuntu:/etc$ lsb_release -a
>> No LSB modules are available.
>> Distributor ID: Ubuntu
>> Description: Ubuntu 20.04.3 LTS
>> Release: 20.04
>> Codename: focal
>>
>> wacke@Ubuntu:/etc$ uname -a
>> Linux Ubuntu 5.14.8-051408-generic #202109280455 SMP Tue Sep 28
>> 04:57:31 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>>
>> wacke@Ubuntu:/etc$ sudo ip route show
>> 169.254.0.0/16 dev ens33 scope link metric 1000
>> 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
>> 192.168.187.0/24 dev ens33 proto kernel scope link src 192.168.187.129
>> metric 100
>> 192.168.187.0/24 dev ens35 proto kernel scope link src 192.168.187.128
>> metric 101
>> 192.168.199.0/24 dev wg0 proto kernel scope link src 192.168.199.1
>>
>> wacke@Ubuntu:/etc$ sudo cat /etc/wireguard/wg0.conf
>> [Interface]
>> PrivateKey = oLLeTlvwWsfGoyQpDwZo0k7AsMf0LkkfycGPwFjamEA=
>> ListenPort = 5999
>>
>> [Peer]
>> PublicKey = JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
>> AllowedIPs = 192.168.199.2/32
>> PersistentKeepalive = 25
>>
>> [Peer]
>> PublicKey = keIPXZFgvB79biV74kGc5R9vAHpzyVyQHci8KBSDIUw=
>> AllowedIPs = 192.168.199.3/32
>> PersistentKeepalive = 25
>>
>> wacke@Ubuntu:/etc$ sudo wg setconf wg0 /etc/wireguard/wg0.conf
>> wacke@Ubuntu:/etc$ sudo ip address add dev wg0 192.168.199.1/24
>> wacke@Ubuntu:/etc$ sudo ip link set up dev wg0
>> wacke@Ubuntu:/etc$ sudo wg show
>> interface: wg0
>>     public key: Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
>>     private key: (hidden)
>>     listening port: 5999
>>
>> peer: JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
>>     endpoint: 192.168.187.1:35560
>>     allowed ips: 192.168.199.2/32
>>     transfer: 2.89 KiB received, 1.80 KiB sent
>>     persistent keepalive: every 25 seconds
>>
>> peer: keIPXZFgvB79biV74kGc5R9vAHpzyVyQHci8KBSDIUw=
>>     allowed ips: 192.168.199.3/32
>>     persistent keepalive: every 25 seconds
>>
>> Client info:
>>
>> HOME-Server:/NAS/Software/Software # uname -a
>> Linux HOME-Server 5.14.14-2-default #1 SMP Thu Oct 21 05:05:03 UTC
>> 2021 (2b5383f) x86_64 x86_64 x86_64 GNU/Linux
>>
>> HOME-Server:/NAS/Software/Software # cat wg0.conf
>> [Interface]
>> PrivateKey = 4Pw1cetxd9TdfH3TSab+9UcRBlHwZ1o/vmrUgkerZls=
>>
>> [Peer]
>> PublicKey = Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
>> Endpoint = 192.168.187.128:5999
>> AllowedIPs = 192.168.199.0/24
>> PersistentKeepalive = 25
>>
>> HOME-Server:/NAS/Software/Software # ip link add wg0 type wireguard
>> HOME-Server:/NAS/Software/Software # ip address add dev wg0 192.168.199.2/24
>> HOME-Server:/NAS/Software/Software # wg setconf wg0 wg0.conf
>> HOME-Server:/NAS/Software/Software # ip link set wg0 up
>> HOME-Server:/NAS/Software/Software # wg show
>> interface: wg0
>>     public key: JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
>>     private key: (hidden)
>>     listening port: 53058
>>
>> peer: Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
>>     endpoint: 192.168.187.128:5999
>>     allowed ips: 192.168.199.0/24
>>     transfer: 0 B received, 59.26 KiB sent
>>     persistent keepalive: every 25 seconds
>>
>>
>> tcpdump of client side:
>>
>> HOME-Server:/NAS/Software/Software # tcpdump -i any -vn "(dst port 5999)"
>> tcpdump: data link type LINUX_SLL2
>> tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2),
>> snapshot length 262144 bytes
>> 22:10:37.794301 vmnet1 Out IP (tos 0x88, ttl 64, id 54724, offset 0,
>> flags [none], proto UDP (17), length 176)
>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>> 22:10:43.170306 vmnet1 Out IP (tos 0x88, ttl 64, id 55327, offset 0,
>> flags [none], proto UDP (17), length 176)
>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>> 22:10:48.546281 vmnet1 Out IP (tos 0x88, ttl 64, id 56142, offset 0,
>> flags [none], proto UDP (17), length 176)
>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>> 22:10:53.670343 vmnet1 Out IP (tos 0x88, ttl 64, id 57053, offset 0,
>> flags [none], proto UDP (17), length 176)
>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>> 22:11:18.754308 vmnet1 Out IP (tos 0x88, ttl 64, id 58505, offset 0,
>> flags [none], proto UDP (17), length 176)
>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>> 22:11:24.130286 vmnet1 Out IP (tos 0x88, ttl 64, id 58621, offset 0,
>> flags [none], proto UDP (17), length 176)
>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>> 22:11:29.506305 vmnet1 Out IP (tos 0x88, ttl 64, id 58810, offset 0,
>> flags [none], proto UDP (17), length 176)
>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>> 22:11:34.882326 vmnet1 Out IP (tos 0x88, ttl 64, id 59444, offset 0,
>> flags [none], proto UDP (17), length 176)
>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>> ^C
>> 8 packets captured
>> 8 packets received by filter
>> 0 packets dropped by kernel
>>
>> tcpdump of server side:
>>
>> wacke@Ubuntu:~$ sudo tcpdump -i any -vn "(src port 5999)"
>> tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1),
>> capture size 262144 bytes
>> 22:10:11.754862 IP (tos 0x88, ttl 64, id 22024, offset 0, flags
>> [none], proto UDP (17), length 120)
>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:10:16.874400 IP (tos 0x88, ttl 64, id 22399, offset 0, flags
>> [none], proto UDP (17), length 120)
>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:10:21.994433 IP (tos 0x88, ttl 64, id 23057, offset 0, flags
>> [none], proto UDP (17), length 120)
>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:10:27.370558 IP (tos 0x88, ttl 64, id 23912, offset 0, flags
>> [none], proto UDP (17), length 120)
>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:10:32.494605 IP (tos 0x88, ttl 64, id 24944, offset 0, flags
>> [none], proto UDP (17), length 120)
>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:10:37.610779 IP (tos 0x88, ttl 64, id 25327, offset 0, flags
>> [none], proto UDP (17), length 120)
>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:10:42.986834 IP (tos 0x88, ttl 64, id 25498, offset 0, flags
>> [none], proto UDP (17), length 120)
>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:10:48.362908 IP (tos 0x88, ttl 64, id 26521, offset 0, flags
>> [none], proto UDP (17), length 120)
>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:10:53.486942 IP (tos 0x88, ttl 64, id 27214, offset 0, flags
>> [none], proto UDP (17), length 120)
>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:11:18.571437 IP (tos 0x88, ttl 64, id 31112, offset 0, flags
>> [none], proto UDP (17), length 120)
>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:11:23.947460 IP (tos 0x88, ttl 64, id 31897, offset 0, flags
>> [none], proto UDP (17), length 120)
>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:11:29.323608 IP (tos 0x88, ttl 64, id 32715, offset 0, flags
>> [none], proto UDP (17), length 120)
>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> 22:11:34.699676 IP (tos 0x88, ttl 64, id 33840, offset 0, flags
>> [none], proto UDP (17), length 120)
>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>> ^C
>> 13 packets captured
>> 13 packets received by filter
>> 0 packets dropped by kernel
>>
>> As the tcpdump log showed, the wireguard server always chose the ip
>> with lowest metric as source ip (192.168.187.129) to reply to the
>> client (while the client tried to connect 192.168.187.128).
>>
>> Hope the info above can help to debug. Many thanks.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Source IP selection or multi-home or multi-interface issue - solved with source-based route - was: Re: Wireguard source ip selection issue with multi interfaces
  2022-10-19 22:56   ` Christoph Loesch
@ 2022-10-21 13:11     ` Christoph Loesch
  2022-10-21 13:25       ` Janne Johansson
  0 siblings, 1 reply; 7+ messages in thread
From: Christoph Loesch @ 2022-10-21 13:11 UTC (permalink / raw)
  To: wireguard

So I tried to come up with a simple setup to showcase the issue and then I realized:
"How should wireguard know which IP is the correct source IP address?"

In the simple setup wireguard even selected an IP from an interface which link is down which is kind of strange.
After removing the IP from the linkdown interface, wireguard took the IP as source from the next available interface.
Adding the IP back to the linkdown interface and wireguard prefered that IP again.
I digged into the source code but couldn't find the part where the source-IP is selected for outgoing connections (like a simple ping).
I am not a developer but would like to try to understand it. Could someone give me a pointer?

Nevertheless thinking about the question how to select the IP, I wondered if it would be possible to set the IP in wireguard settings, so I do not have to update the routes with the src parameter.
Then I saw the address setting and setting the IP there just fixed my "issue". I don't know why I overlooked that setting.. sorry for that

Thank you and kind regards

Am 20.10.2022 um 00:56 schrieb Christoph Loesch:
> Now I've read enough (rather old and also pretty recent) posts about that topic and I want to help finally fix this issue.
>
> Not just because it bugs our network too after we optimized/simplified our router-setup but also because it seems over the years many users seem to beaffected. Until now every thread I've read ends with an open question rather than a solution.
>
> In Jason's post here from 2017 I found his script that should reproduce theissue, I tried it and could not reproduce the issue with that script..
> https://lists.zx2c4.com/pipermail/wireguard/2017-November/002092.html
> To be honest I do not fully understand it to modify it to reproduce the issue in my environment.
>
> Here I posted one example and the (temporary) solution by deleting the route that wireguard had set and re-creating the same route but defining the (correct) IP as src.
> https://lists.zx2c4.com/pipermail/wireguard/2021-November/007324.html
>
> How could I help to fix this issue?
> If it helps, I could give full access to a test-router which has the same setup as all other routers in our network.
>
> Kind regards
>
>
> Am 23.11.2021 um 14:52 schrieb Christoph Loesch:
>> Hi,
>>
>> Is this related to my issue with using wrong source ip on outgoing connections?
>> (https://lists.zx2c4.com/pipermail/wireguard/2021-November/007307.html)
>>
>> Which bug would that be? Would be interesting in more detail to understand the issue.
>> I'm having a hard time to reproduce (or somehow "fix") the issue I am facing, since this issue is on only 5 devices from over 20 devices where all are configured same.
>>
>> The configuration or better all routes do not even have any metric defined so I am unsure what to add/remove to change the behaviour.
>> Also in my configuration there are multiple interfaces (br0,br1,eth0,eth1,etc..) as it is a router but there are no multiple default routes.
>> Comparing perfectly fine working devices to one of those five devices which have this issue I am not able to find a difference in IP-config or routes.
>> Even kernel and firmware versions are all same.
>>
>> The only difference I see is while looking at tcpdump where a simple pingto the wireguard server's internal IP (172.27.0.1/24 in my case) originates with the public IP of the device.
>>
>> If it is a known bug, it would be nice to know what (and if) I could try to fix and possibly confirm the bug. Or even better beeing able to reproduce the issue.
>>
>> Thanks and kind regards,
>> Christoph
>>
>> Am 19.11.2021 um 14:10 schrieb 曹煜:
>>> Hi,
>>> As Jason said in mailing list
>>> (https://lists.zx2c4.com/pipermail/wireguard/2017-November/002018.html)
>>> years ago, if wireguard reply the client with wrong src ip, then it is
>>> a bug. And now I'm trying to set up a minimal configuration.
>>>
>>> Server info:
>>>
>>> Ubuntu 20.04 in VMware with two Host-only network adapters:
>>>
>>> wacke@Ubuntu:/etc$ lsb_release -a
>>> No LSB modules are available.
>>> Distributor ID: Ubuntu
>>> Description: Ubuntu 20.04.3 LTS
>>> Release: 20.04
>>> Codename: focal
>>>
>>> wacke@Ubuntu:/etc$ uname -a
>>> Linux Ubuntu 5.14.8-051408-generic #202109280455 SMP Tue Sep 28
>>> 04:57:31 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> wacke@Ubuntu:/etc$ sudo ip route show
>>> 169.254.0.0/16 dev ens33 scope link metric 1000
>>> 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
>>> 192.168.187.0/24 dev ens33 proto kernel scope link src 192.168.187.129
>>> metric 100
>>> 192.168.187.0/24 dev ens35 proto kernel scope link src 192.168.187.128
>>> metric 101
>>> 192.168.199.0/24 dev wg0 proto kernel scope link src 192.168.199.1
>>>
>>> wacke@Ubuntu:/etc$ sudo cat /etc/wireguard/wg0.conf
>>> [Interface]
>>> PrivateKey = oLLeTlvwWsfGoyQpDwZo0k7AsMf0LkkfycGPwFjamEA=
>>> ListenPort = 5999
>>>
>>> [Peer]
>>> PublicKey = JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
>>> AllowedIPs = 192.168.199.2/32
>>> PersistentKeepalive = 25
>>>
>>> [Peer]
>>> PublicKey = keIPXZFgvB79biV74kGc5R9vAHpzyVyQHci8KBSDIUw=
>>> AllowedIPs = 192.168.199.3/32
>>> PersistentKeepalive = 25
>>>
>>> wacke@Ubuntu:/etc$ sudo wg setconf wg0 /etc/wireguard/wg0.conf
>>> wacke@Ubuntu:/etc$ sudo ip address add dev wg0 192.168.199.1/24
>>> wacke@Ubuntu:/etc$ sudo ip link set up dev wg0
>>> wacke@Ubuntu:/etc$ sudo wg show
>>> interface: wg0
>>>     public key: Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
>>>     private key: (hidden)
>>>     listening port: 5999
>>>
>>> peer: JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
>>>     endpoint: 192.168.187.1:35560
>>>     allowed ips: 192.168.199.2/32
>>>     transfer: 2.89 KiB received, 1.80 KiB sent
>>>     persistent keepalive: every 25 seconds
>>>
>>> peer: keIPXZFgvB79biV74kGc5R9vAHpzyVyQHci8KBSDIUw=
>>>     allowed ips: 192.168.199.3/32
>>>     persistent keepalive: every 25 seconds
>>>
>>> Client info:
>>>
>>> HOME-Server:/NAS/Software/Software # uname -a
>>> Linux HOME-Server 5.14.14-2-default #1 SMP Thu Oct 21 05:05:03 UTC
>>> 2021 (2b5383f) x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> HOME-Server:/NAS/Software/Software # cat wg0.conf
>>> [Interface]
>>> PrivateKey = 4Pw1cetxd9TdfH3TSab+9UcRBlHwZ1o/vmrUgkerZls=
>>>
>>> [Peer]
>>> PublicKey = Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
>>> Endpoint = 192.168.187.128:5999
>>> AllowedIPs = 192.168.199.0/24
>>> PersistentKeepalive = 25
>>>
>>> HOME-Server:/NAS/Software/Software # ip link add wg0 type wireguard
>>> HOME-Server:/NAS/Software/Software # ip address add dev wg0 192.168.199.2/24
>>> HOME-Server:/NAS/Software/Software # wg setconf wg0 wg0.conf
>>> HOME-Server:/NAS/Software/Software # ip link set wg0 up
>>> HOME-Server:/NAS/Software/Software # wg show
>>> interface: wg0
>>>     public key: JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
>>>     private key: (hidden)
>>>     listening port: 53058
>>>
>>> peer: Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
>>>     endpoint: 192.168.187.128:5999
>>>     allowed ips: 192.168.199.0/24
>>>     transfer: 0 B received, 59.26 KiB sent
>>>     persistent keepalive: every 25 seconds
>>>
>>>
>>> tcpdump of client side:
>>>
>>> HOME-Server:/NAS/Software/Software # tcpdump -i any -vn "(dst port 5999)"
>>> tcpdump: data link type LINUX_SLL2
>>> tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2),
>>> snapshot length 262144 bytes
>>> 22:10:37.794301 vmnet1 Out IP (tos 0x88, ttl 64, id 54724, offset 0,
>>> flags [none], proto UDP (17), length 176)
>>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>>> 22:10:43.170306 vmnet1 Out IP (tos 0x88, ttl 64, id 55327, offset 0,
>>> flags [none], proto UDP (17), length 176)
>>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>>> 22:10:48.546281 vmnet1 Out IP (tos 0x88, ttl 64, id 56142, offset 0,
>>> flags [none], proto UDP (17), length 176)
>>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>>> 22:10:53.670343 vmnet1 Out IP (tos 0x88, ttl 64, id 57053, offset 0,
>>> flags [none], proto UDP (17), length 176)
>>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>>> 22:11:18.754308 vmnet1 Out IP (tos 0x88, ttl 64, id 58505, offset 0,
>>> flags [none], proto UDP (17), length 176)
>>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>>> 22:11:24.130286 vmnet1 Out IP (tos 0x88, ttl 64, id 58621, offset 0,
>>> flags [none], proto UDP (17), length 176)
>>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>>> 22:11:29.506305 vmnet1 Out IP (tos 0x88, ttl 64, id 58810, offset 0,
>>> flags [none], proto UDP (17), length 176)
>>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>>> 22:11:34.882326 vmnet1 Out IP (tos 0x88, ttl 64, id 59444, offset 0,
>>> flags [none], proto UDP (17), length 176)
>>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>>> ^C
>>> 8 packets captured
>>> 8 packets received by filter
>>> 0 packets dropped by kernel
>>>
>>> tcpdump of server side:
>>>
>>> wacke@Ubuntu:~$ sudo tcpdump -i any -vn "(src port 5999)"
>>> tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1),
>>> capture size 262144 bytes
>>> 22:10:11.754862 IP (tos 0x88, ttl 64, id 22024, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:10:16.874400 IP (tos 0x88, ttl 64, id 22399, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:10:21.994433 IP (tos 0x88, ttl 64, id 23057, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:10:27.370558 IP (tos 0x88, ttl 64, id 23912, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:10:32.494605 IP (tos 0x88, ttl 64, id 24944, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:10:37.610779 IP (tos 0x88, ttl 64, id 25327, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:10:42.986834 IP (tos 0x88, ttl 64, id 25498, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:10:48.362908 IP (tos 0x88, ttl 64, id 26521, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:10:53.486942 IP (tos 0x88, ttl 64, id 27214, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:11:18.571437 IP (tos 0x88, ttl 64, id 31112, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:11:23.947460 IP (tos 0x88, ttl 64, id 31897, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:11:29.323608 IP (tos 0x88, ttl 64, id 32715, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:11:34.699676 IP (tos 0x88, ttl 64, id 33840, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> ^C
>>> 13 packets captured
>>> 13 packets received by filter
>>> 0 packets dropped by kernel
>>>
>>> As the tcpdump log showed, the wireguard server always chose the ip
>>> with lowest metric as source ip (192.168.187.129) to reply to the
>>> client (while the client tried to connect 192.168.187.128).
>>>
>>> Hope the info above can help to debug. Many thanks.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Source IP selection or multi-home or multi-interface issue - solved with source-based route - was: Re: Wireguard source ip selection issue with multi interfaces
  2022-10-21 13:11     ` Christoph Loesch
@ 2022-10-21 13:25       ` Janne Johansson
  2022-10-21 14:12         ` 曹煜
  0 siblings, 1 reply; 7+ messages in thread
From: Janne Johansson @ 2022-10-21 13:25 UTC (permalink / raw)
  To: Christoph Loesch; +Cc: wireguard

> "How should wireguard know which IP is the correct source IP address?"
>
> I digged into the source code but couldn't find the part where the source-IP is selected for outgoing connections (like a simple ping).
> I am not a developer but would like to try to understand it. Could someone give me a pointer?

I think it is left to the OS and it's tcp stack to pick the ip (and
interface) based on the UDP destination ip and the current routing
table when sending udp packets.

-- 
May the most significant bit of your life be positive.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Source IP selection or multi-home or multi-interface issue - solved with source-based route - was: Re: Wireguard source ip selection issue with multi interfaces
  2022-10-21 13:25       ` Janne Johansson
@ 2022-10-21 14:12         ` 曹煜
  0 siblings, 0 replies; 7+ messages in thread
From: 曹煜 @ 2022-10-21 14:12 UTC (permalink / raw)
  To: Janne Johansson; +Cc: wireguard

Hi,
Actually, I've hacked the wireguard source code by myself months ago,
and it's working as expected on openwrt with mwan3.
See details and the last hack patch on my github issue:
https://github.com/openwrt/packages/issues/9538
As I said on the issue, the problem is wireguard always reset the
incoming ip and ipmark, and then select the source ip from main
routing table.

Janne Johansson <icepic.dz@gmail.com> 于2022年10月21日周五 21:27写道:
>
> > "How should wireguard know which IP is the correct source IP address?"
> >
> > I digged into the source code but couldn't find the part where the source-IP is selected for outgoing connections (like a simple ping).
> > I am not a developer but would like to try to understand it. Could someone give me a pointer?
>
> I think it is left to the OS and it's tcp stack to pick the ip (and
> interface) based on the UDP destination ip and the current routing
> table when sending udp packets.
>
> --
> May the most significant bit of your life be positive.

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2022-10-21 14:16 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-19 13:10 Wireguard source ip selection issue with multi interfaces 曹煜
2021-11-23 13:52 ` Christoph Loesch
2022-10-19 22:47   ` Source IP selection or multi-home or multi-interface issue - solved with source-based route - was: " Christoph Loesch
2022-10-19 22:56   ` Christoph Loesch
2022-10-21 13:11     ` Christoph Loesch
2022-10-21 13:25       ` Janne Johansson
2022-10-21 14:12         ` 曹煜

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).