Re: Source IP selection or multi-home or multi-interface issue - solved with source-based route - was: Re: Wireguard source ip selection issue with multi interfaces

[Christoph Loesch <wireguard-mail@chil.at>]

So I tried to come up with a simple setup to showcase the issue and then I realized:
"How should wireguard know which IP is the correct source IP address?"

In the simple setup wireguard even selected an IP from an interface which link is down which is kind of strange.
After removing the IP from the linkdown interface, wireguard took the IP as source from the next available interface.
Adding the IP back to the linkdown interface and wireguard prefered that IP again.
I digged into the source code but couldn't find the part where the source-IP is selected for outgoing connections (like a simple ping).
I am not a developer but would like to try to understand it. Could someone give me a pointer?

Nevertheless thinking about the question how to select the IP, I wondered if it would be possible to set the IP in wireguard settings, so I do not have to update the routes with the src parameter.
Then I saw the address setting and setting the IP there just fixed my "issue". I don't know why I overlooked that setting.. sorry for that

Thank you and kind regards

Am 20.10.2022 um 00:56 schrieb Christoph Loesch:
> Now I've read enough (rather old and also pretty recent) posts about that topic and I want to help finally fix this issue.
>
> Not just because it bugs our network too after we optimized/simplified our router-setup but also because it seems over the years many users seem to beaffected. Until now every thread I've read ends with an open question rather than a solution.
>
> In Jason's post here from 2017 I found his script that should reproduce theissue, I tried it and could not reproduce the issue with that script..
> https://lists.zx2c4.com/pipermail/wireguard/2017-November/002092.html
> To be honest I do not fully understand it to modify it to reproduce the issue in my environment.
>
> Here I posted one example and the (temporary) solution by deleting the route that wireguard had set and re-creating the same route but defining the (correct) IP as src.
> https://lists.zx2c4.com/pipermail/wireguard/2021-November/007324.html
>
> How could I help to fix this issue?
> If it helps, I could give full access to a test-router which has the same setup as all other routers in our network.
>
> Kind regards
>
>
> Am 23.11.2021 um 14:52 schrieb Christoph Loesch:
>> Hi,
>>
>> Is this related to my issue with using wrong source ip on outgoing connections?
>> (https://lists.zx2c4.com/pipermail/wireguard/2021-November/007307.html)
>>
>> Which bug would that be? Would be interesting in more detail to understand the issue.
>> I'm having a hard time to reproduce (or somehow "fix") the issue I am facing, since this issue is on only 5 devices from over 20 devices where all are configured same.
>>
>> The configuration or better all routes do not even have any metric defined so I am unsure what to add/remove to change the behaviour.
>> Also in my configuration there are multiple interfaces (br0,br1,eth0,eth1,etc..) as it is a router but there are no multiple default routes.
>> Comparing perfectly fine working devices to one of those five devices which have this issue I am not able to find a difference in IP-config or routes.
>> Even kernel and firmware versions are all same.
>>
>> The only difference I see is while looking at tcpdump where a simple pingto the wireguard server's internal IP (172.27.0.1/24 in my case) originates with the public IP of the device.
>>
>> If it is a known bug, it would be nice to know what (and if) I could try to fix and possibly confirm the bug. Or even better beeing able to reproduce the issue.
>>
>> Thanks and kind regards,
>> Christoph
>>
>> Am 19.11.2021 um 14:10 schrieb 曹煜:
>>> Hi,
>>> As Jason said in mailing list
>>> (https://lists.zx2c4.com/pipermail/wireguard/2017-November/002018.html)
>>> years ago, if wireguard reply the client with wrong src ip, then it is
>>> a bug. And now I'm trying to set up a minimal configuration.
>>>
>>> Server info:
>>>
>>> Ubuntu 20.04 in VMware with two Host-only network adapters:
>>>
>>> wacke@Ubuntu:/etc$ lsb_release -a
>>> No LSB modules are available.
>>> Distributor ID: Ubuntu
>>> Description: Ubuntu 20.04.3 LTS
>>> Release: 20.04
>>> Codename: focal
>>>
>>> wacke@Ubuntu:/etc$ uname -a
>>> Linux Ubuntu 5.14.8-051408-generic #202109280455 SMP Tue Sep 28
>>> 04:57:31 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> wacke@Ubuntu:/etc$ sudo ip route show
>>> 169.254.0.0/16 dev ens33 scope link metric 1000
>>> 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
>>> 192.168.187.0/24 dev ens33 proto kernel scope link src 192.168.187.129
>>> metric 100
>>> 192.168.187.0/24 dev ens35 proto kernel scope link src 192.168.187.128
>>> metric 101
>>> 192.168.199.0/24 dev wg0 proto kernel scope link src 192.168.199.1
>>>
>>> wacke@Ubuntu:/etc$ sudo cat /etc/wireguard/wg0.conf
>>> [Interface]
>>> PrivateKey = oLLeTlvwWsfGoyQpDwZo0k7AsMf0LkkfycGPwFjamEA=
>>> ListenPort = 5999
>>>
>>> [Peer]
>>> PublicKey = JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
>>> AllowedIPs = 192.168.199.2/32
>>> PersistentKeepalive = 25
>>>
>>> [Peer]
>>> PublicKey = keIPXZFgvB79biV74kGc5R9vAHpzyVyQHci8KBSDIUw=
>>> AllowedIPs = 192.168.199.3/32
>>> PersistentKeepalive = 25
>>>
>>> wacke@Ubuntu:/etc$ sudo wg setconf wg0 /etc/wireguard/wg0.conf
>>> wacke@Ubuntu:/etc$ sudo ip address add dev wg0 192.168.199.1/24
>>> wacke@Ubuntu:/etc$ sudo ip link set up dev wg0
>>> wacke@Ubuntu:/etc$ sudo wg show
>>> interface: wg0
>>>     public key: Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
>>>     private key: (hidden)
>>>     listening port: 5999
>>>
>>> peer: JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
>>>     endpoint: 192.168.187.1:35560
>>>     allowed ips: 192.168.199.2/32
>>>     transfer: 2.89 KiB received, 1.80 KiB sent
>>>     persistent keepalive: every 25 seconds
>>>
>>> peer: keIPXZFgvB79biV74kGc5R9vAHpzyVyQHci8KBSDIUw=
>>>     allowed ips: 192.168.199.3/32
>>>     persistent keepalive: every 25 seconds
>>>
>>> Client info:
>>>
>>> HOME-Server:/NAS/Software/Software # uname -a
>>> Linux HOME-Server 5.14.14-2-default #1 SMP Thu Oct 21 05:05:03 UTC
>>> 2021 (2b5383f) x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> HOME-Server:/NAS/Software/Software # cat wg0.conf
>>> [Interface]
>>> PrivateKey = 4Pw1cetxd9TdfH3TSab+9UcRBlHwZ1o/vmrUgkerZls=
>>>
>>> [Peer]
>>> PublicKey = Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
>>> Endpoint = 192.168.187.128:5999
>>> AllowedIPs = 192.168.199.0/24
>>> PersistentKeepalive = 25
>>>
>>> HOME-Server:/NAS/Software/Software # ip link add wg0 type wireguard
>>> HOME-Server:/NAS/Software/Software # ip address add dev wg0 192.168.199.2/24
>>> HOME-Server:/NAS/Software/Software # wg setconf wg0 wg0.conf
>>> HOME-Server:/NAS/Software/Software # ip link set wg0 up
>>> HOME-Server:/NAS/Software/Software # wg show
>>> interface: wg0
>>>     public key: JYvK2/FBbbSrDbkH29ejWejkBJS3ch/SFtyZwNf5O1c=
>>>     private key: (hidden)
>>>     listening port: 53058
>>>
>>> peer: Vkdx0MhQkc9anLaUehR/Df1zuKjxceflxMCAeAaCWCo=
>>>     endpoint: 192.168.187.128:5999
>>>     allowed ips: 192.168.199.0/24
>>>     transfer: 0 B received, 59.26 KiB sent
>>>     persistent keepalive: every 25 seconds
>>>
>>>
>>> tcpdump of client side:
>>>
>>> HOME-Server:/NAS/Software/Software # tcpdump -i any -vn "(dst port 5999)"
>>> tcpdump: data link type LINUX_SLL2
>>> tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2),
>>> snapshot length 262144 bytes
>>> 22:10:37.794301 vmnet1 Out IP (tos 0x88, ttl 64, id 54724, offset 0,
>>> flags [none], proto UDP (17), length 176)
>>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>>> 22:10:43.170306 vmnet1 Out IP (tos 0x88, ttl 64, id 55327, offset 0,
>>> flags [none], proto UDP (17), length 176)
>>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>>> 22:10:48.546281 vmnet1 Out IP (tos 0x88, ttl 64, id 56142, offset 0,
>>> flags [none], proto UDP (17), length 176)
>>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>>> 22:10:53.670343 vmnet1 Out IP (tos 0x88, ttl 64, id 57053, offset 0,
>>> flags [none], proto UDP (17), length 176)
>>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>>> 22:11:18.754308 vmnet1 Out IP (tos 0x88, ttl 64, id 58505, offset 0,
>>> flags [none], proto UDP (17), length 176)
>>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>>> 22:11:24.130286 vmnet1 Out IP (tos 0x88, ttl 64, id 58621, offset 0,
>>> flags [none], proto UDP (17), length 176)
>>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>>> 22:11:29.506305 vmnet1 Out IP (tos 0x88, ttl 64, id 58810, offset 0,
>>> flags [none], proto UDP (17), length 176)
>>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>>> 22:11:34.882326 vmnet1 Out IP (tos 0x88, ttl 64, id 59444, offset 0,
>>> flags [none], proto UDP (17), length 176)
>>>       192.168.187.1.53058 > 192.168.187.128.5999: UDP, length 148
>>> ^C
>>> 8 packets captured
>>> 8 packets received by filter
>>> 0 packets dropped by kernel
>>>
>>> tcpdump of server side:
>>>
>>> wacke@Ubuntu:~$ sudo tcpdump -i any -vn "(src port 5999)"
>>> tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1),
>>> capture size 262144 bytes
>>> 22:10:11.754862 IP (tos 0x88, ttl 64, id 22024, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:10:16.874400 IP (tos 0x88, ttl 64, id 22399, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:10:21.994433 IP (tos 0x88, ttl 64, id 23057, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:10:27.370558 IP (tos 0x88, ttl 64, id 23912, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:10:32.494605 IP (tos 0x88, ttl 64, id 24944, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:10:37.610779 IP (tos 0x88, ttl 64, id 25327, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:10:42.986834 IP (tos 0x88, ttl 64, id 25498, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:10:48.362908 IP (tos 0x88, ttl 64, id 26521, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:10:53.486942 IP (tos 0x88, ttl 64, id 27214, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:11:18.571437 IP (tos 0x88, ttl 64, id 31112, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:11:23.947460 IP (tos 0x88, ttl 64, id 31897, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:11:29.323608 IP (tos 0x88, ttl 64, id 32715, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> 22:11:34.699676 IP (tos 0x88, ttl 64, id 33840, offset 0, flags
>>> [none], proto UDP (17), length 120)
>>>       192.168.187.129.5999 > 192.168.187.1.53058: UDP, length 92
>>> ^C
>>> 13 packets captured
>>> 13 packets received by filter
>>> 0 packets dropped by kernel
>>>
>>> As the tcpdump log showed, the wireguard server always chose the ip
>>> with lowest metric as source ip (192.168.187.129) to reply to the
>>> client (while the client tried to connect 192.168.187.128).
>>>
>>> Hope the info above can help to debug. Many thanks.