Possible regression between 5.18.2 and 6.2.1

Possible regression between 5.18.2 and 6.2.1

[Dan Crawford]

I recently updated a server from kernel version 5.18.2 to 6.2.1 and
discovered that WG clients could no longer connect; there were no
changes to configs. Reverting to 5.18.2 resolves the issue.

My server config looks something like

[Interface]
Address = 192.168.1.0/24
ListenPort = 51820
PrivateKey = XXX
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

[Peer]
PublicKey = XXX
AllowedIPs = 192.168.1.3/32

and my client config looks something like


[Interface]
Address = 192.168.1.3/32
DNS = 1.1.1.1
PrivateKey = XXX

[Peer]
AllowedIPs = 0.0.0.0/0
Endpoint = example.com:51820
PublicKey = XXX

On the server I get mysterious "packet has unallowed src ip" errors.
Playing around with various combinations of subnets and iptables
invocations doesn't seem to help. Was there a change to the config spec
that I missed? Or otherwise any other ideas what might be going on?

Thanks.

Re: Possible regression between 5.18.2 and 6.2.1

[Linux regression tracking (Thorsten Leemhuis)]

[CCing the regression list, as it should be in the loop for regressions:
https://docs.kernel.org/admin-guide/reporting-regressions.html]

[Also adding Jason to the list of recipients, as I'm not sure how
closely he follows the lists]

[TLDR: I'm adding this report to the list of tracked Linux kernel
regressions; the text you find below is based on a few templates
paragraphs you might have encountered already in similar form.
See link in footer if these mails annoy you.]

On 06.03.23 10:51, Dan Crawford wrote:
> I recently updated a server from kernel version 5.18.2 to 6.2.1 and
> discovered that WG clients could no longer connect; there were no
> changes to configs. Reverting to 5.18.2 resolves the issue.
> 
> My server config looks something like
> 
> [Interface]
> Address = 192.168.1.0/24
> ListenPort = 51820
> PrivateKey = XXX
> PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
> PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
> 
> [Peer]
> PublicKey = XXX
> AllowedIPs = 192.168.1.3/32
> 
> and my client config looks something like
> 
> 
> [Interface]
> Address = 192.168.1.3/32
> DNS = 1.1.1.1
> PrivateKey = XXX
> 
> [Peer]
> AllowedIPs = 0.0.0.0/0
> Endpoint = example.com:51820
> PublicKey = XXX
> 
> On the server I get mysterious "packet has unallowed src ip" errors.
> Playing around with various combinations of subnets and iptables
> invocations doesn't seem to help. Was there a change to the config spec
> that I missed? Or otherwise any other ideas what might be going on?

Thanks for the report. To be sure the issue doesn't fall through the
cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
tracking bot:

#regzbot ^introduced v5.18..v6.2
#regzbot title net: wireguard: clients can no longer connect
#regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already
discussed somewhere else? It was fixed already? You want to clarify when
the regression started to happen? Or point out I got the title or
something else totally wrong? Then just reply and tell me -- ideally
while also telling regzbot about it, as explained by the page listed in
the footer of this mail.

Developers: When fixing the issue, remember to add 'Link:' tags pointing
to the report (the parent of this mail). See page linked in footer for
details.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

Re: Possible regression between 5.18.2 and 6.2.1

[Jason A. Donenfeld]

Hi Dan,

Hard to imagine that this is a WireGuard bug, but more likely
something having to do with SNAT or something.

What is the unallowed src IP when you get that error? Can you debug
further? Maybe bisect a bit? Otherwise, not much I can do.

The diff between those versions you listed is pretty minimal, so I
suspect your bug is elsewhere.

Jason

Re: Possible regression between 5.18.2 and 6.2.1

[Dan Crawford]

Thanks for the suggestions. I've done some bisecting and I've found
that the issue appears due to wg-quick, which means I can easily work
around it. First, apologies but there's a typo in my original
email, I upgraded from 5.12.8, not 5.18.2.

On versions prior to 5.14.0, wg-quick correctly adds routes for the
appropriate addresses (line 341 and then line 177). However, on versions
after 5.14.0, the condition doesn't work quite right and the ip route
add command does not run (line 177).

To investigate this I print ip -4 route show dev wg1 match 192.168.1.3,
on both 5.13.0 and 5.14.0, at line 177.

On 5.13.0 I get no output, and the ip route add command runs.
However, on 5.14.0 the output is

192.168.1.0/24 proto kernel scope link src 192.168.1.0 

and so the ip route add command does not run.

Obviously I can easily work around the issue by patching the conditional
out of wg-quick. But I don't have any clue why the output of ip varies
between 5.13 and 5.14. I'm also surprised no-one has encountered
this issue either (unless I missed something while searching).

Possibly one way to resolve the issue is to replace the conditional with

[[ -n $(ip $proto route show dev "$INTERFACE" match "$1" proto boot 2>/dev/null) ]]


Thanks
Dan


On Fri Mar 31, 2023 at 1:39 AM AEDT, Jason A. Donenfeld wrote:
> Hi Dan,
>
> Hard to imagine that this is a WireGuard bug, but more likely
> something having to do with SNAT or something.
>
> What is the unallowed src IP when you get that error? Can you debug
> further? Maybe bisect a bit? Otherwise, not much I can do.
>
> The diff between those versions you listed is pretty minimal, so I
> suspect your bug is elsewhere.
>
> Jason

Re: Possible regression between 5.18.2 and 6.2.1

[Linux regression tracking #update (Thorsten Leemhuis)]

On 02.04.23 03:14, Dan Crawford wrote:
> Thanks for the suggestions. I've done some bisecting and I've found
> that the issue appears due to wg-quick, which means I can easily work
> around it. First, apologies but there's a typo in my original
> email, I upgraded from 5.12.8, not 5.18.2.
> 
> On versions prior to 5.14.0, wg-quick correctly adds routes for the
> appropriate addresses (line 341 and then line 177). However, on versions
> after 5.14.0, the condition doesn't work quite right and the ip route
> add command does not run (line 177).
> 
> To investigate this I print ip -4 route show dev wg1 match 192.168.1.3,
> on both 5.13.0 and 5.14.0, at line 177.
> 
> On 5.13.0 I get no output, and the ip route add command runs.
> However, on 5.14.0 the output is
> 
> 192.168.1.0/24 proto kernel scope link src 192.168.1.0 

Thx for your investigation.

This per the Linux kernel policy might qualify as regression. But the
thing is: the change was introduced quite a while ago already, hence
fixing it now might itself lead to other regressions, as other scripts
might have started to rely on the new behavior. If anyone wanted to
discuss this upstream we'd also likely need a bisection to know which
change caused the new behavior. Given all this and the relative simple...

> and so the ip route add command does not run.
> 
> Obviously I can easily work around the issue by patching the conditional
> out of wg-quick. But I don't have any clue why the output of ip varies
> between 5.13 and 5.14. I'm also surprised no-one has encountered
> this issue either (unless I missed something while searching).
> 
> Possibly one way to resolve the issue is to replace the conditional with
> 
> [[ -n $(ip $proto route show dev "$INTERFACE" match "$1" proto boot 2>/dev/null) ]]

...workaround you apparently have found I'd say it's better to fix this
in wg-quick. I'll thus for now will remove this from the list of tracked
Linux kernel regressions:

#regzbot inconclusive: small change broke script, workaround found;
fixing this now might lead to other regressions
#regzbot ignore-activity

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.



> On Fri Mar 31, 2023 at 1:39 AM AEDT, Jason A. Donenfeld wrote:
>> Hi Dan,
>>
>> Hard to imagine that this is a WireGuard bug, but more likely
>> something having to do with SNAT or something.
>>
>> What is the unallowed src IP when you get that error? Can you debug
>> further? Maybe bisect a bit? Otherwise, not much I can do.
>>
>> The diff between those versions you listed is pretty minimal, so I
>> suspect your bug is elsewhere.
>>
>> Jason
> 
>