Bug report: Policy routed packets are dropped by wireguard

Bug report: Policy routed packets are dropped by wireguard

[Michael Wu]

Hello,

I'd like to confirm that the following problem seemed to be a bug, and
not configuration error:
https://lists.zx2c4.com/pipermail/wireguard/2019-September/004545.html
(Content dupated at the end of this email for easier reference)

On system 1 (kernel 4.15), everything worked as expected. However, on
system 2 (kernel 5.4), the issue described above is observed.

Regards,
Michael


[System 1]:
Ubuntu 18.04.5 LTS
4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64
x86_64 x86_64 GNU/Linux

[System 2]:
Ubuntu 20.04.2 LTS
5.4.0-73-generic #82-Ubuntu SMP Wed Apr 14 17:39:42 UTC 2021 x86_64
x86_64 x86_64 GNU/Linux

[Policy routed packets are dropped by wireguard]
Original mail: https://lists.zx2c4.com/pipermail/wireguard/2019-September/004545.html

Hello!

I'm looking for technical advice.
Currently I'm trying to pass marked sessions through wireguard VPN network.

Marking is done by cgroups classid matching:
> iptables -A OUTPUT -m cgroup --cgroup 3735928559 -j MARK --set-xmark 0x1c3/0xffffffff

The only route in the `vpn` table is default routing through wg0:
> ip route add default dev wg0 table vpn

Routing rule is pretty simple:
> ip rule add fwmark 451 table vpn

Now I pass some packets on the interface:
> cgexec -g net_cls:vpn ping 10.0.1.1

I see packets reaching interface but dropped in the driver:
> tcpdump -i wg0 host 10.0.1.1
> ...
> 6 packets dropped by interface

Value in 4th column (TX drop) is increasing in the `/proc/net/dev` for wg0.

If I add route to default routing table and do ping without assigning
cgroup to the process then all is perfectly fine.
> ip route add 10.0.1.0/24 dev wg0

> ping 10.0.1.1
> PING 10.0.1.1 (10.0.1.1) 56(84) bytes of data.
> 64 bytes from 10.0.1.1: icmp_seq=1 ttl=64 time=46.1 ms

Is it some kind of a bug of misconfiguration?

> uname -r
> 5.2.1-gentoo

Installed Gentoo package atom.
> =net-vpn/wireguard-0.0.20190913

Thanks for any help!

Re: Bug report: Policy routed packets are dropped by wireguard

[Jason A. Donenfeld]

Hi,

On Mon, May 17, 2021 at 5:22 PM Michael Wu <michael.apply@gmail.com> wrote:
> I'd like to confirm that the following problem seemed to be a bug, and
> not configuration error:
> https://lists.zx2c4.com/pipermail/wireguard/2019-September/004545.html
> (Content dupated at the end of this email for easier reference)
>
> On system 1 (kernel 4.15), everything worked as expected. However, on
> system 2 (kernel 5.4), the issue described above is observed.
>
> Regards,
> Michael
>
>
> [System 1]:
> Ubuntu 18.04.5 LTS
> 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64
> x86_64 x86_64 GNU/Linux
>
> [System 2]:
> Ubuntu 20.04.2 LTS
> 5.4.0-73-generic #82-Ubuntu SMP Wed Apr 14 17:39:42 UTC 2021 x86_64
> x86_64 x86_64 GNU/Linux
>
> [Policy routed packets are dropped by wireguard]
> Original mail: https://lists.zx2c4.com/pipermail/wireguard/2019-September/004545.html
>
> Hello!
>
> I'm looking for technical advice.
> Currently I'm trying to pass marked sessions through wireguard VPN network.
>
> Marking is done by cgroups classid matching:
> > iptables -A OUTPUT -m cgroup --cgroup 3735928559 -j MARK --set-xmark 0x1c3/0xffffffff
>
> The only route in the `vpn` table is default routing through wg0:
> > ip route add default dev wg0 table vpn
>
> Routing rule is pretty simple:
> > ip rule add fwmark 451 table vpn
>
> Now I pass some packets on the interface:
> > cgexec -g net_cls:vpn ping 10.0.1.1
>
> I see packets reaching interface but dropped in the driver:
> > tcpdump -i wg0 host 10.0.1.1
> > ...
> > 6 packets dropped by interface
>
> Value in 4th column (TX drop) is increasing in the `/proc/net/dev` for wg0.
>
> If I add route to default routing table and do ping without assigning
> cgroup to the process then all is perfectly fine.
> > ip route add 10.0.1.0/24 dev wg0
>
> > ping 10.0.1.1
> > PING 10.0.1.1 (10.0.1.1) 56(84) bytes of data.
> > 64 bytes from 10.0.1.1: icmp_seq=1 ttl=64 time=46.1 ms
>
> Is it some kind of a bug of misconfiguration?

I think I might have fixed this upstream in October:

https://git.zx2c4.com/wireguard-linux/commit/?id=46d6c5ae953cc0be38efd0e469284df7c4328cf8

I suspect that this patch needs to be in your kernel in order for the
bug to go away.

I added a "warn-only" test to the compat layer here:

https://git.zx2c4.com/wireguard-linux-compat/commit/?id=99e954f4871d9a760451c5ada99dfaae5df256e5

These are tested on all the Ubuntu kernels. Looking at those here and
ctrl+f-ing for the warning message on
https://www.wireguard.com/build-status/#data-wireguard-linux-compat we
can see which ones are still unpatched:

- 5.4.0-74.83-ubuntu-focal: good
- 5.4.0-74.83~18.04.1-ubuntu-bionic-hwe-5.4: good
- 5.3.0-74.70-ubuntu-bionic-hwe: BAD
- 4.15.0-144.148-ubuntu-bionic: good

So it looks like, of the most recent Canonical kernels, only their 5.3
is missing the patch, but their 4.15 and 5.4 trees are working fine.

So I'd assume the problem is that the 5.4 kernel you're running is old
and needs an update? Or perhaps you've unearthed a different problem?
But please make sure you're running 5.4.0-74.83 first, and then let me
know.

Thanks,
Jason