wg-quick with default route fails on nfs root filesystem

wg-quick with default route fails on nfs root filesystem

[Chris]

When wg-quick detects a default route through the tunnel it does this through a 
new routing table with a default route.
However not to destroy the existing non-default routes these will looked up and 
used first. This results in the follwing policy rule entries:
(The priority numers may be different from system to system)

32764:    from all lookup main suppress_prefixlength 0
32765:    not from all fwmark 0xca6c lookup 51820

It is very important of course, that the suppress_prefix rule comes first in the 
list, before
the second rule introduces the new default route (preventig the wireguard 
traffic through it's own tunnel).

The way to archive this is done by the following command sequence:

ip -4 rule add not fwmark 51820 table 51820
ip -4 rule add table main suppress_prefixlength 0

The sequence of the commands is important as the latter command gets the higher 
priority (lower numer).

BUT:
In case your root filesystem needs the local network, the second command will 
not be reached as the
first command (setting the new default route) kills the root filesystem and the 
system stalls!!!!!!

One possible solution:
Instead of adding the suppress_prefixlength 0 command secondly it must be first.
The you must find the priority of that rule and the add the default route with 
the same priority.
A rule with same priority will be added AFTER the other rules.

Example:
ip -4 rule add table main suppress_prefixlength 0
PRIO=$(ip rule list from all|grep suppress_prefixlength|sed -e 
'{s/^\(.*\)\:.*/\1/;q}')
ip -4 rule add not fwmark 51820 table 51820 priority $PRIO

This will lead to the correct sequence:
32765:    from all lookup main suppress_prefixlength 0
32765:    not from all fwmark 0xca6c lookup 51820
(Note the same priority number)

There are probably better ways to cirumvent cutting off the root filesystem.

Chris

Re: wg-quick with default route fails on nfs root filesystem

[Tom Yan]

Hi Chris,

So I did a test and took a look at the log:
...
wg-quick[2003]: [#] ip -4 route add 0.0.0.0/0 dev aliyun table 51820
wg-quick[2003]: [#] ip -4 rule add not fwmark 51820 table 51820
wg-quick[2003]: [#] ip -4 rule add table main suppress_prefixlength 0
...

What if wg-quick adds the route after adding both rules? AFAIK, a rule
will be a no-op if the table it looks up is empty.

Regards,
Tom

On Wed, 14 Jul 2021 at 18:00, Chris <wireguard@spam-free.eu> wrote:
>
> When wg-quick detects a default route through the tunnel it does this through a
> new routing table with a default route.
> However not to destroy the existing non-default routes these will looked up and
> used first. This results in the follwing policy rule entries:
> (The priority numers may be different from system to system)
>
> 32764:    from all lookup main suppress_prefixlength 0
> 32765:    not from all fwmark 0xca6c lookup 51820
>
> It is very important of course, that the suppress_prefix rule comes first in the
> list, before
> the second rule introduces the new default route (preventig the wireguard
> traffic through it's own tunnel).
>
> The way to archive this is done by the following command sequence:
>
> ip -4 rule add not fwmark 51820 table 51820
> ip -4 rule add table main suppress_prefixlength 0
>
> The sequence of the commands is important as the latter command gets the higher
> priority (lower numer).
>
> BUT:
> In case your root filesystem needs the local network, the second command will
> not be reached as the
> first command (setting the new default route) kills the root filesystem and the
> system stalls!!!!!!
>
> One possible solution:
> Instead of adding the suppress_prefixlength 0 command secondly it must be first.
> The you must find the priority of that rule and the add the default route with
> the same priority.
> A rule with same priority will be added AFTER the other rules.
>
> Example:
> ip -4 rule add table main suppress_prefixlength 0
> PRIO=$(ip rule list from all|grep suppress_prefixlength|sed -e
> '{s/^\(.*\)\:.*/\1/;q}')
> ip -4 rule add not fwmark 51820 table 51820 priority $PRIO
>
> This will lead to the correct sequence:
> 32765:    from all lookup main suppress_prefixlength 0
> 32765:    not from all fwmark 0xca6c lookup 51820
> (Note the same priority number)
>
> There are probably better ways to cirumvent cutting off the root filesystem.
>
> Chris
>