Problems with Ubuntu 16.04 kernels and wireguard 1.0.20200611

Problems with Ubuntu 16.04 kernels and wireguard 1.0.20200611

[Gregory ORIOL]

Hi,

Since commit e24c9a9265af40781fa27b5de11dd5b78925c5be to
wireguard-linux-compat with a fix for some Ubuntu versions, we are
experiencing a problem with some older, but still LTS, versions of
Ubuntu 16.04: older kernels 4.4.0-148, 4.4.0-166 fail to build with
wireguard 1.0.20200611

So, any system running an (or still having an installed) "older" kernel
and doing an apt upgrade to install wireguard 1.0.20200611 would fail
during the wireguard dkms step, while trying to build wireguard for all
the kernels available.

The problem gets more problematic when a newer kernel 4.4.0-184 gets
installed with the same apt upgrade: then, trying to downgrade wireguard
also fails; none of the 1.0.20200611 or 1.0.20200520 versions work
anymore with this combination of old/new kernels...

To recap :
# wireguard 1.0.20200520
 - ok with kernels 4.4.0-148, 4.4.0-166
 - fails with kernel 4.4.0-184
# wireguard 1.0.20200611
 - fails with kernels 4.4.0-148, 4.4.0-166
 - ok with kernel 4.4.0-184

(nb: we see it now with -184 but it could have started with an earlier
version)

We could partially fix this by manually getting each deb/src and doing
dkms install:
dkms install wireguard/1.0.20200520 -k 4.4.0-148-generic
dkms install wireguard/1.0.20200611 -k 4.4.0-184-generic
...
But apt upgrade is still broken.

While we could boot onto the newer kernel and remove the older ones to
get rid of the problem, this situation would prevent from having a
"previous working" kernel on the system, which is not very safe.

Could there be a fix for this made to wireguard-linux-compat for those
versions?

Best regards,
Greg

Re: Problems with Ubuntu 16.04 kernels and wireguard 1.0.20200611

[Jason A. Donenfeld]

Hi Gregory,

On Wed, Jun 17, 2020 at 2:01 AM Gregory ORIOL <fortin81@gmail.com> wrote:
> Since commit e24c9a9265af40781fa27b5de11dd5b78925c5be to
> wireguard-linux-compat with a fix for some Ubuntu versions, we are
> experiencing a problem with some older, but still LTS, versions of
> Ubuntu 16.04: older kernels 4.4.0-148, 4.4.0-166 fail to build with
> wireguard 1.0.20200611
>
> So, any system running an (or still having an installed) "older" kernel
> and doing an apt upgrade to install wireguard 1.0.20200611 would fail
> during the wireguard dkms step, while trying to build wireguard for all
> the kernels available.
>
> The problem gets more problematic when a newer kernel 4.4.0-184 gets
> installed with the same apt upgrade: then, trying to downgrade wireguard
> also fails; none of the 1.0.20200611 or 1.0.20200520 versions work
> anymore with this combination of old/new kernels...
>
> To recap :
> # wireguard 1.0.20200520
>  - ok with kernels 4.4.0-148, 4.4.0-166
>  - fails with kernel 4.4.0-184
> # wireguard 1.0.20200611
>  - fails with kernels 4.4.0-148, 4.4.0-166
>  - ok with kernel 4.4.0-184
>
> (nb: we see it now with -184 but it could have started with an earlier
> version)
>
> We could partially fix this by manually getting each deb/src and doing
> dkms install:
> dkms install wireguard/1.0.20200520 -k 4.4.0-148-generic
> dkms install wireguard/1.0.20200611 -k 4.4.0-184-generic
> ...
> But apt upgrade is still broken.
>
> While we could boot onto the newer kernel and remove the older ones to
> get rid of the problem, this situation would prevent from having a
> "previous working" kernel on the system, which is not very safe.
>
> Could there be a fix for this made to wireguard-linux-compat for those
> versions?

Unfortunately, I don't have a super good solution for you right now.
The wireguard-linux-compat repo is developed against the latest Ubuntu
kernels that they put out once every three weeks. You can see them
being tested at the bottom of
<https://www.wireguard.com/build-status/>. The backport against
upstream mainline kernels is z-granular (for an x.y.z versioning
scheme), but Ubuntu's release cycle and versioning scheme is a bit too
chaotic to make it reasonable to try to manage all the differences
between their kernels every three weeks. So for distro kernels --
Ubuntu, RHEL, Debian, etc -- we typically just develop against the
latest one, and try to make sure that we release it at the right time
so users aren't caught with no working version. This means,
unfortunately, that when there are badly breaking changes, like in
this last cycle, you have to uninstall the old kernels or mask them
from dkms, in order to get dkms to avoid building for them and only
building for the new kernel. There might be other more complicated
solutions that closely track version dependencies or do compile time
feature probing, but that comes with a maintenance burden far too
arduous for a distro frankenkernel.

But there is hope!

Canonical is adding WireGuard to 18.04 and 16.04, and this is coming
in two steps:

Step 1) The wireguard-dkms and wireguard-tools packages will be added
to the package archives, so that you won't have to use the PPA. This
means that Canonical's kernel team will include wireguard-dkms in
their development tests, so that they won't accidentally ship kernels
with build breakage, like what you experienced last week.

Step 2) The wireguard-dkms package will get built by Canonical,
signed, and shipped alongside the other modules, so that you won't
have to install wireguard-dkms, and it will just come out of the box
with the normal kernel updates. This is already the case with 20.04
and 19.10. They're working on it now for 18.04, and I really really
hope to see that happen by the next cycle. And maybe if we ask apw
(CC'd) nicely, he'll even do it for 16.04 too.

Regards,
Jason

Re: Problems with Ubuntu 16.04 kernels and wireguard 1.0.20200611

[Gregory ORIOL]

Hi,

Thanks for your feedback!

Couldn't that issue be fixed by adapting the IF condition that has been
changed in commit e24c9a9265af40781fa27b5de11dd5b78925c5be?
I know it's not very practical to support/check this, plus it's not
wireguard's fault, but that will likely cause trouble to other users too.

We could obviously wait a few weeks with a broken apt upgrade state
until a new new kernel version is released and then switch to it, and
remove the old ones, so then we will have an n-1 kernel -184 working and
the new-new one as current. But that feels like a very unstable
situation for our systems until then...

Regards,
Gregory


Jason A. Donenfeld a écrit le 17/06/2020 à 10:13 :

> Hi Gregory,
>
> On Wed, Jun 17, 2020 at 2:01 AM Gregory ORIOL <fortin81@gmail.com> wrote:
>> Since commit e24c9a9265af40781fa27b5de11dd5b78925c5be to
>> wireguard-linux-compat with a fix for some Ubuntu versions, we are
>> experiencing a problem with some older, but still LTS, versions of
>> Ubuntu 16.04: older kernels 4.4.0-148, 4.4.0-166 fail to build with
>> wireguard 1.0.20200611
>>
>> So, any system running an (or still having an installed) "older" kernel
>> and doing an apt upgrade to install wireguard 1.0.20200611 would fail
>> during the wireguard dkms step, while trying to build wireguard for all
>> the kernels available.
>>
>> The problem gets more problematic when a newer kernel 4.4.0-184 gets
>> installed with the same apt upgrade: then, trying to downgrade wireguard
>> also fails; none of the 1.0.20200611 or 1.0.20200520 versions work
>> anymore with this combination of old/new kernels...
>>
>> To recap :
>> # wireguard 1.0.20200520
>>  - ok with kernels 4.4.0-148, 4.4.0-166
>>  - fails with kernel 4.4.0-184
>> # wireguard 1.0.20200611
>>  - fails with kernels 4.4.0-148, 4.4.0-166
>>  - ok with kernel 4.4.0-184
>>
>> (nb: we see it now with -184 but it could have started with an earlier
>> version)
>>
>> We could partially fix this by manually getting each deb/src and doing
>> dkms install:
>> dkms install wireguard/1.0.20200520 -k 4.4.0-148-generic
>> dkms install wireguard/1.0.20200611 -k 4.4.0-184-generic
>> ...
>> But apt upgrade is still broken.
>>
>> While we could boot onto the newer kernel and remove the older ones to
>> get rid of the problem, this situation would prevent from having a
>> "previous working" kernel on the system, which is not very safe.
>>
>> Could there be a fix for this made to wireguard-linux-compat for those
>> versions?
> Unfortunately, I don't have a super good solution for you right now.
> The wireguard-linux-compat repo is developed against the latest Ubuntu
> kernels that they put out once every three weeks. You can see them
> being tested at the bottom of
> <https://www.wireguard.com/build-status/>. The backport against
> upstream mainline kernels is z-granular (for an x.y.z versioning
> scheme), but Ubuntu's release cycle and versioning scheme is a bit too
> chaotic to make it reasonable to try to manage all the differences
> between their kernels every three weeks. So for distro kernels --
> Ubuntu, RHEL, Debian, etc -- we typically just develop against the
> latest one, and try to make sure that we release it at the right time
> so users aren't caught with no working version. This means,
> unfortunately, that when there are badly breaking changes, like in
> this last cycle, you have to uninstall the old kernels or mask them
> from dkms, in order to get dkms to avoid building for them and only
> building for the new kernel. There might be other more complicated
> solutions that closely track version dependencies or do compile time
> feature probing, but that comes with a maintenance burden far too
> arduous for a distro frankenkernel.
>
> But there is hope!
>
> Canonical is adding WireGuard to 18.04 and 16.04, and this is coming
> in two steps:
>
> Step 1) The wireguard-dkms and wireguard-tools packages will be added
> to the package archives, so that you won't have to use the PPA. This
> means that Canonical's kernel team will include wireguard-dkms in
> their development tests, so that they won't accidentally ship kernels
> with build breakage, like what you experienced last week.
>
> Step 2) The wireguard-dkms package will get built by Canonical,
> signed, and shipped alongside the other modules, so that you won't
> have to install wireguard-dkms, and it will just come out of the box
> with the normal kernel updates. This is already the case with 20.04
> and 19.10. They're working on it now for 18.04, and I really really
> hope to see that happen by the next cycle. And maybe if we ask apw
> (CC'd) nicely, he'll even do it for 16.04 too.
>
> Regards,
> Jason