Development discussion of WireGuard
 help / color / mirror / Atom feed
* Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs
@ 2020-05-22  8:04 Jason A. Donenfeld
  2020-05-22  8:44 ` Jason A. Donenfeld
                   ` (2 more replies)
  0 siblings, 3 replies; 12+ messages in thread
From: Jason A. Donenfeld @ 2020-05-22  8:04 UTC (permalink / raw)
  To: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team,
	Martin Hauke, Joe Doss, Neal Gompa, WireGuard mailing list

Hi distro kernel maintainers,

I currently have build.wireguard.com churning away at every new kernel
release between 3.10 and 5.5 for each and every wireguard-linux-compat
commit, making sure that we don't break anything. It's a pretty CPU
intensive, but it means we don't break mainline kernels ever when
releasing. Scroll down to the bottom to see what I mean.

This is not the case, however, with distro kernels, which differ
notoriously from mainline in surprising ways. There's been some
breakage, and unless we do something about it, I imagine there will
continue to be much breakage. Because so many users use Linux via your
kernels, it seems imperative that we support these well. And with
distros like Ubuntu now supporting WireGuard directly through
wireguard-linux-compat, it seems ever more important that we minimize
breakage so as to not create release delays downstream of us.

At the moment, wireguard-linux-compat supports:
- Debian oldoldstable (8), oldstable (9), stable (10), testing (11), sid
- Ubuntu 14.04, 16.04, 18.04, 19.04, 19.10, 20.04
- OpenSUSE 42, 15, 15.2 (leap)
- RHEL 7.8, 8.2
- CentOS 8.1

The logic here is that we support the latest single kernel for each
major supported release of these distros. For example, on Debian
stable, we support 4.19.0-9 but not 4.19.0-8 anymore.

I'd like to put these on build.wireguard.com to mitigate breakage. In
order to make this happen, I'll need two things:
- A URL I can scape that will give me the latest kernel versions for
relevant distros.
- A URL I can construct using a selected version to download a boring
kernel source tarball.
I would prefer to not involve git, if possible, and for these URLs to
point to the sources for actual kernels that are shipping as the
standard latest-kernel for each of the above releases.

If you can provide those pieces of information for my CI scripts, it
seems likely that we can drastically reduce breakage for your
distribution frankenkernels. If you cannot provide that, but other
distros can, I will probably naturally prioritize support for those
other distros that make maintenance easier, simply as a matter of
habit rather than something intentional.

Let me know if this is something that interests you. I'm happy to get
creative on what sorts of pages we scrape for this info.

Thanks,
Jason

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs
  2020-05-22  8:04 Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs Jason A. Donenfeld
@ 2020-05-22  8:44 ` Jason A. Donenfeld
  2020-05-23  5:59   ` Jason A. Donenfeld
  2020-05-22 12:43 ` Georg Faerber
  2020-05-23  7:07 ` Neal Gompa
  2 siblings, 1 reply; 12+ messages in thread
From: Jason A. Donenfeld @ 2020-05-22  8:44 UTC (permalink / raw)
  To: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team,
	Martin Hauke, Joe Doss, Neal Gompa, WireGuard mailing list

On Fri, May 22, 2020 at 02:04:58AM -0600, Jason A. Donenfeld wrote:
> I'd like to put these on build.wireguard.com to mitigate breakage. In
> order to make this happen, I'll need two things:
> - A URL I can scape that will give me the latest kernel versions for
> relevant distros.
> - A URL I can construct using a selected version to download a boring
> kernel source tarball.
> I would prefer to not involve git, if possible, and for these URLs to
> point to the sources for actual kernels that are shipping as the
> standard latest-kernel for each of the above releases.

Here's a bash example that achieves part 1, to give a more concrete
idea of what I'm after:

    declare -A UBUNTU_SERIES_KERNELS
    while read -r line; do
        [[ $line =~ ^\ *linux\ +\|\ ([0-9.-]+)\ +\|\ ([a-z]+)(-[a-z]+)?\ +\|\ source$ ]] || continue
        version="${BASH_REMATCH[1]}" series="${BASH_REMATCH[2]}" repo="${BASH_REMATCH[3]}"
        [[ $repo != -proposed ]] || continue
        read -r smaller < <(printf '%s\n3.10\n' "$version" | sort -V)
        [[ $smaller == 3.10 ]] || continue
        UBUNTU_SERIES_KERNELS[$series]+="$version "
    done < <(curl -Ls https://people.canonical.com/~ubuntu-archive/madison.cgi?package=linux&a=&c=&s=)
    UBUNTU_KERNELS=( )
    for series in "${!UBUNTU_SERIES_KERNELS[@]}"; do
        read -r kernel < <(sort -Vur <<<"${UBUNTU_SERIES_KERNELS[$series]// /$'\n'}")
        [[ -n $kernel ]] && UBUNTU_KERNELS+=( "$kernel" )
    done
    printf -- '- %s\n' "${UBUNTU_KERNELS[@]}"

The output of this script is something like:

- 4.15.0-101.102
- 5.4.0-26.30
- 5.4.0-31.35
- 4.4.0-179.209
- 3.13.0-170.220
- 5.3.0-53.47
- 5.0.0-38.41

The next step would be turning these version numbers into URLs to the
tarballs. Hopefully the team at Canonical can assist with providing
that.

Jason

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs
  2020-05-22  8:04 Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs Jason A. Donenfeld
  2020-05-22  8:44 ` Jason A. Donenfeld
@ 2020-05-22 12:43 ` Georg Faerber
  2020-05-23 12:31   ` Georg Faerber
  2020-05-23  7:07 ` Neal Gompa
  2 siblings, 1 reply; 12+ messages in thread
From: Georg Faerber @ 2020-05-22 12:43 UTC (permalink / raw)
  To: wireguard

Hi,

Regarding the Debian-side of things:

On 20-05-22 02:04:58, Jason A. Donenfeld wrote:
> I'd like to put these on build.wireguard.com to mitigate breakage. In
> order to make this happen, I'll need two things:
> - A URL I can scape that will give me the latest kernel versions for
> relevant distros.

Versions are available via [1], the 'f' param returns json output.
API docs are available via [2].

> - A URL I can construct using a selected version to download a boring
> kernel source tarball.
> I would prefer to not involve git, if possible, and for these URLs to
> point to the sources for actual kernels that are shipping as the
> standard latest-kernel for each of the above releases.

One way to get the source tarball, without relying on git, is using
[3], specifically [4]. To construct the URL to fetch the source tarball
from, one would need to concatenate:

https://snapshot.debian.org/archive/debian/ + $first_seen + $path + $name

An example URL would look like [5].

I'm not sure if there are API docs available.

Hope this helps,
cheers,
Georg


[1] https://api.ftp-master.debian.org/madison?package=linux&f
[2] https://ftp-team.pages.debian.net/dak/epydoc/dakweb.queries.madison-module.html
[3] https://snapshot.debian.org/
[4] https://snapshot.debian.org/mr/package/linux/5.6.7-1/srcfiles?fileinfo=1
[5] https://snapshot.debian.org/archive/debian/20200429T204608Z/pool/main/l/linux/linux_5.6.7.orig.tar.xz

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs
  2020-05-22  8:44 ` Jason A. Donenfeld
@ 2020-05-23  5:59   ` Jason A. Donenfeld
  0 siblings, 0 replies; 12+ messages in thread
From: Jason A. Donenfeld @ 2020-05-23  5:59 UTC (permalink / raw)
  To: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team,
	Martin Hauke, Joe Doss, Neal Gompa, WireGuard mailing list

Hey Canonical,

Things are now live on build.wireguard.com. Scroll to the bottom of [1]
to find your kernels. I haven't received much input from you on what
kernels you wanted me to develop against, so I've gone ahead with this
jenky thing:

declare -A URL_MAP
for series in focal eoan bionic xenial trusty; do
        refs="$(curl https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/$series/info/refs?service=git-upload-pack)"
        read -r kernel < <(while read -r line; do
                [[ $line =~ ^[a-f0-9]{44}\ refs/tags/Ubuntu-([0-9.-]+)$ ]] || continue
                echo "${BASH_REMATCH[1]}"
        done <<<"$refs" | sort -Vur) || unset kernel
        [[ -z $kernel ]] || URL_MAP["$kernel-ubuntu-$series"]="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/$series/snapshot/Ubuntu-$kernel.tar.gz"
        read -r kernel < <(while read -r line; do
                [[ $line =~ ^[a-f0-9]{44}\ refs/tags/Ubuntu-hwe-([0-9._-]+)$ ]] || continue
                echo "${BASH_REMATCH[1]}"
        done <<<"$refs" | sort -Vur) || unset kernel
        [[ -z $kernel ]] || URL_MAP["$kernel-ubuntu-$series-hwe"]="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/$series/snapshot/Ubuntu-hwe-$kernel.tar.gz"
done

The output from that highly grotesque bit of bash is:

$ for k in "${!URL_MAP[@]}"; do echo "$k: ${URL_MAP[$k]}"; done | sort -Vur
5.4.0-33.37-ubuntu-focal: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/focal/snapshot/Ubuntu-5.4.0-33.37.tar.gz
5.4-5.4.0-31.35_18.04.2-ubuntu-bionic-hwe: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/bionic/snapshot/Ubuntu-hwe-5.4-5.4.0-31.35_18.04.2.tar.gz
5.3.0-55.49-ubuntu-eoan: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/eoan/snapshot/Ubuntu-5.3.0-55.49.tar.gz
4.15.0-102.103_16.04.1-ubuntu-xenial-hwe: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/snapshot/Ubuntu-hwe-4.15.0-102.103_16.04.1.tar.gz
4.15.0-102.103-ubuntu-bionic: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/bionic/snapshot/Ubuntu-4.15.0-102.103.tar.gz
4.4.0-1033.40-ubuntu-xenial: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/snapshot/Ubuntu-4.4.0-1033.40.tar.gz
3.13.0-169.219-ubuntu-trusty: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/trusty/snapshot/Ubuntu-3.13.0-169.219.tar.gz

In coming up with this without your input there are several downsides,
due in part to my own ignorance about how your kernels are organized:

- Are these actually the kernels you want me to develop against? I
  suspect there's a better list from elsewhere, but I don't know how to
  find that.

- Downloading dynamically generated tarballs from cgit seems suboptimal
  for your server, since it has to gzip on the fly. I would very much
  understand if you preferred me to use a different tarball source.

- I have to detect UTS_UBUNTU_RELEASE_ABI in rules.d and fish it out, to
  detect Ubuntu kernels. It'd be better if the tarball contained the
  actual source that you're compiling.

As well, in developing against these kernels, I'll be making snapshot
releases of the packages at the usual regular intervals when all of
these are green. Since I only support one Ubuntu release kernel at a
time, this means I might break older kernels. For example, I might make
a fix for 5.4.0-33.37-focal that breaks 5.4.0-32.36-focal. That's fine
by me and expected, but it does mean that if I'm developing against the
latest and greatest, you'll want to coordinate when you SRU the new dkms
packages. This seems like it won't be a big deal anyhow with the
prebuilt wireguard.ko, but it is an important detail to keep in mind.
And of course, if you want me to develop against a different set of
kernels, I'm more than happy to scrap my gross cgit tag scraping and use
something different that you prefer.

Preliminary usage of this CI has enabled me to adjust the compat layer
for your upcoming 18.04 kernel (and 16.04-hwe) [3]. So already that's a
nice win for the new CI: squashing bugs before they happen in the wild.

However, you'll see on the build-status page [1] that (as of writing),
compilation fails for 5.4.0-33.37, 5.4-5.4.0-31.35_18.04.2, and
3.13.0-169.219, for reasons other than wireguard. I think I see what's
wrong with those kernel trees and could send patches to fix them, but
I'm not sure that'd be appropriate: ultimately non-wireguard fixes to
your commercially supported kernels are your responsibility. (Plus the
last time I sent a patch to fix your 3.13 kernel, it was ignored for a
few weeks before finally being nack'd because that's only a kernel for
paid customers or something confusing like that [2]). So I'll leave it
to you to fix these and get the CI all green again. But the fact that
some of these kernels do have unrelated breakage reinforces my original
question in this email: am I developing and testing against the right
set of kernels? Those build failures make me suspect I've done something
wrong in my kernel selection algorithm above.

So, let me know what you'd like to do here. I think the work I've done
here should give us a pretty easy basis on which to proceed. But I will
need a bit of input on what kernels I should actually be focusing on, so
that this is actually a productive use of development time.

Thanks,
Jason

PS: Your 4.4 kernel is missing [4], which prohibits it from compiling on
gcc≥7. This makes diagnostics from newer compilers impossible and
complicates the build system a bit. Greg backported this to 4.4.57 in
2017, so it should be pretty easy for you guys to cherry pick.

[1] https://www.wireguard.com/build-status/
[2] https://lists.ubuntu.com/archives/kernel-team/2020-May/109542.html
[3] https://git.zx2c4.com/wireguard-linux-compat/commit/?id=997bef04a20533aadab3a4dd13d05e58191102bd
[4] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-4.4.y&id=4cb0c0b73d1c79a8ce260836b3f27650aa1c57f1

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs
  2020-05-22  8:04 Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs Jason A. Donenfeld
  2020-05-22  8:44 ` Jason A. Donenfeld
  2020-05-22 12:43 ` Georg Faerber
@ 2020-05-23  7:07 ` Neal Gompa
  2020-05-23  7:21   ` Jason A. Donenfeld
  2 siblings, 1 reply; 12+ messages in thread
From: Neal Gompa @ 2020-05-23  7:07 UTC (permalink / raw)
  To: Jason A. Donenfeld
  Cc: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team,
	Martin Hauke, Joe Doss, WireGuard mailing list, Carl George

On Fri, May 22, 2020 at 4:05 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
>
> Hi distro kernel maintainers,
>
> I currently have build.wireguard.com churning away at every new kernel
> release between 3.10 and 5.5 for each and every wireguard-linux-compat
> commit, making sure that we don't break anything. It's a pretty CPU
> intensive, but it means we don't break mainline kernels ever when
> releasing. Scroll down to the bottom to see what I mean.
>
> This is not the case, however, with distro kernels, which differ
> notoriously from mainline in surprising ways. There's been some
> breakage, and unless we do something about it, I imagine there will
> continue to be much breakage. Because so many users use Linux via your
> kernels, it seems imperative that we support these well. And with
> distros like Ubuntu now supporting WireGuard directly through
> wireguard-linux-compat, it seems ever more important that we minimize
> breakage so as to not create release delays downstream of us.
>
> At the moment, wireguard-linux-compat supports:
> - Debian oldoldstable (8), oldstable (9), stable (10), testing (11), sid
> - Ubuntu 14.04, 16.04, 18.04, 19.04, 19.10, 20.04
> - OpenSUSE 42, 15, 15.2 (leap)
> - RHEL 7.8, 8.2
> - CentOS 8.1
>
> The logic here is that we support the latest single kernel for each
> major supported release of these distros. For example, on Debian
> stable, we support 4.19.0-9 but not 4.19.0-8 anymore.
>
> I'd like to put these on build.wireguard.com to mitigate breakage. In
> order to make this happen, I'll need two things:
> - A URL I can scape that will give me the latest kernel versions for
> relevant distros.
> - A URL I can construct using a selected version to download a boring
> kernel source tarball.
> I would prefer to not involve git, if possible, and for these URLs to
> point to the sources for actual kernels that are shipping as the
> standard latest-kernel for each of the above releases.
>
> If you can provide those pieces of information for my CI scripts, it
> seems likely that we can drastically reduce breakage for your
> distribution frankenkernels. If you cannot provide that, but other
> distros can, I will probably naturally prioritize support for those
> other distros that make maintenance easier, simply as a matter of
> habit rather than something intentional.
>

For CentOS kernels, you can query the kernel builds that have been
made for CentOS 7 and CentOS 8 by requesting the list of tags from the
kernel package[1].

The Pagure API[2] provides a straightforward way to do this, even with
curl and jq:

$ curl --silent --header "Content-Type: application/json" \
https://git.centos.org/api/0/rpms/kernel/git/tags | jq '.["tags"] '

This will return a JSON list of tags. For CentOS 7, you'd want the
tags beginning with "imports/c7/". For CentOS 8, you'd want the tags
beginning with "imports/c8/". The string after the prefix is the NVR
(name-version-release) of the package, and is the main part of the
Source RPM file name. The highest NVR in each tag namespace is the
latest package. There are various ways to do the comparison, but the
simplest one would be to sort them with the highest number on top, as
the numbers are guaranteed to numerically advance. I'm not good with
jq, so I couldn't figure out how to do it as a one-liner.

Sources are shipped in the form of Source RPMs and pushed to the
CentOS Vault[3][4]. Sources are archived at each CentOS point release,
meaning that you need to be aware of the CentOS point release where
the kernel was shipped.

So for example, for CentOS 7.8, you'd be looking at the following
locations for source packages:
* http://vault.centos.org/7.8.2003/os/Source/SPackages/
* http://vault.centos.org/7.8.2003/updates/Source/SPackages/

CentOS 8.1 is a bit simpler, with only the one location:
http://vault.centos.org/8.1.1911/BaseOS/Source/SPackages/

With the information from the API and knowing the location of where
the packages are, you can fetch a kernel source package like so:

$ wget http://vault.centos.org/8.1.1911/BaseOS/Source/SPackages/${kernel_version_release}.src.rpm

All the stuff in the source RPM is enough to rebuild the kernel. If
you just want to grab the tarball inside, then you can grab it by
unpacking the Source RPM and grabbing the tarball inside.


[1]: https://git.centos.org/rpms/kernel
[2]: https://pagure.io/api
[3]: http://vault.centos.org/7.8.2003/
[4]: http://vault.centos.org/8.1.1911/



--
真実はいつも一つ!/ Always, there's only one truth!

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs
  2020-05-23  7:07 ` Neal Gompa
@ 2020-05-23  7:21   ` Jason A. Donenfeld
  2020-05-23 11:49     ` Neal Gompa
  0 siblings, 1 reply; 12+ messages in thread
From: Jason A. Donenfeld @ 2020-05-23  7:21 UTC (permalink / raw)
  To: Neal Gompa
  Cc: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team,
	Martin Hauke, Joe Doss, WireGuard mailing list, Carl George

Hi Neal,

Thanks for these very helpful instructions. In trying to implement
this, one snag I encountered was that the .src.rpm file has inside of
it the linux-{version}.tar.xz file that I actually want. Is there
anywhere on the internet that hosts these .tar.xz files directly?

Jason

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs
  2020-05-23  7:21   ` Jason A. Donenfeld
@ 2020-05-23 11:49     ` Neal Gompa
  2020-05-24  0:01       ` Jason A. Donenfeld
  0 siblings, 1 reply; 12+ messages in thread
From: Neal Gompa @ 2020-05-23 11:49 UTC (permalink / raw)
  To: Jason A. Donenfeld
  Cc: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team,
	Martin Hauke, Joe Doss, WireGuard mailing list, Carl George

On Sat, May 23, 2020 at 3:21 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
>
> Hi Neal,
>
> Thanks for these very helpful instructions. In trying to implement
> this, one snag I encountered was that the .src.rpm file has inside of
> it the linux-{version}.tar.xz file that I actually want. Is there
> anywhere on the internet that hosts these .tar.xz files directly?
>

There *is*, but that starts getting into using the CentOS Git system
more directly. Per your original email, you weren't interested in that
path, so I didn't go there.


-- 
真実はいつも一つ!/ Always, there's only one truth!

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs
  2020-05-22 12:43 ` Georg Faerber
@ 2020-05-23 12:31   ` Georg Faerber
  0 siblings, 0 replies; 12+ messages in thread
From: Georg Faerber @ 2020-05-23 12:31 UTC (permalink / raw)
  To: wireguard

Hi,

The example URL was referring to the orig source, which is probably not
of interest:

On 20-05-22 12:43:44, Georg Faerber wrote:
> [5] https://snapshot.debian.org/archive/debian/20200429T204608Z/pool/main/l/linux/linux_5.6.7.orig.tar.xz

The Debian stuff is included in:

https://snapshot.debian.org/archive/debian/20200429T204608Z/pool/main/l/linux/linux_5.6.7-1.debian.tar.xz

Cheers,
Georg

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs
  2020-05-23 11:49     ` Neal Gompa
@ 2020-05-24  0:01       ` Jason A. Donenfeld
  2020-05-24  0:20         ` Neal Gompa
  0 siblings, 1 reply; 12+ messages in thread
From: Jason A. Donenfeld @ 2020-05-24  0:01 UTC (permalink / raw)
  To: Neal Gompa
  Cc: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team,
	Martin Hauke, Joe Doss, WireGuard mailing list, Carl George

On Sat, May 23, 2020 at 5:50 AM Neal Gompa <ngompa13@gmail.com> wrote:
>
> On Sat, May 23, 2020 at 3:21 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> >
> > Hi Neal,
> >
> > Thanks for these very helpful instructions. In trying to implement
> > this, one snag I encountered was that the .src.rpm file has inside of
> > it the linux-{version}.tar.xz file that I actually want. Is there
> > anywhere on the internet that hosts these .tar.xz files directly?
> >
>
> There *is*, but that starts getting into using the CentOS Git system
> more directly. Per your original email, you weren't interested in that
> path, so I didn't go there.

Well, one trick is that most web git viewers (such as cgit) can
produce a tarball on the fly from a URL. Something like this exist
here?

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs
  2020-05-24  0:01       ` Jason A. Donenfeld
@ 2020-05-24  0:20         ` Neal Gompa
  2020-05-24  1:10           ` Jason A. Donenfeld
  0 siblings, 1 reply; 12+ messages in thread
From: Neal Gompa @ 2020-05-24  0:20 UTC (permalink / raw)
  To: Jason A. Donenfeld
  Cc: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team,
	Martin Hauke, Joe Doss, WireGuard mailing list, Carl George

On Sat, May 23, 2020 at 8:01 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
>
> On Sat, May 23, 2020 at 5:50 AM Neal Gompa <ngompa13@gmail.com> wrote:
> >
> > On Sat, May 23, 2020 at 3:21 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> > >
> > > Hi Neal,
> > >
> > > Thanks for these very helpful instructions. In trying to implement
> > > this, one snag I encountered was that the .src.rpm file has inside of
> > > it the linux-{version}.tar.xz file that I actually want. Is there
> > > anywhere on the internet that hosts these .tar.xz files directly?
> > >
> >
> > There *is*, but that starts getting into using the CentOS Git system
> > more directly. Per your original email, you weren't interested in that
> > path, so I didn't go there.
>
> Well, one trick is that most web git viewers (such as cgit) can
> produce a tarball on the fly from a URL. Something like this exist
> here?

So, CentOS uses a Dist-Git[1] system, which means the part that is
actually tracked in git is the packaging files, and there's an
associated binary repo reference stored in the Git repo for fetching
binary artifacts. Dist-Git is the common ancestor to virtually all
current approaches to storing binaries efficiently alongside a Git
repo (Git-Annex and Git LFS came much later).

It seems it's possible to do this without having to use Git itself
(which is better than with git-annex and git-lfs), but it's a bit
cumbersome...

The way that you'd fetch the tarball in this scenario would be to
something like the following:

1. Get the tags via API, but with commits:

$ curl --silent --header "Content-Type: application/json" \
https://git.centos.org/api/0/rpms/kernel/git/tags?with_commits=true \
| jq '.["tags"] '

This gives you now a JSON dictionary with the tags as keys and the
associated commit as values.

2. Get the corresponding commit and fetch the .kernel.metadata file
containing the lookaside references:

$ curl --silent
https://git.centos.org/rpms/kernel/raw/${kernel_tag_commit}/f/.kernel.metadata \
> kernel_sources

3. Fetch the source matching to your kernel source package:

$ wget https://git.centos.org/sources/kernel/${rel}/${kernel_src_checksum} \
--output-document=${kernel_src_filename}

The $kernel_src_checksum and $kernel_src_filename would be populated
from the entries in the kernel_sources files you downloaded earlier.

Note: $rel is either "c7" or "c8" depending on which one you're trying to get.


[1]: https://github.com/release-engineering/dist-git





--
真実はいつも一つ!/ Always, there's only one truth!

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs
  2020-05-24  0:20         ` Neal Gompa
@ 2020-05-24  1:10           ` Jason A. Donenfeld
  2020-05-24  1:41             ` Jason A. Donenfeld
  0 siblings, 1 reply; 12+ messages in thread
From: Jason A. Donenfeld @ 2020-05-24  1:10 UTC (permalink / raw)
  To: Neal Gompa
  Cc: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team,
	Martin Hauke, Joe Doss, WireGuard mailing list, Carl George

On Sat, May 23, 2020 at 6:21 PM Neal Gompa <ngompa13@gmail.com> wrote:
>
> On Sat, May 23, 2020 at 8:01 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> >
> > On Sat, May 23, 2020 at 5:50 AM Neal Gompa <ngompa13@gmail.com> wrote:
> > >
> > > On Sat, May 23, 2020 at 3:21 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> > > >
> > > > Hi Neal,
> > > >
> > > > Thanks for these very helpful instructions. In trying to implement
> > > > this, one snag I encountered was that the .src.rpm file has inside of
> > > > it the linux-{version}.tar.xz file that I actually want. Is there
> > > > anywhere on the internet that hosts these .tar.xz files directly?
> > > >
> > >
> > > There *is*, but that starts getting into using the CentOS Git system
> > > more directly. Per your original email, you weren't interested in that
> > > path, so I didn't go there.
> >
> > Well, one trick is that most web git viewers (such as cgit) can
> > produce a tarball on the fly from a URL. Something like this exist
> > here?
>
> So, CentOS uses a Dist-Git[1] system, which means the part that is
> actually tracked in git is the packaging files, and there's an
> associated binary repo reference stored in the Git repo for fetching
> binary artifacts. Dist-Git is the common ancestor to virtually all
> current approaches to storing binaries efficiently alongside a Git
> repo (Git-Annex and Git LFS came much later).
>
> It seems it's possible to do this without having to use Git itself
> (which is better than with git-annex and git-lfs), but it's a bit
> cumbersome...
>
> The way that you'd fetch the tarball in this scenario would be to
> something like the following:
>
> 1. Get the tags via API, but with commits:
>
> $ curl --silent --header "Content-Type: application/json" \
> https://git.centos.org/api/0/rpms/kernel/git/tags?with_commits=true \
> | jq '.["tags"] '
>
> This gives you now a JSON dictionary with the tags as keys and the
> associated commit as values.
>
> 2. Get the corresponding commit and fetch the .kernel.metadata file
> containing the lookaside references:
>
> $ curl --silent
> https://git.centos.org/rpms/kernel/raw/${kernel_tag_commit}/f/.kernel.metadata \
> > kernel_sources
>
> 3. Fetch the source matching to your kernel source package:
>
> $ wget https://git.centos.org/sources/kernel/${rel}/${kernel_src_checksum} \
> --output-document=${kernel_src_filename}
>
> The $kernel_src_checksum and $kernel_src_filename would be populated
> from the entries in the kernel_sources files you downloaded earlier.
>
> Note: $rel is either "c7" or "c8" depending on which one you're trying to get.
>
>
> [1]: https://github.com/release-engineering/dist-git

Perfect, thanks. CentOS kernels are now on
https://www.wireguard.com/build-status/ , though el7 is having build
issues.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs
  2020-05-24  1:10           ` Jason A. Donenfeld
@ 2020-05-24  1:41             ` Jason A. Donenfeld
  0 siblings, 0 replies; 12+ messages in thread
From: Jason A. Donenfeld @ 2020-05-24  1:41 UTC (permalink / raw)
  To: Neal Gompa
  Cc: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team,
	Martin Hauke, Joe Doss, WireGuard mailing list, Carl George

On Sat, May 23, 2020 at 7:10 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
>
> On Sat, May 23, 2020 at 6:21 PM Neal Gompa <ngompa13@gmail.com> wrote:
> >
> > On Sat, May 23, 2020 at 8:01 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> > >
> > > On Sat, May 23, 2020 at 5:50 AM Neal Gompa <ngompa13@gmail.com> wrote:
> > > >
> > > > On Sat, May 23, 2020 at 3:21 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> > > > >
> > > > > Hi Neal,
> > > > >
> > > > > Thanks for these very helpful instructions. In trying to implement
> > > > > this, one snag I encountered was that the .src.rpm file has inside of
> > > > > it the linux-{version}.tar.xz file that I actually want. Is there
> > > > > anywhere on the internet that hosts these .tar.xz files directly?
> > > > >
> > > >
> > > > There *is*, but that starts getting into using the CentOS Git system
> > > > more directly. Per your original email, you weren't interested in that
> > > > path, so I didn't go there.
> > >
> > > Well, one trick is that most web git viewers (such as cgit) can
> > > produce a tarball on the fly from a URL. Something like this exist
> > > here?
> >
> > So, CentOS uses a Dist-Git[1] system, which means the part that is
> > actually tracked in git is the packaging files, and there's an
> > associated binary repo reference stored in the Git repo for fetching
> > binary artifacts. Dist-Git is the common ancestor to virtually all
> > current approaches to storing binaries efficiently alongside a Git
> > repo (Git-Annex and Git LFS came much later).
> >
> > It seems it's possible to do this without having to use Git itself
> > (which is better than with git-annex and git-lfs), but it's a bit
> > cumbersome...
> >
> > The way that you'd fetch the tarball in this scenario would be to
> > something like the following:
> >
> > 1. Get the tags via API, but with commits:
> >
> > $ curl --silent --header "Content-Type: application/json" \
> > https://git.centos.org/api/0/rpms/kernel/git/tags?with_commits=true \
> > | jq '.["tags"] '
> >
> > This gives you now a JSON dictionary with the tags as keys and the
> > associated commit as values.
> >
> > 2. Get the corresponding commit and fetch the .kernel.metadata file
> > containing the lookaside references:
> >
> > $ curl --silent
> > https://git.centos.org/rpms/kernel/raw/${kernel_tag_commit}/f/.kernel.metadata \
> > > kernel_sources
> >
> > 3. Fetch the source matching to your kernel source package:
> >
> > $ wget https://git.centos.org/sources/kernel/${rel}/${kernel_src_checksum} \
> > --output-document=${kernel_src_filename}
> >
> > The $kernel_src_checksum and $kernel_src_filename would be populated
> > from the entries in the kernel_sources files you downloaded earlier.
> >
> > Note: $rel is either "c7" or "c8" depending on which one you're trying to get.
> >
> >
> > [1]: https://github.com/release-engineering/dist-git
>
> Perfect, thanks. CentOS kernels are now on
> https://www.wireguard.com/build-status/ , though el7 is having build
> issues.

c7 issue: https://bugzilla.redhat.com/show_bug.cgi?id=1839419
c8 issue: https://bugzilla.redhat.com/show_bug.cgi?id=1839415
c8s issue: https://bugzilla.redhat.com/show_bug.cgi?id=1839421

I've hacked around the c8 issue in the build system, but the c7 and
c8s issues will need to be fixed by Red Hat.

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2020-05-24  2:15 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-05-22  8:04 Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs Jason A. Donenfeld
2020-05-22  8:44 ` Jason A. Donenfeld
2020-05-23  5:59   ` Jason A. Donenfeld
2020-05-22 12:43 ` Georg Faerber
2020-05-23 12:31   ` Georg Faerber
2020-05-23  7:07 ` Neal Gompa
2020-05-23  7:21   ` Jason A. Donenfeld
2020-05-23 11:49     ` Neal Gompa
2020-05-24  0:01       ` Jason A. Donenfeld
2020-05-24  0:20         ` Neal Gompa
2020-05-24  1:10           ` Jason A. Donenfeld
2020-05-24  1:41             ` Jason A. Donenfeld

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).