* Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs @ 2020-05-22 8:04 Jason A. Donenfeld 2020-05-22 8:44 ` Jason A. Donenfeld ` (2 more replies) 0 siblings, 3 replies; 12+ messages in thread From: Jason A. Donenfeld @ 2020-05-22 8:04 UTC (permalink / raw) To: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team, Martin Hauke, Joe Doss, Neal Gompa, WireGuard mailing list Hi distro kernel maintainers, I currently have build.wireguard.com churning away at every new kernel release between 3.10 and 5.5 for each and every wireguard-linux-compat commit, making sure that we don't break anything. It's a pretty CPU intensive, but it means we don't break mainline kernels ever when releasing. Scroll down to the bottom to see what I mean. This is not the case, however, with distro kernels, which differ notoriously from mainline in surprising ways. There's been some breakage, and unless we do something about it, I imagine there will continue to be much breakage. Because so many users use Linux via your kernels, it seems imperative that we support these well. And with distros like Ubuntu now supporting WireGuard directly through wireguard-linux-compat, it seems ever more important that we minimize breakage so as to not create release delays downstream of us. At the moment, wireguard-linux-compat supports: - Debian oldoldstable (8), oldstable (9), stable (10), testing (11), sid - Ubuntu 14.04, 16.04, 18.04, 19.04, 19.10, 20.04 - OpenSUSE 42, 15, 15.2 (leap) - RHEL 7.8, 8.2 - CentOS 8.1 The logic here is that we support the latest single kernel for each major supported release of these distros. For example, on Debian stable, we support 4.19.0-9 but not 4.19.0-8 anymore. I'd like to put these on build.wireguard.com to mitigate breakage. In order to make this happen, I'll need two things: - A URL I can scape that will give me the latest kernel versions for relevant distros. - A URL I can construct using a selected version to download a boring kernel source tarball. I would prefer to not involve git, if possible, and for these URLs to point to the sources for actual kernels that are shipping as the standard latest-kernel for each of the above releases. If you can provide those pieces of information for my CI scripts, it seems likely that we can drastically reduce breakage for your distribution frankenkernels. If you cannot provide that, but other distros can, I will probably naturally prioritize support for those other distros that make maintenance easier, simply as a matter of habit rather than something intentional. Let me know if this is something that interests you. I'm happy to get creative on what sorts of pages we scrape for this info. Thanks, Jason ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs 2020-05-22 8:04 Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs Jason A. Donenfeld @ 2020-05-22 8:44 ` Jason A. Donenfeld 2020-05-23 5:59 ` Jason A. Donenfeld 2020-05-22 12:43 ` Georg Faerber 2020-05-23 7:07 ` Neal Gompa 2 siblings, 1 reply; 12+ messages in thread From: Jason A. Donenfeld @ 2020-05-22 8:44 UTC (permalink / raw) To: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team, Martin Hauke, Joe Doss, Neal Gompa, WireGuard mailing list On Fri, May 22, 2020 at 02:04:58AM -0600, Jason A. Donenfeld wrote: > I'd like to put these on build.wireguard.com to mitigate breakage. In > order to make this happen, I'll need two things: > - A URL I can scape that will give me the latest kernel versions for > relevant distros. > - A URL I can construct using a selected version to download a boring > kernel source tarball. > I would prefer to not involve git, if possible, and for these URLs to > point to the sources for actual kernels that are shipping as the > standard latest-kernel for each of the above releases. Here's a bash example that achieves part 1, to give a more concrete idea of what I'm after: declare -A UBUNTU_SERIES_KERNELS while read -r line; do [[ $line =~ ^\ *linux\ +\|\ ([0-9.-]+)\ +\|\ ([a-z]+)(-[a-z]+)?\ +\|\ source$ ]] || continue version="${BASH_REMATCH[1]}" series="${BASH_REMATCH[2]}" repo="${BASH_REMATCH[3]}" [[ $repo != -proposed ]] || continue read -r smaller < <(printf '%s\n3.10\n' "$version" | sort -V) [[ $smaller == 3.10 ]] || continue UBUNTU_SERIES_KERNELS[$series]+="$version " done < <(curl -Ls https://people.canonical.com/~ubuntu-archive/madison.cgi?package=linux&a=&c=&s=) UBUNTU_KERNELS=( ) for series in "${!UBUNTU_SERIES_KERNELS[@]}"; do read -r kernel < <(sort -Vur <<<"${UBUNTU_SERIES_KERNELS[$series]// /$'\n'}") [[ -n $kernel ]] && UBUNTU_KERNELS+=( "$kernel" ) done printf -- '- %s\n' "${UBUNTU_KERNELS[@]}" The output of this script is something like: - 4.15.0-101.102 - 5.4.0-26.30 - 5.4.0-31.35 - 4.4.0-179.209 - 3.13.0-170.220 - 5.3.0-53.47 - 5.0.0-38.41 The next step would be turning these version numbers into URLs to the tarballs. Hopefully the team at Canonical can assist with providing that. Jason ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs 2020-05-22 8:44 ` Jason A. Donenfeld @ 2020-05-23 5:59 ` Jason A. Donenfeld 0 siblings, 0 replies; 12+ messages in thread From: Jason A. Donenfeld @ 2020-05-23 5:59 UTC (permalink / raw) To: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team, Martin Hauke, Joe Doss, Neal Gompa, WireGuard mailing list Hey Canonical, Things are now live on build.wireguard.com. Scroll to the bottom of [1] to find your kernels. I haven't received much input from you on what kernels you wanted me to develop against, so I've gone ahead with this jenky thing: declare -A URL_MAP for series in focal eoan bionic xenial trusty; do refs="$(curl https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/$series/info/refs?service=git-upload-pack)" read -r kernel < <(while read -r line; do [[ $line =~ ^[a-f0-9]{44}\ refs/tags/Ubuntu-([0-9.-]+)$ ]] || continue echo "${BASH_REMATCH[1]}" done <<<"$refs" | sort -Vur) || unset kernel [[ -z $kernel ]] || URL_MAP["$kernel-ubuntu-$series"]="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/$series/snapshot/Ubuntu-$kernel.tar.gz" read -r kernel < <(while read -r line; do [[ $line =~ ^[a-f0-9]{44}\ refs/tags/Ubuntu-hwe-([0-9._-]+)$ ]] || continue echo "${BASH_REMATCH[1]}" done <<<"$refs" | sort -Vur) || unset kernel [[ -z $kernel ]] || URL_MAP["$kernel-ubuntu-$series-hwe"]="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/$series/snapshot/Ubuntu-hwe-$kernel.tar.gz" done The output from that highly grotesque bit of bash is: $ for k in "${!URL_MAP[@]}"; do echo "$k: ${URL_MAP[$k]}"; done | sort -Vur 5.4.0-33.37-ubuntu-focal: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/focal/snapshot/Ubuntu-5.4.0-33.37.tar.gz 5.4-5.4.0-31.35_18.04.2-ubuntu-bionic-hwe: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/bionic/snapshot/Ubuntu-hwe-5.4-5.4.0-31.35_18.04.2.tar.gz 5.3.0-55.49-ubuntu-eoan: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/eoan/snapshot/Ubuntu-5.3.0-55.49.tar.gz 4.15.0-102.103_16.04.1-ubuntu-xenial-hwe: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/snapshot/Ubuntu-hwe-4.15.0-102.103_16.04.1.tar.gz 4.15.0-102.103-ubuntu-bionic: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/bionic/snapshot/Ubuntu-4.15.0-102.103.tar.gz 4.4.0-1033.40-ubuntu-xenial: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/snapshot/Ubuntu-4.4.0-1033.40.tar.gz 3.13.0-169.219-ubuntu-trusty: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/trusty/snapshot/Ubuntu-3.13.0-169.219.tar.gz In coming up with this without your input there are several downsides, due in part to my own ignorance about how your kernels are organized: - Are these actually the kernels you want me to develop against? I suspect there's a better list from elsewhere, but I don't know how to find that. - Downloading dynamically generated tarballs from cgit seems suboptimal for your server, since it has to gzip on the fly. I would very much understand if you preferred me to use a different tarball source. - I have to detect UTS_UBUNTU_RELEASE_ABI in rules.d and fish it out, to detect Ubuntu kernels. It'd be better if the tarball contained the actual source that you're compiling. As well, in developing against these kernels, I'll be making snapshot releases of the packages at the usual regular intervals when all of these are green. Since I only support one Ubuntu release kernel at a time, this means I might break older kernels. For example, I might make a fix for 5.4.0-33.37-focal that breaks 5.4.0-32.36-focal. That's fine by me and expected, but it does mean that if I'm developing against the latest and greatest, you'll want to coordinate when you SRU the new dkms packages. This seems like it won't be a big deal anyhow with the prebuilt wireguard.ko, but it is an important detail to keep in mind. And of course, if you want me to develop against a different set of kernels, I'm more than happy to scrap my gross cgit tag scraping and use something different that you prefer. Preliminary usage of this CI has enabled me to adjust the compat layer for your upcoming 18.04 kernel (and 16.04-hwe) [3]. So already that's a nice win for the new CI: squashing bugs before they happen in the wild. However, you'll see on the build-status page [1] that (as of writing), compilation fails for 5.4.0-33.37, 5.4-5.4.0-31.35_18.04.2, and 3.13.0-169.219, for reasons other than wireguard. I think I see what's wrong with those kernel trees and could send patches to fix them, but I'm not sure that'd be appropriate: ultimately non-wireguard fixes to your commercially supported kernels are your responsibility. (Plus the last time I sent a patch to fix your 3.13 kernel, it was ignored for a few weeks before finally being nack'd because that's only a kernel for paid customers or something confusing like that [2]). So I'll leave it to you to fix these and get the CI all green again. But the fact that some of these kernels do have unrelated breakage reinforces my original question in this email: am I developing and testing against the right set of kernels? Those build failures make me suspect I've done something wrong in my kernel selection algorithm above. So, let me know what you'd like to do here. I think the work I've done here should give us a pretty easy basis on which to proceed. But I will need a bit of input on what kernels I should actually be focusing on, so that this is actually a productive use of development time. Thanks, Jason PS: Your 4.4 kernel is missing [4], which prohibits it from compiling on gcc≥7. This makes diagnostics from newer compilers impossible and complicates the build system a bit. Greg backported this to 4.4.57 in 2017, so it should be pretty easy for you guys to cherry pick. [1] https://www.wireguard.com/build-status/ [2] https://lists.ubuntu.com/archives/kernel-team/2020-May/109542.html [3] https://git.zx2c4.com/wireguard-linux-compat/commit/?id=997bef04a20533aadab3a4dd13d05e58191102bd [4] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-4.4.y&id=4cb0c0b73d1c79a8ce260836b3f27650aa1c57f1 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs 2020-05-22 8:04 Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs Jason A. Donenfeld 2020-05-22 8:44 ` Jason A. Donenfeld @ 2020-05-22 12:43 ` Georg Faerber 2020-05-23 12:31 ` Georg Faerber 2020-05-23 7:07 ` Neal Gompa 2 siblings, 1 reply; 12+ messages in thread From: Georg Faerber @ 2020-05-22 12:43 UTC (permalink / raw) To: wireguard Hi, Regarding the Debian-side of things: On 20-05-22 02:04:58, Jason A. Donenfeld wrote: > I'd like to put these on build.wireguard.com to mitigate breakage. In > order to make this happen, I'll need two things: > - A URL I can scape that will give me the latest kernel versions for > relevant distros. Versions are available via [1], the 'f' param returns json output. API docs are available via [2]. > - A URL I can construct using a selected version to download a boring > kernel source tarball. > I would prefer to not involve git, if possible, and for these URLs to > point to the sources for actual kernels that are shipping as the > standard latest-kernel for each of the above releases. One way to get the source tarball, without relying on git, is using [3], specifically [4]. To construct the URL to fetch the source tarball from, one would need to concatenate: https://snapshot.debian.org/archive/debian/ + $first_seen + $path + $name An example URL would look like [5]. I'm not sure if there are API docs available. Hope this helps, cheers, Georg [1] https://api.ftp-master.debian.org/madison?package=linux&f [2] https://ftp-team.pages.debian.net/dak/epydoc/dakweb.queries.madison-module.html [3] https://snapshot.debian.org/ [4] https://snapshot.debian.org/mr/package/linux/5.6.7-1/srcfiles?fileinfo=1 [5] https://snapshot.debian.org/archive/debian/20200429T204608Z/pool/main/l/linux/linux_5.6.7.orig.tar.xz ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs 2020-05-22 12:43 ` Georg Faerber @ 2020-05-23 12:31 ` Georg Faerber 0 siblings, 0 replies; 12+ messages in thread From: Georg Faerber @ 2020-05-23 12:31 UTC (permalink / raw) To: wireguard Hi, The example URL was referring to the orig source, which is probably not of interest: On 20-05-22 12:43:44, Georg Faerber wrote: > [5] https://snapshot.debian.org/archive/debian/20200429T204608Z/pool/main/l/linux/linux_5.6.7.orig.tar.xz The Debian stuff is included in: https://snapshot.debian.org/archive/debian/20200429T204608Z/pool/main/l/linux/linux_5.6.7-1.debian.tar.xz Cheers, Georg ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs 2020-05-22 8:04 Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs Jason A. Donenfeld 2020-05-22 8:44 ` Jason A. Donenfeld 2020-05-22 12:43 ` Georg Faerber @ 2020-05-23 7:07 ` Neal Gompa 2020-05-23 7:21 ` Jason A. Donenfeld 2 siblings, 1 reply; 12+ messages in thread From: Neal Gompa @ 2020-05-23 7:07 UTC (permalink / raw) To: Jason A. Donenfeld Cc: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team, Martin Hauke, Joe Doss, WireGuard mailing list, Carl George On Fri, May 22, 2020 at 4:05 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > Hi distro kernel maintainers, > > I currently have build.wireguard.com churning away at every new kernel > release between 3.10 and 5.5 for each and every wireguard-linux-compat > commit, making sure that we don't break anything. It's a pretty CPU > intensive, but it means we don't break mainline kernels ever when > releasing. Scroll down to the bottom to see what I mean. > > This is not the case, however, with distro kernels, which differ > notoriously from mainline in surprising ways. There's been some > breakage, and unless we do something about it, I imagine there will > continue to be much breakage. Because so many users use Linux via your > kernels, it seems imperative that we support these well. And with > distros like Ubuntu now supporting WireGuard directly through > wireguard-linux-compat, it seems ever more important that we minimize > breakage so as to not create release delays downstream of us. > > At the moment, wireguard-linux-compat supports: > - Debian oldoldstable (8), oldstable (9), stable (10), testing (11), sid > - Ubuntu 14.04, 16.04, 18.04, 19.04, 19.10, 20.04 > - OpenSUSE 42, 15, 15.2 (leap) > - RHEL 7.8, 8.2 > - CentOS 8.1 > > The logic here is that we support the latest single kernel for each > major supported release of these distros. For example, on Debian > stable, we support 4.19.0-9 but not 4.19.0-8 anymore. > > I'd like to put these on build.wireguard.com to mitigate breakage. In > order to make this happen, I'll need two things: > - A URL I can scape that will give me the latest kernel versions for > relevant distros. > - A URL I can construct using a selected version to download a boring > kernel source tarball. > I would prefer to not involve git, if possible, and for these URLs to > point to the sources for actual kernels that are shipping as the > standard latest-kernel for each of the above releases. > > If you can provide those pieces of information for my CI scripts, it > seems likely that we can drastically reduce breakage for your > distribution frankenkernels. If you cannot provide that, but other > distros can, I will probably naturally prioritize support for those > other distros that make maintenance easier, simply as a matter of > habit rather than something intentional. > For CentOS kernels, you can query the kernel builds that have been made for CentOS 7 and CentOS 8 by requesting the list of tags from the kernel package[1]. The Pagure API[2] provides a straightforward way to do this, even with curl and jq: $ curl --silent --header "Content-Type: application/json" \ https://git.centos.org/api/0/rpms/kernel/git/tags | jq '.["tags"] ' This will return a JSON list of tags. For CentOS 7, you'd want the tags beginning with "imports/c7/". For CentOS 8, you'd want the tags beginning with "imports/c8/". The string after the prefix is the NVR (name-version-release) of the package, and is the main part of the Source RPM file name. The highest NVR in each tag namespace is the latest package. There are various ways to do the comparison, but the simplest one would be to sort them with the highest number on top, as the numbers are guaranteed to numerically advance. I'm not good with jq, so I couldn't figure out how to do it as a one-liner. Sources are shipped in the form of Source RPMs and pushed to the CentOS Vault[3][4]. Sources are archived at each CentOS point release, meaning that you need to be aware of the CentOS point release where the kernel was shipped. So for example, for CentOS 7.8, you'd be looking at the following locations for source packages: * http://vault.centos.org/7.8.2003/os/Source/SPackages/ * http://vault.centos.org/7.8.2003/updates/Source/SPackages/ CentOS 8.1 is a bit simpler, with only the one location: http://vault.centos.org/8.1.1911/BaseOS/Source/SPackages/ With the information from the API and knowing the location of where the packages are, you can fetch a kernel source package like so: $ wget http://vault.centos.org/8.1.1911/BaseOS/Source/SPackages/${kernel_version_release}.src.rpm All the stuff in the source RPM is enough to rebuild the kernel. If you just want to grab the tarball inside, then you can grab it by unpacking the Source RPM and grabbing the tarball inside. [1]: https://git.centos.org/rpms/kernel [2]: https://pagure.io/api [3]: http://vault.centos.org/7.8.2003/ [4]: http://vault.centos.org/8.1.1911/ -- 真実はいつも一つ!/ Always, there's only one truth! ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs 2020-05-23 7:07 ` Neal Gompa @ 2020-05-23 7:21 ` Jason A. Donenfeld 2020-05-23 11:49 ` Neal Gompa 0 siblings, 1 reply; 12+ messages in thread From: Jason A. Donenfeld @ 2020-05-23 7:21 UTC (permalink / raw) To: Neal Gompa Cc: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team, Martin Hauke, Joe Doss, WireGuard mailing list, Carl George Hi Neal, Thanks for these very helpful instructions. In trying to implement this, one snag I encountered was that the .src.rpm file has inside of it the linux-{version}.tar.xz file that I actually want. Is there anywhere on the internet that hosts these .tar.xz files directly? Jason ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs 2020-05-23 7:21 ` Jason A. Donenfeld @ 2020-05-23 11:49 ` Neal Gompa 2020-05-24 0:01 ` Jason A. Donenfeld 0 siblings, 1 reply; 12+ messages in thread From: Neal Gompa @ 2020-05-23 11:49 UTC (permalink / raw) To: Jason A. Donenfeld Cc: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team, Martin Hauke, Joe Doss, WireGuard mailing list, Carl George On Sat, May 23, 2020 at 3:21 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > Hi Neal, > > Thanks for these very helpful instructions. In trying to implement > this, one snag I encountered was that the .src.rpm file has inside of > it the linux-{version}.tar.xz file that I actually want. Is there > anywhere on the internet that hosts these .tar.xz files directly? > There *is*, but that starts getting into using the CentOS Git system more directly. Per your original email, you weren't interested in that path, so I didn't go there. -- 真実はいつも一つ!/ Always, there's only one truth! ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs 2020-05-23 11:49 ` Neal Gompa @ 2020-05-24 0:01 ` Jason A. Donenfeld 2020-05-24 0:20 ` Neal Gompa 0 siblings, 1 reply; 12+ messages in thread From: Jason A. Donenfeld @ 2020-05-24 0:01 UTC (permalink / raw) To: Neal Gompa Cc: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team, Martin Hauke, Joe Doss, WireGuard mailing list, Carl George On Sat, May 23, 2020 at 5:50 AM Neal Gompa <ngompa13@gmail.com> wrote: > > On Sat, May 23, 2020 at 3:21 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > > > Hi Neal, > > > > Thanks for these very helpful instructions. In trying to implement > > this, one snag I encountered was that the .src.rpm file has inside of > > it the linux-{version}.tar.xz file that I actually want. Is there > > anywhere on the internet that hosts these .tar.xz files directly? > > > > There *is*, but that starts getting into using the CentOS Git system > more directly. Per your original email, you weren't interested in that > path, so I didn't go there. Well, one trick is that most web git viewers (such as cgit) can produce a tarball on the fly from a URL. Something like this exist here? ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs 2020-05-24 0:01 ` Jason A. Donenfeld @ 2020-05-24 0:20 ` Neal Gompa 2020-05-24 1:10 ` Jason A. Donenfeld 0 siblings, 1 reply; 12+ messages in thread From: Neal Gompa @ 2020-05-24 0:20 UTC (permalink / raw) To: Jason A. Donenfeld Cc: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team, Martin Hauke, Joe Doss, WireGuard mailing list, Carl George On Sat, May 23, 2020 at 8:01 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > On Sat, May 23, 2020 at 5:50 AM Neal Gompa <ngompa13@gmail.com> wrote: > > > > On Sat, May 23, 2020 at 3:21 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > > > > > Hi Neal, > > > > > > Thanks for these very helpful instructions. In trying to implement > > > this, one snag I encountered was that the .src.rpm file has inside of > > > it the linux-{version}.tar.xz file that I actually want. Is there > > > anywhere on the internet that hosts these .tar.xz files directly? > > > > > > > There *is*, but that starts getting into using the CentOS Git system > > more directly. Per your original email, you weren't interested in that > > path, so I didn't go there. > > Well, one trick is that most web git viewers (such as cgit) can > produce a tarball on the fly from a URL. Something like this exist > here? So, CentOS uses a Dist-Git[1] system, which means the part that is actually tracked in git is the packaging files, and there's an associated binary repo reference stored in the Git repo for fetching binary artifacts. Dist-Git is the common ancestor to virtually all current approaches to storing binaries efficiently alongside a Git repo (Git-Annex and Git LFS came much later). It seems it's possible to do this without having to use Git itself (which is better than with git-annex and git-lfs), but it's a bit cumbersome... The way that you'd fetch the tarball in this scenario would be to something like the following: 1. Get the tags via API, but with commits: $ curl --silent --header "Content-Type: application/json" \ https://git.centos.org/api/0/rpms/kernel/git/tags?with_commits=true \ | jq '.["tags"] ' This gives you now a JSON dictionary with the tags as keys and the associated commit as values. 2. Get the corresponding commit and fetch the .kernel.metadata file containing the lookaside references: $ curl --silent https://git.centos.org/rpms/kernel/raw/${kernel_tag_commit}/f/.kernel.metadata \ > kernel_sources 3. Fetch the source matching to your kernel source package: $ wget https://git.centos.org/sources/kernel/${rel}/${kernel_src_checksum} \ --output-document=${kernel_src_filename} The $kernel_src_checksum and $kernel_src_filename would be populated from the entries in the kernel_sources files you downloaded earlier. Note: $rel is either "c7" or "c8" depending on which one you're trying to get. [1]: https://github.com/release-engineering/dist-git -- 真実はいつも一つ!/ Always, there's only one truth! ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs 2020-05-24 0:20 ` Neal Gompa @ 2020-05-24 1:10 ` Jason A. Donenfeld 2020-05-24 1:41 ` Jason A. Donenfeld 0 siblings, 1 reply; 12+ messages in thread From: Jason A. Donenfeld @ 2020-05-24 1:10 UTC (permalink / raw) To: Neal Gompa Cc: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team, Martin Hauke, Joe Doss, WireGuard mailing list, Carl George On Sat, May 23, 2020 at 6:21 PM Neal Gompa <ngompa13@gmail.com> wrote: > > On Sat, May 23, 2020 at 8:01 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > > > On Sat, May 23, 2020 at 5:50 AM Neal Gompa <ngompa13@gmail.com> wrote: > > > > > > On Sat, May 23, 2020 at 3:21 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > > > > > > > Hi Neal, > > > > > > > > Thanks for these very helpful instructions. In trying to implement > > > > this, one snag I encountered was that the .src.rpm file has inside of > > > > it the linux-{version}.tar.xz file that I actually want. Is there > > > > anywhere on the internet that hosts these .tar.xz files directly? > > > > > > > > > > There *is*, but that starts getting into using the CentOS Git system > > > more directly. Per your original email, you weren't interested in that > > > path, so I didn't go there. > > > > Well, one trick is that most web git viewers (such as cgit) can > > produce a tarball on the fly from a URL. Something like this exist > > here? > > So, CentOS uses a Dist-Git[1] system, which means the part that is > actually tracked in git is the packaging files, and there's an > associated binary repo reference stored in the Git repo for fetching > binary artifacts. Dist-Git is the common ancestor to virtually all > current approaches to storing binaries efficiently alongside a Git > repo (Git-Annex and Git LFS came much later). > > It seems it's possible to do this without having to use Git itself > (which is better than with git-annex and git-lfs), but it's a bit > cumbersome... > > The way that you'd fetch the tarball in this scenario would be to > something like the following: > > 1. Get the tags via API, but with commits: > > $ curl --silent --header "Content-Type: application/json" \ > https://git.centos.org/api/0/rpms/kernel/git/tags?with_commits=true \ > | jq '.["tags"] ' > > This gives you now a JSON dictionary with the tags as keys and the > associated commit as values. > > 2. Get the corresponding commit and fetch the .kernel.metadata file > containing the lookaside references: > > $ curl --silent > https://git.centos.org/rpms/kernel/raw/${kernel_tag_commit}/f/.kernel.metadata \ > > kernel_sources > > 3. Fetch the source matching to your kernel source package: > > $ wget https://git.centos.org/sources/kernel/${rel}/${kernel_src_checksum} \ > --output-document=${kernel_src_filename} > > The $kernel_src_checksum and $kernel_src_filename would be populated > from the entries in the kernel_sources files you downloaded earlier. > > Note: $rel is either "c7" or "c8" depending on which one you're trying to get. > > > [1]: https://github.com/release-engineering/dist-git Perfect, thanks. CentOS kernels are now on https://www.wireguard.com/build-status/ , though el7 is having build issues. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs 2020-05-24 1:10 ` Jason A. Donenfeld @ 2020-05-24 1:41 ` Jason A. Donenfeld 0 siblings, 0 replies; 12+ messages in thread From: Jason A. Donenfeld @ 2020-05-24 1:41 UTC (permalink / raw) To: Neal Gompa Cc: unit193, Daniel Kahn Gillmor, Andy Whitcroft, Ubuntu Kernel Team, Martin Hauke, Joe Doss, WireGuard mailing list, Carl George On Sat, May 23, 2020 at 7:10 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > On Sat, May 23, 2020 at 6:21 PM Neal Gompa <ngompa13@gmail.com> wrote: > > > > On Sat, May 23, 2020 at 8:01 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > > > > > On Sat, May 23, 2020 at 5:50 AM Neal Gompa <ngompa13@gmail.com> wrote: > > > > > > > > On Sat, May 23, 2020 at 3:21 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > > > > > > > > > Hi Neal, > > > > > > > > > > Thanks for these very helpful instructions. In trying to implement > > > > > this, one snag I encountered was that the .src.rpm file has inside of > > > > > it the linux-{version}.tar.xz file that I actually want. Is there > > > > > anywhere on the internet that hosts these .tar.xz files directly? > > > > > > > > > > > > > There *is*, but that starts getting into using the CentOS Git system > > > > more directly. Per your original email, you weren't interested in that > > > > path, so I didn't go there. > > > > > > Well, one trick is that most web git viewers (such as cgit) can > > > produce a tarball on the fly from a URL. Something like this exist > > > here? > > > > So, CentOS uses a Dist-Git[1] system, which means the part that is > > actually tracked in git is the packaging files, and there's an > > associated binary repo reference stored in the Git repo for fetching > > binary artifacts. Dist-Git is the common ancestor to virtually all > > current approaches to storing binaries efficiently alongside a Git > > repo (Git-Annex and Git LFS came much later). > > > > It seems it's possible to do this without having to use Git itself > > (which is better than with git-annex and git-lfs), but it's a bit > > cumbersome... > > > > The way that you'd fetch the tarball in this scenario would be to > > something like the following: > > > > 1. Get the tags via API, but with commits: > > > > $ curl --silent --header "Content-Type: application/json" \ > > https://git.centos.org/api/0/rpms/kernel/git/tags?with_commits=true \ > > | jq '.["tags"] ' > > > > This gives you now a JSON dictionary with the tags as keys and the > > associated commit as values. > > > > 2. Get the corresponding commit and fetch the .kernel.metadata file > > containing the lookaside references: > > > > $ curl --silent > > https://git.centos.org/rpms/kernel/raw/${kernel_tag_commit}/f/.kernel.metadata \ > > > kernel_sources > > > > 3. Fetch the source matching to your kernel source package: > > > > $ wget https://git.centos.org/sources/kernel/${rel}/${kernel_src_checksum} \ > > --output-document=${kernel_src_filename} > > > > The $kernel_src_checksum and $kernel_src_filename would be populated > > from the entries in the kernel_sources files you downloaded earlier. > > > > Note: $rel is either "c7" or "c8" depending on which one you're trying to get. > > > > > > [1]: https://github.com/release-engineering/dist-git > > Perfect, thanks. CentOS kernels are now on > https://www.wireguard.com/build-status/ , though el7 is having build > issues. c7 issue: https://bugzilla.redhat.com/show_bug.cgi?id=1839419 c8 issue: https://bugzilla.redhat.com/show_bug.cgi?id=1839415 c8s issue: https://bugzilla.redhat.com/show_bug.cgi?id=1839421 I've hacked around the c8 issue in the build system, but the c7 and c8s issues will need to be fixed by Red Hat. ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2020-05-24 2:15 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-05-22 8:04 Adding Debian, Ubuntu, OpenSUSE, RHEL, CentOS kernels to WireGuard CI: Seeking URLs Jason A. Donenfeld 2020-05-22 8:44 ` Jason A. Donenfeld 2020-05-23 5:59 ` Jason A. Donenfeld 2020-05-22 12:43 ` Georg Faerber 2020-05-23 12:31 ` Georg Faerber 2020-05-23 7:07 ` Neal Gompa 2020-05-23 7:21 ` Jason A. Donenfeld 2020-05-23 11:49 ` Neal Gompa 2020-05-24 0:01 ` Jason A. Donenfeld 2020-05-24 0:20 ` Neal Gompa 2020-05-24 1:10 ` Jason A. Donenfeld 2020-05-24 1:41 ` Jason A. Donenfeld
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).