Github messages for voidlinux
 help / color / mirror / Atom feed
* [PR PATCH] RFC: Check for reproducible builds.
@ 2021-04-30  8:18 Gottox
  2021-04-30 12:53 ` [PR REVIEW] " ericonr
                   ` (10 more replies)
  0 siblings, 11 replies; 12+ messages in thread
From: Gottox @ 2021-04-30  8:18 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 1472 bytes --]

There is a new pull request by Gottox against master on the void-packages repository

https://github.com/Gottox/void-packages repro-check
https://github.com/void-linux/void-packages/pull/30588

RFC: Check for reproducible builds.
### Introduction

In void-packages the packages are anything but reproducible. Many other distributions, first and formost [NixOS](https://nixos.org/) and even [Debian](https://wiki.debian.org/ReproducibleBuilds) already did a lot of work to generate packages with stable checksums. Void's build system is able to do something similiar - with a few constraints - without much work.

### This is a starting point, not more.

As a first step to actually get an idea how bad the situation is I implemented a simple checker that compares the checksum of packages defined in templates to the actual result and spits out warnings they don't match.

This also introduces new variables to the templates:

`pkg_checksum_<arch>`, where <arch> is a sanitized version of the resulting architecture (`x86_64_musl` for `x86_64-musl` for example)

### Constraints:

* the packages are currently build with the githash backed in. This is an issue as the build is only stable within a certain commit.

### ToDo

* The documentation is currently not done, but will be added later.
* Find a way to make our package format reproducable across commits.

A patch file from https://github.com/void-linux/void-packages/pull/30588.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-repro-check-30588.patch --]
[-- Type: text/x-diff, Size: 3481 bytes --]

From 997b6505d79698cb042bf549c1faa8e31c0a5158 Mon Sep 17 00:00:00 2001
From: Enno Boland <gottox@voidlinux.org>
Date: Fri, 30 Apr 2021 10:02:58 +0200
Subject: [PATCH] common/hooks: add hook to check for resulting package
 checksum missmatches

---
 common/hooks/post-pkg/01-check-reproduce.sh | 43 +++++++++++++++++++++
 etc/defaults.conf                           |  6 +++
 xbps-src                                    |  3 +-
 3 files changed, 51 insertions(+), 1 deletion(-)
 create mode 100644 common/hooks/post-pkg/01-check-reproduce.sh

diff --git a/common/hooks/post-pkg/01-check-reproduce.sh b/common/hooks/post-pkg/01-check-reproduce.sh
new file mode 100644
index 000000000000..f56abb1b11df
--- /dev/null
+++ b/common/hooks/post-pkg/01-check-reproduce.sh
@@ -0,0 +1,43 @@
+# This hook compares the checksum of the package with the saved value
+
+hook() {
+	local arch= binpkg= checksum_ptr= checksum_have= checksum_want=
+
+	if [ -z "$XBPS_CHECK_REPRODUCIBLE" ]; then
+		return 0;
+	fi
+
+	if [ -z "$XBPS_USE_BUILD_MTIME" ]; then
+		msg_warn "reproducability check will only report correct results when\n"
+		msg_warn "XBPS_USE_BUILD_MTIME is enabled.\n"
+	fi
+
+	if [ -z "$XBPS_CROSS_BUILD" -a -n "$XBPS_ARCH" -a "$XBPS_ARCH" != "$XBPS_TARGET_MACHINE" ]; then
+		arch=${XBPS_ARCH}
+	elif [ -n "$XBPS_TARGET_MACHINE" ]; then
+		arch=$XBPS_TARGET_MACHINE
+	else
+		arch=$XBPS_MACHINE
+	fi
+	binpkg=${pkgver}.${arch}.xbps
+
+	checksum_ptr="pkg_checksum_${arch//-/_}"
+	checksum_want=${!checksum_ptr}
+
+	checksum_have=$(sha256sum "$binpkg" | awk '{ print $1 }')
+
+	if [ -z "${checksum_want}" ]; then
+		msg_normal "$pkgver: template does not define a pkg_checksum\n"
+		msg_normal "$pkgver: if the build is reproducable define the package checksum in the template:\n"
+		msg_normal "$pkgver: $checksum_ptr="$checksum_want"\n"
+		return 0
+	fi
+
+	if [ "${checksum_have}" != "${checksum_want}" ]; then
+		msg_warn "${pkgver}: Checksum mismatch. reproducable build seems to be broken.\n"
+		msg_warn "${pkgver}: Gather relevant system info:\n"
+		msg_normal "CPU: $(grep "^model name" /proc/cpuinfo | head -n 1 | sed 's/.*: //')"
+	else
+		msg_normal "${pkgver}: Checksums patch; build seems to be reproducable.\n"
+	fi
+}
diff --git a/etc/defaults.conf b/etc/defaults.conf
index 6147954a18af..55b9568c812a 100644
--- a/etc/defaults.conf
+++ b/etc/defaults.conf
@@ -130,6 +130,12 @@ XBPS_SUCMD="sudo /bin/sh -c"
 #XBPS_CHROOT_CMD=uchroot
 #XBPS_CHROOT_CMD_ARGS=""
 
+# [OPTIONAL]
+# If enabled, xbps-src will check the resulting checksum of a package against
+# a defined one. This helps to detect packages that have non-deterministic builds
+#
+#XBPS_CHECK_REPRODUCIBLE=yes
+
 # [OPTIONAL]
 # Enable to use the standard mtime of files. Otherwise it will be rewritten to
 # the HEAD commit time. Requires git when disabled.
diff --git a/xbps-src b/xbps-src
index c3cd7e5db10b..7fdb2dd41b57 100755
--- a/xbps-src
+++ b/xbps-src
@@ -635,7 +635,8 @@ export XBPS_SHUTILSDIR XBPS_CROSSPFDIR XBPS_TRIGGERSDIR \
     XBPS_DESTDIR XBPS_MACHINE XBPS_TEMP_MASTERDIR XBPS_BINPKG_EXISTS \
     XBPS_LIBEXECDIR XBPS_DISTDIR XBPS_DISTFILES_MIRROR XBPS_ALLOW_RESTRICTED \
     XBPS_USE_GIT_COMMIT_DATE XBPS_PKG_COMPTYPE XBPS_REPO_COMPTYPE \
-    XBPS_BUILDHELPERDIR XBPS_USE_BUILD_MTIME XBPS_BUILD_ENVIRONMENT
+    XBPS_BUILDHELPERDIR XBPS_CHECK_REPRODUCIBLE XBPS_USE_BUILD_MTIME \
+    XBPS_BUILD_ENVIRONMENT
 
 for i in REPOSITORY DESTDIR BUILDDIR SRCDISTDIR; do
     eval val="\$XBPS_$i"

^ permalink raw reply	[flat|nested] 12+ messages in thread
end of thread, other threads:[~2021-05-21 13:49 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-30  8:18 [PR PATCH] RFC: Check for reproducible builds Gottox
2021-04-30 12:53 ` [PR REVIEW] " ericonr
2021-04-30 13:01 ` ericonr
2021-04-30 15:55 ` ericonr
2021-04-30 16:59 ` Chocimier
2021-04-30 17:18 ` ericonr
2021-04-30 17:45 ` ericonr
2021-04-30 23:56 ` Gottox
2021-05-01  0:00 ` Gottox
2021-05-01  0:57 ` ericonr
2021-05-01  1:01 ` ericonr
2021-05-21 13:49 ` [PR PATCH] [Closed]: " Gottox

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).