Github messages for voidlinux
 help / color / Atom feed
* [PR PATCH] go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4
@ 2019-11-10  0:25 voidlinux-github
  2019-11-10  0:45 ` voidlinux-github
                   ` (8 more replies)
  0 siblings, 9 replies; 10+ messages in thread
From: voidlinux-github @ 2019-11-10  0:25 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 2452 bytes --]

There is a new pull request by q66 against master on the void-packages repository

https://github.com/void-ppc/void-packages go-bootstrap
https://github.com/void-linux/void-packages/pull/16287

go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4
This adds a `go1.12-bootstrap` package newly used instead of `go1.4-bootstrap` to bootstrap the main Go compiler. The reason for this is to allow bootstrapping on platforms the ancient codebase does not support, particularly `ppc64le` and `aarch64`.

The new bootstrap package can technically run on all the targets that are in the intersection of what the official compiler supports and what Void supports.

There is a problem that the official binaries require glibc (just its dynamic linker, the libc itself and libpthread). In order to deal with that on musl, fetch a binary copy of glibc from Debian (which is reproducibly built, so it should be trustable) and patch the official binaries so that they use the newly bundled libc and dynamic linker. This allows the compiler to execute in a musl system, and build the proper target compiler (which will use the musl dynamic linker and libc, of course).

I chose the approach after evaluating all other choices, including `gcompat` and `gccgo`; in the end this was the only thing that truly and reliably worked.

The second part of this patchset switches the `go` package to use the new 1.12 bootstrap. Since 1.12, `GOCACHE` must be off when bootstrapping, so that line was removed. I also added new `broken` lines to disable the package on big endian PowerPC systems (ppc64 BE support uses incorrect ABI and requires POWER8 hardware, making it not safely distributable; 32-bit ppc was just never supported at all).

Additionally, I also added a `broken` when cross-building on the same architecture to a different libc (e.g. `x86_64` to `x86_64-musl`). This is because the build system does not account for this scenario and just does not cross build; this creates a package without failing but the contained compiler is just a compiler for the host system and will not function on the target. So at least until this problem is taken care of, disable it in order to not silently build broken packages.

If this is accepted, we can drop the `go1.4-bootstrap` package afterwards.

@the-maldridge @nilium 

A patch file from https://github.com/void-linux/void-packages/pull/16287.patch is attached

[-- Warning: decoded text below may be mangled --]
[-- Attachment #2: github-pr-go-bootstrap-16287.patch --]
[-- Type: text/x-diff, Size: 6225 bytes --]

From 583eb4f8b4d58cacb84f2a16bcd48e88aabb3f90 Mon Sep 17 00:00:00 2001
From: q66 <daniel@octaforge.org>
Date: Sun, 10 Mar 2019 21:53:45 +0100
Subject: [PATCH 1/2] New package: go1.12-bootstrap-1.12.13

---
 srcpkgs/go1.12-bootstrap/INSTALL.msg |   4 +
 srcpkgs/go1.12-bootstrap/template    | 107 +++++++++++++++++++++++++++
 2 files changed, 111 insertions(+)
 create mode 100644 srcpkgs/go1.12-bootstrap/INSTALL.msg
 create mode 100644 srcpkgs/go1.12-bootstrap/template

diff --git a/srcpkgs/go1.12-bootstrap/INSTALL.msg b/srcpkgs/go1.12-bootstrap/INSTALL.msg
new file mode 100644
index 00000000000..142aeabbc1b
--- /dev/null
+++ b/srcpkgs/go1.12-bootstrap/INSTALL.msg
@@ -0,0 +1,4 @@
+This is a copy of the official Go language toolchain binaries as provided by
+the project on its download page. Please do keep in mind that it is almost
+definitely not what you want to use and exists purely for the purpose of
+bootstrapping the official compiler package (called simply 'go').
diff --git a/srcpkgs/go1.12-bootstrap/template b/srcpkgs/go1.12-bootstrap/template
new file mode 100644
index 00000000000..2324c6529bf
--- /dev/null
+++ b/srcpkgs/go1.12-bootstrap/template
@@ -0,0 +1,107 @@
+# Template file for 'go1.12-bootstrap'
+pkgname=go1.12-bootstrap
+version=1.12.13
+revision=1
+archs="x86_64* i686* armv[67]l* aarch64* ppc64le*"
+wrksrc="go"
+short_desc="Go 1.12 (bootstrap compiler)"
+maintainer="q66 <daniel@octaforge.org>"
+license="BSD-3-Clause"
+homepage="https://golang.org"
+nostrip=yes
+noverifyrdeps=yes
+nocross=yes
+
+case "$XBPS_TARGET_MACHINE" in
+	x86_64*)
+		_dist_arch="amd64"
+		_deb_arch="amd64"
+		checksum="
+		 da036454cb3353f9f507f0ceed4048feac611065e4e1818b434365eb32ac9bdc
+		 48bc07f034d4681cd3a758809e5a192b9ad2b65a6455e83363ce891fb47a6b73"
+		;;
+	i686*)
+		_dist_arch="386"
+		_deb_arch="i386"
+		checksum="
+		 fafcb585591557b7b16d9b22dec4654193d205cf444b1810ab2988f658585e23
+		 87064f73f9f4e24efca95ebc95531caad3d016f646dd07c42808e5ece55f0ec7"
+		;;
+	arm*)
+		_dist_arch="armv6l"
+		# we're hardfloat on armv6l but debian armhf is armv7l
+		_deb_arch="armel"
+		checksum="
+		 bf061cc3d4951e07904496b5c3d6c82419309d24634835522d786673a3f5438f
+		 2c9633e5030216be7d3d0bb83e3dbb8882541c57afa098a54fdc0162823e112b"
+		;;
+	aarch64*)
+		_dist_arch="arm64"
+		_deb_arch="arm64"
+		checksum="
+		 dcfcb3785292c98f7a75c2276169dfe2d445c19f8ffe1d40b3f7b8f59712d361
+		 b3dd4d3c2523f9a07d3bde9d35ef7c6c30bcd32af31abbc6ce085888e1f8dd57"
+		;;
+	ppc64le*)
+		_dist_arch="ppc64le"
+		_deb_arch="ppc64el"
+		checksum="
+		 77056264abcf5444ed0d9ab7552552ae2145ca8fb6c39d33db3c735eaf3f42d2
+		 05d63c62e413ae638732242cba5d57c92c40d66d5c7ffe23e8adf8ec168fac34"
+		;;
+esac
+
+# the official binaries of go use the dynamic linker and libc from glibc,
+# so on musl hosts, fetch a binary distribution of glibc from a trustable
+# source (Debian) and use patchelf to alter the dynamic linker path as
+# well as add rpath to make the go compiler use the bundled libs
+if [ "$XBPS_TARGET_LIBC" = "musl" ]; then
+	_libc_ver=2.29
+	_libc_rev=3
+	_libc_deb="libc6_${_libc_ver}-${_libc_rev}_${_deb_arch}.deb"
+	hostmakedepends+=" patchelf"
+fi
+
+distfiles="https://dl.google.com/go/go${version}.linux-${_dist_arch}.tar.gz"
+if [ -n "$_libc_ver" ]; then
+	distfiles+=" http://ftp.debian.org/debian/pool/main/g/glibc/${_libc_deb}"
+	skip_extraction+=" ${_libc_deb}"
+fi
+
+post_extract() {
+	[ -z "$_libc_ver" ] && return 0
+
+	mkdir ${wrksrc}/libc6
+	pushd ${wrksrc}/libc6
+	ar x ${XBPS_SRCDISTDIR}/${pkgname}-${version}/${_libc_deb}
+	tar xf data.tar.xz
+	mv lib/*-linux-*/ld-${_libc_ver}.so ${wrksrc}/bin
+	mv lib/*-linux-*/libc-${_libc_ver}.so ${wrksrc}/bin
+	mv lib/*-linux-*/libpthread-${_libc_ver}.so ${wrksrc}/bin
+	popd
+}
+
+post_build() {
+	[ -z "$_libc_ver" ] && return 0
+
+	local _pelf_path="/usr/lib/go1.12/bin" _pelf_args=
+	_pelf_args+=" --set-interpreter ${_pelf_path}/ld-${_libc_ver}.so"
+	_pelf_args+=" --set-rpath ${_pelf_path}"
+
+	pushd ${wrksrc}/bin
+	patchelf ${_pelf_args} libc-${_libc_ver}.so
+	patchelf ${_pelf_args} libpthread-${_libc_ver}.so
+	patchelf ${_pelf_args} go
+	patchelf ${_pelf_args} godoc
+	ln -sf libc-${_libc_ver}.so libc.so.6
+	ln -sf libpthread-${_libc_ver}.so libpthread.so.0
+	popd
+}
+
+do_install() {
+	vmkdir usr/lib/go1.12
+	vcopy bin usr/lib/go1.12
+	vcopy src usr/lib/go1.12
+	vcopy pkg usr/lib/go1.12
+	vlicense LICENSE
+}

From 0e5c17b5703d34d45a52f5937e14a1e22c5f3716 Mon Sep 17 00:00:00 2001
From: q66 <daniel@octaforge.org>
Date: Sat, 15 Dec 2018 04:11:57 +0100
Subject: [PATCH 2/2] go: use go1.12-bootstrap, disable ppc* BE, disable
 cross-libc

---
 srcpkgs/go/template | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/srcpkgs/go/template b/srcpkgs/go/template
index c263afc1d38..0349e5e3da2 100644
--- a/srcpkgs/go/template
+++ b/srcpkgs/go/template
@@ -4,14 +4,13 @@ version=1.13.4
 revision=1
 create_wrksrc=yes
 build_wrksrc=go
-hostmakedepends="go1.4-bootstrap"
+hostmakedepends="go1.12-bootstrap"
 short_desc="Go Programming Language"
 maintainer="Michael Aldridge <maldridge@voidlinux.org>"
 license="BSD-3-Clause"
 homepage="http://golang.org/"
 distfiles="https://golang.org/dl/go${version}.src.tar.gz"
 checksum=95dbeab442ee2746b9acf0934c8e2fc26414a0565c008631b04addb8c02e7624
-
 nostrip=yes
 noverifyrdeps=yes
 
@@ -21,20 +20,27 @@ case "${XBPS_TARGET_MACHINE}" in
 	mips*) _goarch=mips ;;
 	i686*) _goarch=386 ;;
 	x86_64*) _goarch=amd64 ;;
+	ppc64le*) _goarch=ppc64le ;;
+	ppc64*) broken="Upstream does not support ELFv2 for big endian ppc64";;
+	ppc*) broken="Upstream does not support 32-bit ppc";;
 	*) _goarch=${XBPS_TARGET_MACHINE} ;;
 esac
 
+if [ "$CROSS_BUILD" ]; then
+	if [ "${XBPS_MACHINE%%-musl}" = "${XBPS_TARGET_MACHINE%%-musl}" ]; then
+		broken="Cross-compiling to different libc is not supported"
+	fi
+fi
+
 do_build() {
 	unset GCC CC CXX LD CFLAGS
 	# FIXME: work around leaking go build-style vars when built as a
 	# dependency
 	unset CGO_CXXFLAGS CGO_CFLAGS CGO_ENABLED
 
-
-	export GOCACHE=off
+	export GOROOT_BOOTSTRAP="/usr/lib/go1.12"
 	export GOROOT=$PWD
 	export GOROOT_FINAL="/usr/lib/go"
-	export GOROOT_BOOTSTRAP="/usr/lib/go1.4"
 	export GOARCH=${_goarch}
 
 	cd "src"

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4
  2019-11-10  0:25 [PR PATCH] go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4 voidlinux-github
@ 2019-11-10  0:45 ` voidlinux-github
  2019-11-10  1:58 ` voidlinux-github
                   ` (7 subsequent siblings)
  8 siblings, 0 replies; 10+ messages in thread
From: voidlinux-github @ 2019-11-10  0:45 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 267 bytes --]

New comment by q66 on void-packages repository

https://github.com/void-linux/void-packages/pull/16287#issuecomment-552151212

Comment:
Looks like something is wrong on the musl hosts; I will have to look into that - also the `i686` fails in a strange way (on glibc)

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4
  2019-11-10  0:25 [PR PATCH] go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4 voidlinux-github
  2019-11-10  0:45 ` voidlinux-github
@ 2019-11-10  1:58 ` voidlinux-github
  2019-11-10 16:55 ` [PR PATCH] [Updated] " voidlinux-github
                   ` (6 subsequent siblings)
  8 siblings, 0 replies; 10+ messages in thread
From: voidlinux-github @ 2019-11-10  1:58 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 380 bytes --]

New comment by q66 on void-packages repository

https://github.com/void-linux/void-packages/pull/16287#issuecomment-552154805

Comment:
I think I might be able to ditch the glibc dependency entirely, by just patching stuff to use musl's dynlinker and libraries. This works on x86_64 but it's failing on ppc64le exactly the same as if trying to use gcompat. Gonna investigate more

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PR PATCH] [Updated] go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4
  2019-11-10  0:25 [PR PATCH] go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4 voidlinux-github
  2019-11-10  0:45 ` voidlinux-github
  2019-11-10  1:58 ` voidlinux-github
@ 2019-11-10 16:55 ` " voidlinux-github
  2019-11-10 16:57 ` voidlinux-github
                   ` (5 subsequent siblings)
  8 siblings, 0 replies; 10+ messages in thread
From: voidlinux-github @ 2019-11-10 16:55 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 2461 bytes --]

There is an updated pull request by q66 against master on the void-packages repository

https://github.com/void-ppc/void-packages go-bootstrap
https://github.com/void-linux/void-packages/pull/16287

go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4
This adds a `go1.12-bootstrap` package newly used instead of `go1.4-bootstrap` to bootstrap the main Go compiler. The reason for this is to allow bootstrapping on platforms the ancient codebase does not support, particularly `ppc64le` and `aarch64`.

The new bootstrap package can technically run on all the targets that are in the intersection of what the official compiler supports and what Void supports.

There is a problem that the official binaries require glibc (just its dynamic linker, the libc itself and libpthread). In order to deal with that on musl, fetch a binary copy of glibc from Debian (which is reproducibly built, so it should be trustable) and patch the official binaries so that they use the newly bundled libc and dynamic linker. This allows the compiler to execute in a musl system, and build the proper target compiler (which will use the musl dynamic linker and libc, of course).

I chose the approach after evaluating all other choices, including `gcompat` and `gccgo`; in the end this was the only thing that truly and reliably worked.

The second part of this patchset switches the `go` package to use the new 1.12 bootstrap. Since 1.12, `GOCACHE` must be not set when bootstrapping, so that line was removed. I also added new `broken` lines to disable the package on big endian PowerPC systems (ppc64 BE support uses incorrect ABI and requires POWER8 hardware, making it not safely distributable; 32-bit ppc was just never supported at all).

Additionally, I also added a `broken` when cross-building on the same architecture to a different libc (e.g. `x86_64` to `x86_64-musl`). This is because the build system does not account for this scenario and just does not cross build; this creates a package without failing but the contained compiler is just a compiler for the host system and will not function on the target. So at least until this problem is taken care of, disable it in order to not silently build broken packages.

If this is accepted, we can drop the `go1.4-bootstrap` package afterwards.

@the-maldridge @nilium 

A patch file from https://github.com/void-linux/void-packages/pull/16287.patch is attached

[-- Warning: decoded text below may be mangled --]
[-- Attachment #2: github-pr-go-bootstrap-16287.patch --]
[-- Type: text/x-diff, Size: 4751 bytes --]

From 32e2621c736a217ad6234c1558cc93385d3b9bc4 Mon Sep 17 00:00:00 2001
From: q66 <daniel@octaforge.org>
Date: Sun, 10 Mar 2019 21:53:45 +0100
Subject: [PATCH 1/2] New package: go1.12-bootstrap-1.12.13

---
 srcpkgs/go1.12-bootstrap/INSTALL.msg |  4 ++
 srcpkgs/go1.12-bootstrap/template    | 63 ++++++++++++++++++++++++++++
 2 files changed, 67 insertions(+)
 create mode 100644 srcpkgs/go1.12-bootstrap/INSTALL.msg
 create mode 100644 srcpkgs/go1.12-bootstrap/template

diff --git a/srcpkgs/go1.12-bootstrap/INSTALL.msg b/srcpkgs/go1.12-bootstrap/INSTALL.msg
new file mode 100644
index 00000000000..142aeabbc1b
--- /dev/null
+++ b/srcpkgs/go1.12-bootstrap/INSTALL.msg
@@ -0,0 +1,4 @@
+This is a copy of the official Go language toolchain binaries as provided by
+the project on its download page. Please do keep in mind that it is almost
+definitely not what you want to use and exists purely for the purpose of
+bootstrapping the official compiler package (called simply 'go').
diff --git a/srcpkgs/go1.12-bootstrap/template b/srcpkgs/go1.12-bootstrap/template
new file mode 100644
index 00000000000..24e5df3aa6b
--- /dev/null
+++ b/srcpkgs/go1.12-bootstrap/template
@@ -0,0 +1,63 @@
+# Template file for 'go1.12-bootstrap'
+pkgname=go1.12-bootstrap
+version=1.12.13
+revision=1
+archs="x86_64* i686* armv[67]l* aarch64* ppc64le*"
+wrksrc="go"
+short_desc="Go 1.12 (bootstrap compiler)"
+maintainer="q66 <daniel@octaforge.org>"
+license="BSD-3-Clause"
+homepage="https://golang.org"
+nostrip=yes
+noverifyrdeps=yes
+nocross=yes
+
+if [ "$XBPS_TARGET_LIBC" = "musl" ]; then
+	depends+=" gcompat"
+	hostmakedepends+=" patchelf"
+fi
+
+case "$XBPS_TARGET_MACHINE" in
+	x86_64*)
+		_dist_arch="amd64"
+		checksum="da036454cb3353f9f507f0ceed4048feac611065e4e1818b434365eb32ac9bdc"
+		;;
+	i686*)
+		_dist_arch="386"
+		checksum="fafcb585591557b7b16d9b22dec4654193d205cf444b1810ab2988f658585e23"
+		;;
+	arm*)
+		_dist_arch="armv6l"
+		checksum="bf061cc3d4951e07904496b5c3d6c82419309d24634835522d786673a3f5438f"
+		;;
+	aarch64*)
+		_dist_arch="arm64"
+		checksum="dcfcb3785292c98f7a75c2276169dfe2d445c19f8ffe1d40b3f7b8f59712d361"
+		;;
+	ppc64le*)
+		_dist_arch="ppc64le"
+		checksum="77056264abcf5444ed0d9ab7552552ae2145ca8fb6c39d33db3c735eaf3f42d2"
+		;;
+esac
+
+distfiles="https://dl.google.com/go/go${version}.linux-${_dist_arch}.tar.gz"
+
+post_build() {
+	[ "$XBPS_TARGET_LIBC" != "musl" ] && return 0
+
+	# we don't have lib64 compatibility path on musl 64-bit systems
+	# use patchelf to replace /lib64/<dynlinker> with /lib/<dynlinker>
+
+	local _interp=$(patchelf --print-interpreter ${wrksrc}/bin/go)
+
+	patchelf --set-interpreter ${_interp/lib64\//lib\/} ${wrksrc}/bin/go
+	patchelf --set-interpreter ${_interp/lib64\//lib\/} ${wrksrc}/bin/godoc
+}
+
+do_install() {
+	vmkdir usr/lib/go1.12
+	vcopy bin usr/lib/go1.12
+	vcopy src usr/lib/go1.12
+	vcopy pkg usr/lib/go1.12
+	vlicense LICENSE
+}

From 6bf94162c938565b29b415d4722338ec62f6ab6c Mon Sep 17 00:00:00 2001
From: q66 <daniel@octaforge.org>
Date: Sat, 15 Dec 2018 04:11:57 +0100
Subject: [PATCH 2/2] go: use go1.12-bootstrap, disable ppc* BE, disable
 cross-libc

[ci skip]
---
 srcpkgs/go/template | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/srcpkgs/go/template b/srcpkgs/go/template
index c263afc1d38..0349e5e3da2 100644
--- a/srcpkgs/go/template
+++ b/srcpkgs/go/template
@@ -4,14 +4,13 @@ version=1.13.4
 revision=1
 create_wrksrc=yes
 build_wrksrc=go
-hostmakedepends="go1.4-bootstrap"
+hostmakedepends="go1.12-bootstrap"
 short_desc="Go Programming Language"
 maintainer="Michael Aldridge <maldridge@voidlinux.org>"
 license="BSD-3-Clause"
 homepage="http://golang.org/"
 distfiles="https://golang.org/dl/go${version}.src.tar.gz"
 checksum=95dbeab442ee2746b9acf0934c8e2fc26414a0565c008631b04addb8c02e7624
-
 nostrip=yes
 noverifyrdeps=yes
 
@@ -21,20 +20,27 @@ case "${XBPS_TARGET_MACHINE}" in
 	mips*) _goarch=mips ;;
 	i686*) _goarch=386 ;;
 	x86_64*) _goarch=amd64 ;;
+	ppc64le*) _goarch=ppc64le ;;
+	ppc64*) broken="Upstream does not support ELFv2 for big endian ppc64";;
+	ppc*) broken="Upstream does not support 32-bit ppc";;
 	*) _goarch=${XBPS_TARGET_MACHINE} ;;
 esac
 
+if [ "$CROSS_BUILD" ]; then
+	if [ "${XBPS_MACHINE%%-musl}" = "${XBPS_TARGET_MACHINE%%-musl}" ]; then
+		broken="Cross-compiling to different libc is not supported"
+	fi
+fi
+
 do_build() {
 	unset GCC CC CXX LD CFLAGS
 	# FIXME: work around leaking go build-style vars when built as a
 	# dependency
 	unset CGO_CXXFLAGS CGO_CFLAGS CGO_ENABLED
 
-
-	export GOCACHE=off
+	export GOROOT_BOOTSTRAP="/usr/lib/go1.12"
 	export GOROOT=$PWD
 	export GOROOT_FINAL="/usr/lib/go"
-	export GOROOT_BOOTSTRAP="/usr/lib/go1.4"
 	export GOARCH=${_goarch}
 
 	cd "src"

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4
  2019-11-10  0:25 [PR PATCH] go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4 voidlinux-github
                   ` (2 preceding siblings ...)
  2019-11-10 16:55 ` [PR PATCH] [Updated] " voidlinux-github
@ 2019-11-10 16:57 ` voidlinux-github
  2019-11-10 23:00 ` [PR PATCH] [Updated] " voidlinux-github
                   ` (4 subsequent siblings)
  8 siblings, 0 replies; 10+ messages in thread
From: voidlinux-github @ 2019-11-10 16:57 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 373 bytes --]

New comment by q66 on void-packages repository

https://github.com/void-linux/void-packages/pull/16287#issuecomment-552212781

Comment:
Updated to use `gcompat`, which works after some updates. But CI is on hold until https://github.com/void-linux/void-packages/pull/16320 is merged, which is on hold until https://github.com/void-linux/void-packages/pull/16325 is merged.

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PR PATCH] [Updated] go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4
  2019-11-10  0:25 [PR PATCH] go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4 voidlinux-github
                   ` (3 preceding siblings ...)
  2019-11-10 16:57 ` voidlinux-github
@ 2019-11-10 23:00 ` " voidlinux-github
  2019-11-11 18:33 ` voidlinux-github
                   ` (3 subsequent siblings)
  8 siblings, 0 replies; 10+ messages in thread
From: voidlinux-github @ 2019-11-10 23:00 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 2717 bytes --]

There is an updated pull request by q66 against master on the void-packages repository

https://github.com/void-ppc/void-packages go-bootstrap
https://github.com/void-linux/void-packages/pull/16287

go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4
This adds a `go1.12-bootstrap` package newly used instead of `go1.4-bootstrap` to bootstrap the main Go compiler. The reason for this is to allow bootstrapping on platforms the ancient codebase does not support, particularly `ppc64le` and `aarch64`.

The new bootstrap package can technically run on all the targets that are in the intersection of what the official compiler supports and what Void supports.

~~There is a problem that the official binaries require glibc (just its dynamic linker, the libc itself and libpthread). In order to deal with that on musl, fetch a binary copy of glibc from Debian (which is reproducibly built, so it should be trustable) and patch the official binaries so that they use the newly bundled libc and dynamic linker. This allows the compiler to execute in a musl system, and build the proper target compiler (which will use the musl dynamic linker and libc, of course).~~

~~I chose the approach after evaluating all other choices, including `gcompat` and `gccgo`; in the end this was the only thing that truly and reliably worked.~~

I used gcompat to be able to run the compiler. In order for that to work, gcompat needs to be updated and patched, as in https://github.com/void-linux/void-packages/pull/16320. We still need to use patchelf to patch out the `lib64` requirement.

The second part of this patchset switches the `go` package to use the new 1.12 bootstrap. Since 1.12, `GOCACHE` must be not set when bootstrapping, so that line was removed. I also added new `broken` lines to disable the package on big endian PowerPC systems (ppc64 BE support uses incorrect ABI and requires POWER8 hardware, making it not safely distributable; 32-bit ppc was just never supported at all).

Additionally, I also added a `broken` when cross-building on the same architecture to a different libc (e.g. `x86_64` to `x86_64-musl`). This is because the build system does not account for this scenario and just does not cross build; this creates a package without failing but the contained compiler is just a compiler for the host system and will not function on the target. So at least until this problem is taken care of, disable it in order to not silently build broken packages.

If this is accepted, we can drop the `go1.4-bootstrap` package afterwards.

@the-maldridge @nilium 

A patch file from https://github.com/void-linux/void-packages/pull/16287.patch is attached

[-- Warning: decoded text below may be mangled --]
[-- Attachment #2: github-pr-go-bootstrap-16287.patch --]
[-- Type: text/x-diff, Size: 4770 bytes --]

From 64c7b2f28238ce5fd9e978eaee13912e8d9fe908 Mon Sep 17 00:00:00 2001
From: q66 <daniel@octaforge.org>
Date: Sun, 10 Mar 2019 21:53:45 +0100
Subject: [PATCH 1/2] New package: go1.12-bootstrap-1.12.13

---
 srcpkgs/go1.12-bootstrap/INSTALL.msg |  4 ++
 srcpkgs/go1.12-bootstrap/template    | 64 ++++++++++++++++++++++++++++
 2 files changed, 68 insertions(+)
 create mode 100644 srcpkgs/go1.12-bootstrap/INSTALL.msg
 create mode 100644 srcpkgs/go1.12-bootstrap/template

diff --git a/srcpkgs/go1.12-bootstrap/INSTALL.msg b/srcpkgs/go1.12-bootstrap/INSTALL.msg
new file mode 100644
index 00000000000..142aeabbc1b
--- /dev/null
+++ b/srcpkgs/go1.12-bootstrap/INSTALL.msg
@@ -0,0 +1,4 @@
+This is a copy of the official Go language toolchain binaries as provided by
+the project on its download page. Please do keep in mind that it is almost
+definitely not what you want to use and exists purely for the purpose of
+bootstrapping the official compiler package (called simply 'go').
diff --git a/srcpkgs/go1.12-bootstrap/template b/srcpkgs/go1.12-bootstrap/template
new file mode 100644
index 00000000000..840fcf72b56
--- /dev/null
+++ b/srcpkgs/go1.12-bootstrap/template
@@ -0,0 +1,64 @@
+# Template file for 'go1.12-bootstrap'
+pkgname=go1.12-bootstrap
+version=1.12.13
+revision=1
+archs="x86_64* i686* armv[67]l* aarch64* ppc64le*"
+wrksrc="go"
+short_desc="Go 1.12 (bootstrap compiler)"
+maintainer="q66 <daniel@octaforge.org>"
+license="BSD-3-Clause"
+homepage="https://golang.org"
+nostrip=yes
+noverifyrdeps=yes
+nocross=yes
+lib32disabled=yes
+
+if [ "$XBPS_TARGET_LIBC" = "musl" ]; then
+	depends+=" gcompat"
+	hostmakedepends+=" patchelf"
+fi
+
+case "$XBPS_TARGET_MACHINE" in
+	x86_64*)
+		_dist_arch="amd64"
+		checksum="da036454cb3353f9f507f0ceed4048feac611065e4e1818b434365eb32ac9bdc"
+		;;
+	i686*)
+		_dist_arch="386"
+		checksum="fafcb585591557b7b16d9b22dec4654193d205cf444b1810ab2988f658585e23"
+		;;
+	arm*)
+		_dist_arch="armv6l"
+		checksum="bf061cc3d4951e07904496b5c3d6c82419309d24634835522d786673a3f5438f"
+		;;
+	aarch64*)
+		_dist_arch="arm64"
+		checksum="dcfcb3785292c98f7a75c2276169dfe2d445c19f8ffe1d40b3f7b8f59712d361"
+		;;
+	ppc64le*)
+		_dist_arch="ppc64le"
+		checksum="77056264abcf5444ed0d9ab7552552ae2145ca8fb6c39d33db3c735eaf3f42d2"
+		;;
+esac
+
+distfiles="https://dl.google.com/go/go${version}.linux-${_dist_arch}.tar.gz"
+
+post_build() {
+	[ "$XBPS_TARGET_LIBC" != "musl" ] && return 0
+
+	# we don't have lib64 compatibility path on musl 64-bit systems
+	# use patchelf to replace /lib64/<dynlinker> with /lib/<dynlinker>
+
+	local _interp=$(patchelf --print-interpreter ${wrksrc}/bin/go)
+
+	patchelf --set-interpreter ${_interp/lib64\//lib\/} ${wrksrc}/bin/go
+	patchelf --set-interpreter ${_interp/lib64\//lib\/} ${wrksrc}/bin/godoc
+}
+
+do_install() {
+	vmkdir usr/lib/go1.12
+	vcopy bin usr/lib/go1.12
+	vcopy src usr/lib/go1.12
+	vcopy pkg usr/lib/go1.12
+	vlicense LICENSE
+}

From 87e2b0a3ed04ed16d9d9d0ccfca445a7441aa42a Mon Sep 17 00:00:00 2001
From: q66 <daniel@octaforge.org>
Date: Sat, 15 Dec 2018 04:11:57 +0100
Subject: [PATCH 2/2] go: use go1.12-bootstrap, disable ppc* BE, disable
 cross-libc

[ci skip]
---
 srcpkgs/go/template | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/srcpkgs/go/template b/srcpkgs/go/template
index c263afc1d38..0349e5e3da2 100644
--- a/srcpkgs/go/template
+++ b/srcpkgs/go/template
@@ -4,14 +4,13 @@ version=1.13.4
 revision=1
 create_wrksrc=yes
 build_wrksrc=go
-hostmakedepends="go1.4-bootstrap"
+hostmakedepends="go1.12-bootstrap"
 short_desc="Go Programming Language"
 maintainer="Michael Aldridge <maldridge@voidlinux.org>"
 license="BSD-3-Clause"
 homepage="http://golang.org/"
 distfiles="https://golang.org/dl/go${version}.src.tar.gz"
 checksum=95dbeab442ee2746b9acf0934c8e2fc26414a0565c008631b04addb8c02e7624
-
 nostrip=yes
 noverifyrdeps=yes
 
@@ -21,20 +20,27 @@ case "${XBPS_TARGET_MACHINE}" in
 	mips*) _goarch=mips ;;
 	i686*) _goarch=386 ;;
 	x86_64*) _goarch=amd64 ;;
+	ppc64le*) _goarch=ppc64le ;;
+	ppc64*) broken="Upstream does not support ELFv2 for big endian ppc64";;
+	ppc*) broken="Upstream does not support 32-bit ppc";;
 	*) _goarch=${XBPS_TARGET_MACHINE} ;;
 esac
 
+if [ "$CROSS_BUILD" ]; then
+	if [ "${XBPS_MACHINE%%-musl}" = "${XBPS_TARGET_MACHINE%%-musl}" ]; then
+		broken="Cross-compiling to different libc is not supported"
+	fi
+fi
+
 do_build() {
 	unset GCC CC CXX LD CFLAGS
 	# FIXME: work around leaking go build-style vars when built as a
 	# dependency
 	unset CGO_CXXFLAGS CGO_CFLAGS CGO_ENABLED
 
-
-	export GOCACHE=off
+	export GOROOT_BOOTSTRAP="/usr/lib/go1.12"
 	export GOROOT=$PWD
 	export GOROOT_FINAL="/usr/lib/go"
-	export GOROOT_BOOTSTRAP="/usr/lib/go1.4"
 	export GOARCH=${_goarch}
 
 	cd "src"

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PR PATCH] [Updated] go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4
  2019-11-10  0:25 [PR PATCH] go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4 voidlinux-github
                   ` (4 preceding siblings ...)
  2019-11-10 23:00 ` [PR PATCH] [Updated] " voidlinux-github
@ 2019-11-11 18:33 ` voidlinux-github
  2019-11-11 18:33 ` voidlinux-github
                   ` (2 subsequent siblings)
  8 siblings, 0 replies; 10+ messages in thread
From: voidlinux-github @ 2019-11-11 18:33 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 2717 bytes --]

There is an updated pull request by q66 against master on the void-packages repository

https://github.com/void-ppc/void-packages go-bootstrap
https://github.com/void-linux/void-packages/pull/16287

go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4
This adds a `go1.12-bootstrap` package newly used instead of `go1.4-bootstrap` to bootstrap the main Go compiler. The reason for this is to allow bootstrapping on platforms the ancient codebase does not support, particularly `ppc64le` and `aarch64`.

The new bootstrap package can technically run on all the targets that are in the intersection of what the official compiler supports and what Void supports.

~~There is a problem that the official binaries require glibc (just its dynamic linker, the libc itself and libpthread). In order to deal with that on musl, fetch a binary copy of glibc from Debian (which is reproducibly built, so it should be trustable) and patch the official binaries so that they use the newly bundled libc and dynamic linker. This allows the compiler to execute in a musl system, and build the proper target compiler (which will use the musl dynamic linker and libc, of course).~~

~~I chose the approach after evaluating all other choices, including `gcompat` and `gccgo`; in the end this was the only thing that truly and reliably worked.~~

I used gcompat to be able to run the compiler. In order for that to work, gcompat needs to be updated and patched, as in https://github.com/void-linux/void-packages/pull/16320. We still need to use patchelf to patch out the `lib64` requirement.

The second part of this patchset switches the `go` package to use the new 1.12 bootstrap. Since 1.12, `GOCACHE` must be not set when bootstrapping, so that line was removed. I also added new `broken` lines to disable the package on big endian PowerPC systems (ppc64 BE support uses incorrect ABI and requires POWER8 hardware, making it not safely distributable; 32-bit ppc was just never supported at all).

Additionally, I also added a `broken` when cross-building on the same architecture to a different libc (e.g. `x86_64` to `x86_64-musl`). This is because the build system does not account for this scenario and just does not cross build; this creates a package without failing but the contained compiler is just a compiler for the host system and will not function on the target. So at least until this problem is taken care of, disable it in order to not silently build broken packages.

If this is accepted, we can drop the `go1.4-bootstrap` package afterwards.

@the-maldridge @nilium 

A patch file from https://github.com/void-linux/void-packages/pull/16287.patch is attached

[-- Warning: decoded text below may be mangled --]
[-- Attachment #2: github-pr-go-bootstrap-16287.patch --]
[-- Type: text/x-diff, Size: 4760 bytes --]

From 64c7b2f28238ce5fd9e978eaee13912e8d9fe908 Mon Sep 17 00:00:00 2001
From: q66 <daniel@octaforge.org>
Date: Sun, 10 Mar 2019 21:53:45 +0100
Subject: [PATCH 1/2] New package: go1.12-bootstrap-1.12.13

---
 srcpkgs/go1.12-bootstrap/INSTALL.msg |  4 ++
 srcpkgs/go1.12-bootstrap/template    | 64 ++++++++++++++++++++++++++++
 2 files changed, 68 insertions(+)
 create mode 100644 srcpkgs/go1.12-bootstrap/INSTALL.msg
 create mode 100644 srcpkgs/go1.12-bootstrap/template

diff --git a/srcpkgs/go1.12-bootstrap/INSTALL.msg b/srcpkgs/go1.12-bootstrap/INSTALL.msg
new file mode 100644
index 00000000000..142aeabbc1b
--- /dev/null
+++ b/srcpkgs/go1.12-bootstrap/INSTALL.msg
@@ -0,0 +1,4 @@
+This is a copy of the official Go language toolchain binaries as provided by
+the project on its download page. Please do keep in mind that it is almost
+definitely not what you want to use and exists purely for the purpose of
+bootstrapping the official compiler package (called simply 'go').
diff --git a/srcpkgs/go1.12-bootstrap/template b/srcpkgs/go1.12-bootstrap/template
new file mode 100644
index 00000000000..840fcf72b56
--- /dev/null
+++ b/srcpkgs/go1.12-bootstrap/template
@@ -0,0 +1,64 @@
+# Template file for 'go1.12-bootstrap'
+pkgname=go1.12-bootstrap
+version=1.12.13
+revision=1
+archs="x86_64* i686* armv[67]l* aarch64* ppc64le*"
+wrksrc="go"
+short_desc="Go 1.12 (bootstrap compiler)"
+maintainer="q66 <daniel@octaforge.org>"
+license="BSD-3-Clause"
+homepage="https://golang.org"
+nostrip=yes
+noverifyrdeps=yes
+nocross=yes
+lib32disabled=yes
+
+if [ "$XBPS_TARGET_LIBC" = "musl" ]; then
+	depends+=" gcompat"
+	hostmakedepends+=" patchelf"
+fi
+
+case "$XBPS_TARGET_MACHINE" in
+	x86_64*)
+		_dist_arch="amd64"
+		checksum="da036454cb3353f9f507f0ceed4048feac611065e4e1818b434365eb32ac9bdc"
+		;;
+	i686*)
+		_dist_arch="386"
+		checksum="fafcb585591557b7b16d9b22dec4654193d205cf444b1810ab2988f658585e23"
+		;;
+	arm*)
+		_dist_arch="armv6l"
+		checksum="bf061cc3d4951e07904496b5c3d6c82419309d24634835522d786673a3f5438f"
+		;;
+	aarch64*)
+		_dist_arch="arm64"
+		checksum="dcfcb3785292c98f7a75c2276169dfe2d445c19f8ffe1d40b3f7b8f59712d361"
+		;;
+	ppc64le*)
+		_dist_arch="ppc64le"
+		checksum="77056264abcf5444ed0d9ab7552552ae2145ca8fb6c39d33db3c735eaf3f42d2"
+		;;
+esac
+
+distfiles="https://dl.google.com/go/go${version}.linux-${_dist_arch}.tar.gz"
+
+post_build() {
+	[ "$XBPS_TARGET_LIBC" != "musl" ] && return 0
+
+	# we don't have lib64 compatibility path on musl 64-bit systems
+	# use patchelf to replace /lib64/<dynlinker> with /lib/<dynlinker>
+
+	local _interp=$(patchelf --print-interpreter ${wrksrc}/bin/go)
+
+	patchelf --set-interpreter ${_interp/lib64\//lib\/} ${wrksrc}/bin/go
+	patchelf --set-interpreter ${_interp/lib64\//lib\/} ${wrksrc}/bin/godoc
+}
+
+do_install() {
+	vmkdir usr/lib/go1.12
+	vcopy bin usr/lib/go1.12
+	vcopy src usr/lib/go1.12
+	vcopy pkg usr/lib/go1.12
+	vlicense LICENSE
+}

From aa954407a09c1ddbc5b525de6274b909f77a6bd1 Mon Sep 17 00:00:00 2001
From: q66 <daniel@octaforge.org>
Date: Sat, 15 Dec 2018 04:11:57 +0100
Subject: [PATCH 2/2] go: use go1.12-bootstrap, disable ppc* BE, disable
 cross-libc

---
 srcpkgs/go/template | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/srcpkgs/go/template b/srcpkgs/go/template
index c263afc1d38..0349e5e3da2 100644
--- a/srcpkgs/go/template
+++ b/srcpkgs/go/template
@@ -4,14 +4,13 @@ version=1.13.4
 revision=1
 create_wrksrc=yes
 build_wrksrc=go
-hostmakedepends="go1.4-bootstrap"
+hostmakedepends="go1.12-bootstrap"
 short_desc="Go Programming Language"
 maintainer="Michael Aldridge <maldridge@voidlinux.org>"
 license="BSD-3-Clause"
 homepage="http://golang.org/"
 distfiles="https://golang.org/dl/go${version}.src.tar.gz"
 checksum=95dbeab442ee2746b9acf0934c8e2fc26414a0565c008631b04addb8c02e7624
-
 nostrip=yes
 noverifyrdeps=yes
 
@@ -21,20 +20,27 @@ case "${XBPS_TARGET_MACHINE}" in
 	mips*) _goarch=mips ;;
 	i686*) _goarch=386 ;;
 	x86_64*) _goarch=amd64 ;;
+	ppc64le*) _goarch=ppc64le ;;
+	ppc64*) broken="Upstream does not support ELFv2 for big endian ppc64";;
+	ppc*) broken="Upstream does not support 32-bit ppc";;
 	*) _goarch=${XBPS_TARGET_MACHINE} ;;
 esac
 
+if [ "$CROSS_BUILD" ]; then
+	if [ "${XBPS_MACHINE%%-musl}" = "${XBPS_TARGET_MACHINE%%-musl}" ]; then
+		broken="Cross-compiling to different libc is not supported"
+	fi
+fi
+
 do_build() {
 	unset GCC CC CXX LD CFLAGS
 	# FIXME: work around leaking go build-style vars when built as a
 	# dependency
 	unset CGO_CXXFLAGS CGO_CFLAGS CGO_ENABLED
 
-
-	export GOCACHE=off
+	export GOROOT_BOOTSTRAP="/usr/lib/go1.12"
 	export GOROOT=$PWD
 	export GOROOT_FINAL="/usr/lib/go"
-	export GOROOT_BOOTSTRAP="/usr/lib/go1.4"
 	export GOARCH=${_goarch}
 
 	cd "src"

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4
  2019-11-10  0:25 [PR PATCH] go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4 voidlinux-github
                   ` (5 preceding siblings ...)
  2019-11-11 18:33 ` voidlinux-github
@ 2019-11-11 18:33 ` voidlinux-github
  2019-11-11 19:34 ` voidlinux-github
  2019-11-11 20:11 ` voidlinux-github
  8 siblings, 0 replies; 10+ messages in thread
From: voidlinux-github @ 2019-11-11 18:33 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 190 bytes --]

New comment by q66 on void-packages repository

https://github.com/void-linux/void-packages/pull/16287#issuecomment-552560479

Comment:
All prerequisites merged; giving this another CI run.

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4
  2019-11-10  0:25 [PR PATCH] go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4 voidlinux-github
                   ` (6 preceding siblings ...)
  2019-11-11 18:33 ` voidlinux-github
@ 2019-11-11 19:34 ` voidlinux-github
  2019-11-11 20:11 ` voidlinux-github
  8 siblings, 0 replies; 10+ messages in thread
From: voidlinux-github @ 2019-11-11 19:34 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 553 bytes --]

New comment by nilium on void-packages repository

https://github.com/void-linux/void-packages/pull/16287#issuecomment-552582582

Comment:
Mentioned on IRC but forgot to put a comment here. Pretty sure this is the right way to go for compatibility with ppc and any future architectures that are added in given that Go 1.4 does not have the same support for various architectures as recent releases. Assuming gcompat is fully working, I don’t see any issues (outside of the existing i686 container issue, which isn’t related to these changes).

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4
  2019-11-10  0:25 [PR PATCH] go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4 voidlinux-github
                   ` (7 preceding siblings ...)
  2019-11-11 19:34 ` voidlinux-github
@ 2019-11-11 20:11 ` voidlinux-github
  8 siblings, 0 replies; 10+ messages in thread
From: voidlinux-github @ 2019-11-11 20:11 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 264 bytes --]

New comment by q66 on void-packages repository

https://github.com/void-linux/void-packages/pull/16287#issuecomment-552595579

Comment:
Yeah, seems to be working alright. Go does not really need much in terms of C symbols so using gcompat for this is pretty easy.

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, back to index

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-10  0:25 [PR PATCH] go: add a binary bootstrap package (1.12 series for now) and switch the compiler to it from 1.4 voidlinux-github
2019-11-10  0:45 ` voidlinux-github
2019-11-10  1:58 ` voidlinux-github
2019-11-10 16:55 ` [PR PATCH] [Updated] " voidlinux-github
2019-11-10 16:57 ` voidlinux-github
2019-11-10 23:00 ` [PR PATCH] [Updated] " voidlinux-github
2019-11-11 18:33 ` voidlinux-github
2019-11-11 18:33 ` voidlinux-github
2019-11-11 19:34 ` voidlinux-github
2019-11-11 20:11 ` voidlinux-github

Github messages for voidlinux

Archives are clonable: git clone --mirror http://inbox.vuxu.org/voidlinux-github

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://inbox.vuxu.org/vuxu.github.voidlinux


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git