Github messages for voidlinux
 help / color / mirror / Atom feed
* [PR PATCH] [RFC] New numpy build_helper to support cross compilation
@ 2020-07-11  3:42 ahesford
  2020-07-11  3:48 ` [PR REVIEW] " ericonr
                   ` (13 more replies)
  0 siblings, 14 replies; 15+ messages in thread
From: ahesford @ 2020-07-11  3:42 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]

There is a new pull request by ahesford against master on the void-packages repository

https://github.com/ahesford/void-packages numpy-helper
https://github.com/void-linux/void-packages/pull/23515

[RFC] New numpy build_helper to support cross compilation
I plan on adding some packages that link against `python3-numpy` and require the same kinds of environment manipulations currently done in `python3-scipy` to support cross building. The new templates would be simplified if these manipulations were centralized in a build_helper, so here it is. `python3-scipy` has been reworked to take advantage of the new helper.

A patch file from https://github.com/void-linux/void-packages/pull/23515.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-numpy-helper-23515.patch --]
[-- Type: text/x-diff, Size: 4231 bytes --]

From 4f9cc8c0afb7fa51a4e920b7f5b8a48c4b72cc07 Mon Sep 17 00:00:00 2001
From: "Andrew J. Hesford" <ajh@sideband.org>
Date: Fri, 10 Jul 2020 23:35:09 -0400
Subject: [PATCH 1/2] xbps-src: add numpy build_helper

Packages that link against python3-numpy require special consideration
to ensure that, in a cross-building environment, target libraries and
headers as well as the FORTRAN compiler and linker are properly
identifeid. Previously, this was done directly in the python3-scipy
template to support cross compilation. Now, a build_helper generalizes
these changes to support other package templates.
---
 common/build-helper/numpy.sh | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
 create mode 100644 common/build-helper/numpy.sh

diff --git a/common/build-helper/numpy.sh b/common/build-helper/numpy.sh
new file mode 100644
index 00000000000..876e24296aa
--- /dev/null
+++ b/common/build-helper/numpy.sh
@@ -0,0 +1,36 @@
+#
+# numpy - build-helper for packages that compile against python3-numpy
+#
+# This build-helper makes sure packages can find python3-numpy libraries and
+# headers on the target architecture rather than the host, as well as making
+# sure the gfortran cross compiler is properly identified.
+
+# Even for cross compilation, numpy should be available on the host to ensure
+# that the host interpreter doesn't complain about missing deps
+if [[ $hostmakedepends != *"python3-numpy" ]]; then
+	hostmakedepends+=" python3-numpy"
+fi
+
+if [ "$CROSS_BUILD" ]; then
+	if [[ $makedepends != *"python3-numpy"* ]]; then
+		makedepends+=" python3-numpy"
+	fi
+
+	# python3-setuptools finds numpy libs and headers on the host first;
+	# addding search paths up front allows the target to take priority
+	CFLAGS+=" -I${XBPS_CROSS_BASE}/${py3_sitelib}/numpy/core/include"
+	LDFLAGS+=" -L${XBPS_CROSS_BASE}/${py3_sitelib}/numpy/core/lib"
+
+	# distutils from python3-numpy looks to environment variables F77 and
+	# F90 rather than the XBPS-set FC
+	export F77="${FC}"
+	export F90="${FC}"
+
+	# When compiling and linking FORTRAN, distutils from python3-numpy
+	# refuses respect any linker name except "gfortran"; symlink to the
+	# cross-compiler to that the right linker and compiler will be used
+	if _gfortran=$(command -v "${FC}"); then
+		ln -sf "${_gfortran}" "${XBPS_WRAPPERDIR}/gfortran"
+	fi
+	unset _gfortran
+fi

From d20a0801563845eddddc1a4669091cd64f8385b2 Mon Sep 17 00:00:00 2001
From: "Andrew J. Hesford" <ajh@sideband.org>
Date: Fri, 10 Jul 2020 23:39:29 -0400
Subject: [PATCH 2/2] python3-scipy: use new numpy build_helper

---
 srcpkgs/python3-scipy/template | 22 +++-------------------
 1 file changed, 3 insertions(+), 19 deletions(-)

diff --git a/srcpkgs/python3-scipy/template b/srcpkgs/python3-scipy/template
index 3d602c347c9..0554f9c7261 100644
--- a/srcpkgs/python3-scipy/template
+++ b/srcpkgs/python3-scipy/template
@@ -4,10 +4,10 @@ version=1.5.0
 revision=2
 wrksrc="scipy-${version}"
 build_style=python3-module
+build_helper="numpy"
 make_check_args="--force"
-hostmakedepends="gcc-fortran python3-setuptools
- python3-Cython python3-numpy python3-pybind11"
-makedepends="python3-devel python3-numpy python3-pybind11
+hostmakedepends="gcc-fortran python3-setuptools python3-Cython python3-pybind11"
+makedepends="python3-devel python3-pybind11
  $(vopt_if openblas openblas-devel lapack-devel)"
 depends="python3-numpy"
 checkdepends="python3-nose"
@@ -29,25 +29,9 @@ case "$XBPS_TARGET_MACHINE" in
 	*) ;;
 esac
 
-if [ "$CROSS_BUILD" ]; then
-	# Make sure numpy is found for the target arch first
-	CFLAGS+=" -I${XBPS_CROSS_BASE}/${py3_sitelib}/numpy/core/include"
-	LDFLAGS+=" -L${XBPS_CROSS_BASE}/${py3_sitelib}/numpy/core/lib"
-
-	# Tell numpy.distutils where to find FORTRAN compilers
-	export F77="${FC}"
-	export F90="${FC}"
-fi
-
 LDFLAGS+=" -shared"
 
 pre_build() {
-	if [ "$CROSS_BUILD" ]; then
-		# numpy.distutils refuses to find the right linker for FORTRAN
-		# Link the cross compiler so the module will find it as gfortran
-		ln -sf "/usr/bin/${FC}" "${XBPS_WRAPPERDIR}/gfortran"
-	fi
-
 	# Find the right linear algebra subroutines on the target arch
 	: > site.cfg
 	for _blaslib in $(vopt_if openblas openblas "lapack blas"); do

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PR REVIEW] [RFC] New numpy build_helper to support cross compilation
  2020-07-11  3:42 [PR PATCH] [RFC] New numpy build_helper to support cross compilation ahesford
@ 2020-07-11  3:48 ` ericonr
  2020-07-11  4:02 ` sgn
                   ` (12 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: ericonr @ 2020-07-11  3:48 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 618 bytes --]

New review comment by ericonr on void-packages repository

https://github.com/void-linux/void-packages/pull/23515#discussion_r453150642

Comment:
Most of the other helpers seem to add stuff unconditionally. Why not here?

It could work like `add numpy helper -> numpy is added as a depend in the appropriate places, no need to worry about it`, or do you think people should still list `python3-numpy` somewhere in their dependencies? If so, what's the correct place?

I would have something like

```
makedepends+=" python3-numpy"

if [ "$CROSS_BUILD" ]; then
  hostmakedepends+=" python3-numpy"
[...]
```

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PR REVIEW] [RFC] New numpy build_helper to support cross compilation
  2020-07-11  3:42 [PR PATCH] [RFC] New numpy build_helper to support cross compilation ahesford
  2020-07-11  3:48 ` [PR REVIEW] " ericonr
@ 2020-07-11  4:02 ` sgn
  2020-07-11  4:09 ` [PR PATCH] [Updated] " ahesford
                   ` (11 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: sgn @ 2020-07-11  4:02 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 323 bytes --]

New review comment by sgn on void-packages repository

https://github.com/void-linux/void-packages/pull/23515#discussion_r453151551

Comment:
I think you're missing a star `*` at the end.
My *purely* personal preference is:
```sh
case "$hostmakedepends" in
*python3-numpy*) ;;
*) hostmakedepends+=" pkg" ;;
esac
```

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PR PATCH] [Updated] [RFC] New numpy build_helper to support cross compilation
  2020-07-11  3:42 [PR PATCH] [RFC] New numpy build_helper to support cross compilation ahesford
  2020-07-11  3:48 ` [PR REVIEW] " ericonr
  2020-07-11  4:02 ` sgn
@ 2020-07-11  4:09 ` ahesford
  2020-07-11  4:11 ` [PR REVIEW] " ahesford
                   ` (10 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: ahesford @ 2020-07-11  4:09 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 718 bytes --]

There is an updated pull request by ahesford against master on the void-packages repository

https://github.com/ahesford/void-packages numpy-helper
https://github.com/void-linux/void-packages/pull/23515

[RFC] New numpy build_helper to support cross compilation
I plan on adding some packages that link against `python3-numpy` and require the same kinds of environment manipulations currently done in `python3-scipy` to support cross building. The new templates would be simplified if these manipulations were centralized in a build_helper, so here it is. `python3-scipy` has been reworked to take advantage of the new helper.

A patch file from https://github.com/void-linux/void-packages/pull/23515.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-numpy-helper-23515.patch --]
[-- Type: text/x-diff, Size: 4232 bytes --]

From d2bce90651639fe1c299f3bb6eeadac15c030871 Mon Sep 17 00:00:00 2001
From: "Andrew J. Hesford" <ajh@sideband.org>
Date: Fri, 10 Jul 2020 23:35:09 -0400
Subject: [PATCH 1/2] xbps-src: add numpy build_helper

Packages that link against python3-numpy require special consideration
to ensure that, in a cross-building environment, target libraries and
headers as well as the FORTRAN compiler and linker are properly
identifeid. Previously, this was done directly in the python3-scipy
template to support cross compilation. Now, a build_helper generalizes
these changes to support other package templates.
---
 common/build-helper/numpy.sh | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
 create mode 100644 common/build-helper/numpy.sh

diff --git a/common/build-helper/numpy.sh b/common/build-helper/numpy.sh
new file mode 100644
index 00000000000..42f98bf95cb
--- /dev/null
+++ b/common/build-helper/numpy.sh
@@ -0,0 +1,36 @@
+#
+# numpy - build-helper for packages that compile against python3-numpy
+#
+# This build-helper makes sure packages can find python3-numpy libraries and
+# headers on the target architecture rather than the host, as well as making
+# sure the gfortran cross compiler is properly identified.
+
+# Even for cross compilation, numpy should be available on the host to ensure
+# that the host interpreter doesn't complain about missing deps
+if [[ $hostmakedepends != *"python3-numpy"* ]]; then
+	hostmakedepends+=" python3-numpy"
+fi
+
+if [ "$CROSS_BUILD" ]; then
+	if [[ $makedepends != *"python3-numpy"* ]]; then
+		makedepends+=" python3-numpy"
+	fi
+
+	# python3-setuptools finds numpy libs and headers on the host first;
+	# addding search paths up front allows the target to take priority
+	CFLAGS+=" -I${XBPS_CROSS_BASE}/${py3_sitelib}/numpy/core/include"
+	LDFLAGS+=" -L${XBPS_CROSS_BASE}/${py3_sitelib}/numpy/core/lib"
+
+	# distutils from python3-numpy looks to environment variables F77 and
+	# F90 rather than the XBPS-set FC
+	export F77="${FC}"
+	export F90="${FC}"
+
+	# When compiling and linking FORTRAN, distutils from python3-numpy
+	# refuses respect any linker name except "gfortran"; symlink to the
+	# cross-compiler to that the right linker and compiler will be used
+	if _gfortran=$(command -v "${FC}"); then
+		ln -sf "${_gfortran}" "${XBPS_WRAPPERDIR}/gfortran"
+	fi
+	unset _gfortran
+fi

From e93c37b78365f9b42d720400a52ff878b10755b1 Mon Sep 17 00:00:00 2001
From: "Andrew J. Hesford" <ajh@sideband.org>
Date: Fri, 10 Jul 2020 23:39:29 -0400
Subject: [PATCH 2/2] python3-scipy: use new numpy build_helper

---
 srcpkgs/python3-scipy/template | 22 +++-------------------
 1 file changed, 3 insertions(+), 19 deletions(-)

diff --git a/srcpkgs/python3-scipy/template b/srcpkgs/python3-scipy/template
index 3d602c347c9..0554f9c7261 100644
--- a/srcpkgs/python3-scipy/template
+++ b/srcpkgs/python3-scipy/template
@@ -4,10 +4,10 @@ version=1.5.0
 revision=2
 wrksrc="scipy-${version}"
 build_style=python3-module
+build_helper="numpy"
 make_check_args="--force"
-hostmakedepends="gcc-fortran python3-setuptools
- python3-Cython python3-numpy python3-pybind11"
-makedepends="python3-devel python3-numpy python3-pybind11
+hostmakedepends="gcc-fortran python3-setuptools python3-Cython python3-pybind11"
+makedepends="python3-devel python3-pybind11
  $(vopt_if openblas openblas-devel lapack-devel)"
 depends="python3-numpy"
 checkdepends="python3-nose"
@@ -29,25 +29,9 @@ case "$XBPS_TARGET_MACHINE" in
 	*) ;;
 esac
 
-if [ "$CROSS_BUILD" ]; then
-	# Make sure numpy is found for the target arch first
-	CFLAGS+=" -I${XBPS_CROSS_BASE}/${py3_sitelib}/numpy/core/include"
-	LDFLAGS+=" -L${XBPS_CROSS_BASE}/${py3_sitelib}/numpy/core/lib"
-
-	# Tell numpy.distutils where to find FORTRAN compilers
-	export F77="${FC}"
-	export F90="${FC}"
-fi
-
 LDFLAGS+=" -shared"
 
 pre_build() {
-	if [ "$CROSS_BUILD" ]; then
-		# numpy.distutils refuses to find the right linker for FORTRAN
-		# Link the cross compiler so the module will find it as gfortran
-		ln -sf "/usr/bin/${FC}" "${XBPS_WRAPPERDIR}/gfortran"
-	fi
-
 	# Find the right linear algebra subroutines on the target arch
 	: > site.cfg
 	for _blaslib in $(vopt_if openblas openblas "lapack blas"); do

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PR REVIEW] [RFC] New numpy build_helper to support cross compilation
  2020-07-11  3:42 [PR PATCH] [RFC] New numpy build_helper to support cross compilation ahesford
                   ` (2 preceding siblings ...)
  2020-07-11  4:09 ` [PR PATCH] [Updated] " ahesford
@ 2020-07-11  4:11 ` ahesford
  2020-07-11  4:21 ` ahesford
                   ` (9 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: ahesford @ 2020-07-11  4:11 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 372 bytes --]

New review comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/pull/23515#discussion_r453152475

Comment:
Thanks for catching the missing star; fixed in the latest push.

I actually like your `case` better, but `gir.sh` and `qemu.sh` do the tests the same way as this script, so I think it's best to leave as-is for consistency.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PR REVIEW] [RFC] New numpy build_helper to support cross compilation
  2020-07-11  3:42 [PR PATCH] [RFC] New numpy build_helper to support cross compilation ahesford
                   ` (3 preceding siblings ...)
  2020-07-11  4:11 ` [PR REVIEW] " ahesford
@ 2020-07-11  4:21 ` ahesford
  2020-07-11  4:29 ` ericonr
                   ` (8 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: ahesford @ 2020-07-11  4:21 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 1580 bytes --]

New review comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/pull/23515#discussion_r453153283

Comment:
> Most of the other helpers seem to add stuff unconditionally. Why not here?

This isn't true. The `gir.sh` and `qemu.sh` helpers both do conditional appends to `hostmakedepends` and `makedepends` as is done here. The other helpers, `qmake.sh` and `rust.sh`, don't modify `*depends` at all.

> I would have something like [...]

Switching `makedepends` and `hostmakedepends` would have the same outcome, but I think it breaks with "correct" templating for Python. In many cases, Python packages have to be in `hostmakedepends` so that the host interpreter can run them or at least confirm that they are available at build time. (I actually think there are many templates that get this wrong; the fact that `setuptools` will use `pip` or `easy_install` to download dependencies on the host often hides this mistake by populating `wrksrc` with Python eggs to satisfy missing dependencies.) Often, Python packages (especially pure Python) never need to be in `makedepends` at all.

Packages like `python3-scipy` are special cases because they link C and FORTRAN against a library provided with `python3-numpy`. That needs to be in `makedepends` but, on native builds, we happen to already have provided the needed dependency.

To improve consistency with best practice for other Python dependencies, I think `hostmakedepends` should be the "standard" variable here, with `makedepends` added to fill the hole for cross builds.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PR REVIEW] [RFC] New numpy build_helper to support cross compilation
  2020-07-11  3:42 [PR PATCH] [RFC] New numpy build_helper to support cross compilation ahesford
                   ` (4 preceding siblings ...)
  2020-07-11  4:21 ` ahesford
@ 2020-07-11  4:29 ` ericonr
  2020-07-11 16:14 ` ahesford
                   ` (7 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: ericonr @ 2020-07-11  4:29 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 460 bytes --]

New review comment by ericonr on void-packages repository

https://github.com/void-linux/void-packages/pull/23515#discussion_r453153888

Comment:
I see, that makes sense. Do you think we could add some of this info to the manual, as well? I think it would help a lot. Also with information about the helper and how to best use it - so always add numpy to `hostmakedepends`, while `makedepends` should only be touched by the helper, if I understand correctly. 

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PR REVIEW] [RFC] New numpy build_helper to support cross compilation
  2020-07-11  3:42 [PR PATCH] [RFC] New numpy build_helper to support cross compilation ahesford
                   ` (5 preceding siblings ...)
  2020-07-11  4:29 ` ericonr
@ 2020-07-11 16:14 ` ahesford
  2020-07-11 17:03 ` [PR PATCH] [Updated] " ahesford
                   ` (6 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: ahesford @ 2020-07-11 16:14 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 559 bytes --]

New review comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/pull/23515#discussion_r453209823

Comment:
The helper adds the deps where they are necessary and not already provided by the template. I've been meaning to add a description of the new helper in the manual and will push that later today.

Maybe the host/target dependency issue for Python could use some specific attention in the manual, but that is more general and belongs in a separate PR. I'll give it some thought and put something together later.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PR PATCH] [Updated] [RFC] New numpy build_helper to support cross compilation
  2020-07-11  3:42 [PR PATCH] [RFC] New numpy build_helper to support cross compilation ahesford
                   ` (6 preceding siblings ...)
  2020-07-11 16:14 ` ahesford
@ 2020-07-11 17:03 ` ahesford
  2020-07-12 18:30 ` ahesford
                   ` (5 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: ahesford @ 2020-07-11 17:03 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 718 bytes --]

There is an updated pull request by ahesford against master on the void-packages repository

https://github.com/ahesford/void-packages numpy-helper
https://github.com/void-linux/void-packages/pull/23515

[RFC] New numpy build_helper to support cross compilation
I plan on adding some packages that link against `python3-numpy` and require the same kinds of environment manipulations currently done in `python3-scipy` to support cross building. The new templates would be simplified if these manipulations were centralized in a build_helper, so here it is. `python3-scipy` has been reworked to take advantage of the new helper.

A patch file from https://github.com/void-linux/void-packages/pull/23515.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-numpy-helper-23515.patch --]
[-- Type: text/x-diff, Size: 6048 bytes --]

From 9f1c081071751d77abf54064357564d53c186d8f Mon Sep 17 00:00:00 2001
From: "Andrew J. Hesford" <ajh@sideband.org>
Date: Fri, 10 Jul 2020 23:35:09 -0400
Subject: [PATCH 1/2] xbps-src: add numpy build_helper

Packages that link against python3-numpy require special consideration
to ensure that, in a cross-building environment, target libraries and
headers as well as the FORTRAN compiler and linker are properly
identifeid. Previously, this was done directly in the python3-scipy
template to support cross compilation. Now, a build_helper generalizes
these changes to support other package templates.
---
 Manual.md                    | 10 +++++++---
 common/build-helper/numpy.sh | 36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+), 3 deletions(-)
 create mode 100644 common/build-helper/numpy.sh

diff --git a/Manual.md b/Manual.md
index 10082b68da5..7e2cf9a5c02 100644
--- a/Manual.md
+++ b/Manual.md
@@ -940,9 +940,6 @@ suitable environment for working with certain sets of packages.
 
 The current list of available `build_helper` scripts is the following:
 
-- `rust` specifies environment variables required for cross-compiling crates via cargo and
-for compiling cargo -sys crates.
-
 - `gir` specifies dependencies for native and cross builds to deal with
 GObject Introspection. The following variables may be set in the template to handle
 cross builds which require additional hinting or exhibit problems. `GIR_EXTRA_LIBS_PATH` defines
@@ -951,6 +948,10 @@ additional paths to be searched when linking target binaries to be introspected.
 `qemu-<target_arch>-static` when running the target binary. You can for example specify
 `GIR_EXTRA_OPTIONS="-strace"` to see a trace of what happens when running that binary.
 
+- `numpy` specifies a `python3-numpy` dependency in `hostmakedepends` and
+establishes an environment necessary for `numpy.distutils` to function properly
+when cross compiling packages.
+
 - `qemu` sets additional variables for the `cmake` and `meson` build styles to allow
 executing cross-compiled binaries inside qemu.
 It sets `CMAKE_CROSSCOMPILING_EMULATOR` for cmake and `exe_wrapper` for meson
@@ -962,6 +963,9 @@ This aims to fix cross-builds for when the build-style is mixed: e.g. when in a
 `gnu-configure` style the configure script calls `qmake` or a `Makefile` in
 `gnu-makefile` style, respectively.
 
+- `rust` specifies environment variables required for cross-compiling crates via cargo and
+for compiling cargo -sys crates.
+
 <a id="functions"></a>
 ### Functions
 
diff --git a/common/build-helper/numpy.sh b/common/build-helper/numpy.sh
new file mode 100644
index 00000000000..96870f58294
--- /dev/null
+++ b/common/build-helper/numpy.sh
@@ -0,0 +1,36 @@
+#
+# numpy - build-helper for packages that compile against python3-numpy
+#
+# This build-helper makes sure packages can find python3-numpy libraries and
+# headers on the target architecture rather than the host, as well as making
+# sure the gfortran cross compiler is properly identified.
+
+# Even for cross compilation, numpy should be available on the host to ensure
+# that the host interpreter doesn't complain about missing deps
+if [[ $hostmakedepends != *"python3-numpy"* ]]; then
+	hostmakedepends+=" python3-numpy"
+fi
+
+if [ "$CROSS_BUILD" ]; then
+	if [[ $makedepends != *"python3-numpy"* ]]; then
+		makedepends+=" python3-numpy"
+	fi
+
+	# python3-setuptools finds numpy libs and headers on the host first;
+	# addding search paths up front allows the target to take priority
+	CFLAGS+=" -I${XBPS_CROSS_BASE}/${py3_sitelib}/numpy/core/include"
+	LDFLAGS+=" -L${XBPS_CROSS_BASE}/${py3_sitelib}/numpy/core/lib"
+
+	# distutils from python3-numpy looks to environment variables F77 and
+	# F90 rather than the FC variable set by the environment
+	export F77="${FC}"
+	export F90="${FC}"
+
+	# When compiling and linking FORTRAN, distutils from python3-numpy
+	# refuses respect any linker name except "gfortran"; symlink to the
+	# cross-compiler to that the right linker and compiler will be used
+	if _gfortran=$(command -v "${FC}" 2>/dev/null); then
+		ln -sf "${_gfortran}" "${XBPS_WRAPPERDIR}/gfortran"
+	fi
+	unset _gfortran
+fi

From 72a071ee7390fa844493e5d182ce15737c2e4ee2 Mon Sep 17 00:00:00 2001
From: "Andrew J. Hesford" <ajh@sideband.org>
Date: Fri, 10 Jul 2020 23:39:29 -0400
Subject: [PATCH 2/2] python3-scipy: use new numpy build_helper

---
 srcpkgs/python3-scipy/template | 22 +++-------------------
 1 file changed, 3 insertions(+), 19 deletions(-)

diff --git a/srcpkgs/python3-scipy/template b/srcpkgs/python3-scipy/template
index 3d602c347c9..0554f9c7261 100644
--- a/srcpkgs/python3-scipy/template
+++ b/srcpkgs/python3-scipy/template
@@ -4,10 +4,10 @@ version=1.5.0
 revision=2
 wrksrc="scipy-${version}"
 build_style=python3-module
+build_helper="numpy"
 make_check_args="--force"
-hostmakedepends="gcc-fortran python3-setuptools
- python3-Cython python3-numpy python3-pybind11"
-makedepends="python3-devel python3-numpy python3-pybind11
+hostmakedepends="gcc-fortran python3-setuptools python3-Cython python3-pybind11"
+makedepends="python3-devel python3-pybind11
  $(vopt_if openblas openblas-devel lapack-devel)"
 depends="python3-numpy"
 checkdepends="python3-nose"
@@ -29,25 +29,9 @@ case "$XBPS_TARGET_MACHINE" in
 	*) ;;
 esac
 
-if [ "$CROSS_BUILD" ]; then
-	# Make sure numpy is found for the target arch first
-	CFLAGS+=" -I${XBPS_CROSS_BASE}/${py3_sitelib}/numpy/core/include"
-	LDFLAGS+=" -L${XBPS_CROSS_BASE}/${py3_sitelib}/numpy/core/lib"
-
-	# Tell numpy.distutils where to find FORTRAN compilers
-	export F77="${FC}"
-	export F90="${FC}"
-fi
-
 LDFLAGS+=" -shared"
 
 pre_build() {
-	if [ "$CROSS_BUILD" ]; then
-		# numpy.distutils refuses to find the right linker for FORTRAN
-		# Link the cross compiler so the module will find it as gfortran
-		ln -sf "/usr/bin/${FC}" "${XBPS_WRAPPERDIR}/gfortran"
-	fi
-
 	# Find the right linear algebra subroutines on the target arch
 	: > site.cfg
 	for _blaslib in $(vopt_if openblas openblas "lapack blas"); do

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PR PATCH] [Updated] [RFC] New numpy build_helper to support cross compilation
  2020-07-11  3:42 [PR PATCH] [RFC] New numpy build_helper to support cross compilation ahesford
                   ` (7 preceding siblings ...)
  2020-07-11 17:03 ` [PR PATCH] [Updated] " ahesford
@ 2020-07-12 18:30 ` ahesford
  2020-07-12 18:30 ` ahesford
                   ` (4 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: ahesford @ 2020-07-12 18:30 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 718 bytes --]

There is an updated pull request by ahesford against master on the void-packages repository

https://github.com/ahesford/void-packages numpy-helper
https://github.com/void-linux/void-packages/pull/23515

[RFC] New numpy build_helper to support cross compilation
I plan on adding some packages that link against `python3-numpy` and require the same kinds of environment manipulations currently done in `python3-scipy` to support cross building. The new templates would be simplified if these manipulations were centralized in a build_helper, so here it is. `python3-scipy` has been reworked to take advantage of the new helper.

A patch file from https://github.com/void-linux/void-packages/pull/23515.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-numpy-helper-23515.patch --]
[-- Type: text/x-diff, Size: 4785 bytes --]

From 0649652e1971fba4c7bdfcd376288438b86498d8 Mon Sep 17 00:00:00 2001
From: "Andrew J. Hesford" <ajh@sideband.org>
Date: Fri, 10 Jul 2020 23:35:09 -0400
Subject: [PATCH 1/2] xbps-src: add numpy build_helper

Packages that link against python3-numpy require special consideration
to ensure that, in a cross-building environment, target libraries and
headers as well as the FORTRAN compiler and linker are properly
identifeid. Previously, this was done directly in the python3-scipy
template to support cross compilation. Now, a build_helper generalizes
these changes to support other package templates.
---
 common/build-helper/numpy.sh | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
 create mode 100644 common/build-helper/numpy.sh

diff --git a/common/build-helper/numpy.sh b/common/build-helper/numpy.sh
new file mode 100644
index 00000000000..42f98bf95cb
--- /dev/null
+++ b/common/build-helper/numpy.sh
@@ -0,0 +1,36 @@
+#
+# numpy - build-helper for packages that compile against python3-numpy
+#
+# This build-helper makes sure packages can find python3-numpy libraries and
+# headers on the target architecture rather than the host, as well as making
+# sure the gfortran cross compiler is properly identified.
+
+# Even for cross compilation, numpy should be available on the host to ensure
+# that the host interpreter doesn't complain about missing deps
+if [[ $hostmakedepends != *"python3-numpy"* ]]; then
+	hostmakedepends+=" python3-numpy"
+fi
+
+if [ "$CROSS_BUILD" ]; then
+	if [[ $makedepends != *"python3-numpy"* ]]; then
+		makedepends+=" python3-numpy"
+	fi
+
+	# python3-setuptools finds numpy libs and headers on the host first;
+	# addding search paths up front allows the target to take priority
+	CFLAGS+=" -I${XBPS_CROSS_BASE}/${py3_sitelib}/numpy/core/include"
+	LDFLAGS+=" -L${XBPS_CROSS_BASE}/${py3_sitelib}/numpy/core/lib"
+
+	# distutils from python3-numpy looks to environment variables F77 and
+	# F90 rather than the XBPS-set FC
+	export F77="${FC}"
+	export F90="${FC}"
+
+	# When compiling and linking FORTRAN, distutils from python3-numpy
+	# refuses respect any linker name except "gfortran"; symlink to the
+	# cross-compiler to that the right linker and compiler will be used
+	if _gfortran=$(command -v "${FC}"); then
+		ln -sf "${_gfortran}" "${XBPS_WRAPPERDIR}/gfortran"
+	fi
+	unset _gfortran
+fi

From b6296e31c920578a3923049068d4617d1fb35933 Mon Sep 17 00:00:00 2001
From: "Andrew J. Hesford" <ajh@sideband.org>
Date: Fri, 10 Jul 2020 23:39:29 -0400
Subject: [PATCH 2/2] python3-scipy: update to 1.5.1.

---
 srcpkgs/python3-scipy/template | 26 +++++---------------------
 1 file changed, 5 insertions(+), 21 deletions(-)

diff --git a/srcpkgs/python3-scipy/template b/srcpkgs/python3-scipy/template
index 3d602c347c9..90b4b1b8a1e 100644
--- a/srcpkgs/python3-scipy/template
+++ b/srcpkgs/python3-scipy/template
@@ -1,13 +1,13 @@
 # Template file for 'python3-scipy'
 pkgname=python3-scipy
-version=1.5.0
+version=1.5.1
 revision=2
 wrksrc="scipy-${version}"
 build_style=python3-module
+build_helper="numpy"
 make_check_args="--force"
-hostmakedepends="gcc-fortran python3-setuptools
- python3-Cython python3-numpy python3-pybind11"
-makedepends="python3-devel python3-numpy python3-pybind11
+hostmakedepends="gcc-fortran python3-setuptools python3-Cython python3-pybind11"
+makedepends="python3-devel python3-pybind11
  $(vopt_if openblas openblas-devel lapack-devel)"
 depends="python3-numpy"
 checkdepends="python3-nose"
@@ -16,7 +16,7 @@ maintainer="Alessio Sergi <al3hex@gmail.com>"
 license="BSD-3-Clause"
 homepage="https://scipy.org/scipylib/"
 distfiles="https://github.com/scipy/scipy/releases/download/v${version}/scipy-${version}.tar.xz"
-checksum=23baeaa18803d12d1abdff3f5c148b1085c2dc4028c6b8efce652dde2119b41c
+checksum=0728bd66a5251cfeff17a72280ae5a40ec14add217f94868d1415b3c469b610a
 
 build_options="openblas"
 desc_option_openblas="Enable support for openblas accelerated linear algebra"
@@ -29,25 +29,9 @@ case "$XBPS_TARGET_MACHINE" in
 	*) ;;
 esac
 
-if [ "$CROSS_BUILD" ]; then
-	# Make sure numpy is found for the target arch first
-	CFLAGS+=" -I${XBPS_CROSS_BASE}/${py3_sitelib}/numpy/core/include"
-	LDFLAGS+=" -L${XBPS_CROSS_BASE}/${py3_sitelib}/numpy/core/lib"
-
-	# Tell numpy.distutils where to find FORTRAN compilers
-	export F77="${FC}"
-	export F90="${FC}"
-fi
-
 LDFLAGS+=" -shared"
 
 pre_build() {
-	if [ "$CROSS_BUILD" ]; then
-		# numpy.distutils refuses to find the right linker for FORTRAN
-		# Link the cross compiler so the module will find it as gfortran
-		ln -sf "/usr/bin/${FC}" "${XBPS_WRAPPERDIR}/gfortran"
-	fi
-
 	# Find the right linear algebra subroutines on the target arch
 	: > site.cfg
 	for _blaslib in $(vopt_if openblas openblas "lapack blas"); do

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [RFC] New numpy build_helper to support cross compilation
  2020-07-11  3:42 [PR PATCH] [RFC] New numpy build_helper to support cross compilation ahesford
                   ` (8 preceding siblings ...)
  2020-07-12 18:30 ` ahesford
@ 2020-07-12 18:30 ` ahesford
  2020-07-13  2:45 ` [PR PATCH] [Merged]: " ahesford
                   ` (3 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: ahesford @ 2020-07-12 18:30 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 183 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/pull/23515#issuecomment-657258659

Comment:
Might as well bump scipy while I'm at it.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PR PATCH] [Merged]: [RFC] New numpy build_helper to support cross compilation
  2020-07-11  3:42 [PR PATCH] [RFC] New numpy build_helper to support cross compilation ahesford
                   ` (9 preceding siblings ...)
  2020-07-12 18:30 ` ahesford
@ 2020-07-13  2:45 ` ahesford
  2020-07-13  4:13 ` ericonr
                   ` (2 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: ahesford @ 2020-07-13  2:45 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 555 bytes --]

There's a merged pull request on the void-packages repository

[RFC] New numpy build_helper to support cross compilation
https://github.com/void-linux/void-packages/pull/23515

Description:
I plan on adding some packages that link against `python3-numpy` and require the same kinds of environment manipulations currently done in `python3-scipy` to support cross building. The new templates would be simplified if these manipulations were centralized in a build_helper, so here it is. `python3-scipy` has been reworked to take advantage of the new helper.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [RFC] New numpy build_helper to support cross compilation
  2020-07-11  3:42 [PR PATCH] [RFC] New numpy build_helper to support cross compilation ahesford
                   ` (10 preceding siblings ...)
  2020-07-13  2:45 ` [PR PATCH] [Merged]: " ahesford
@ 2020-07-13  4:13 ` ericonr
  2020-07-13  4:17 ` ahesford
  2020-07-13  4:22 ` ericonr
  13 siblings, 0 replies; 15+ messages in thread
From: ericonr @ 2020-07-13  4:13 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 183 bytes --]

New comment by ericonr on void-packages repository

https://github.com/void-linux/void-packages/pull/23515#issuecomment-657348746

Comment:
You forgot to move scipy to revision 1 :p 

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [RFC] New numpy build_helper to support cross compilation
  2020-07-11  3:42 [PR PATCH] [RFC] New numpy build_helper to support cross compilation ahesford
                   ` (11 preceding siblings ...)
  2020-07-13  4:13 ` ericonr
@ 2020-07-13  4:17 ` ahesford
  2020-07-13  4:22 ` ericonr
  13 siblings, 0 replies; 15+ messages in thread
From: ahesford @ 2020-07-13  4:17 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 268 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/pull/23515#issuecomment-657349435

Comment:
I realized this after the package was built. Those are the hazards of bumping an existing PR at the last minute. Mea culpa... 

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [RFC] New numpy build_helper to support cross compilation
  2020-07-11  3:42 [PR PATCH] [RFC] New numpy build_helper to support cross compilation ahesford
                   ` (12 preceding siblings ...)
  2020-07-13  4:17 ` ahesford
@ 2020-07-13  4:22 ` ericonr
  13 siblings, 0 replies; 15+ messages in thread
From: ericonr @ 2020-07-13  4:22 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 156 bytes --]

New comment by ericonr on void-packages repository

https://github.com/void-linux/void-packages/pull/23515#issuecomment-657350333

Comment:
heh, it happens

^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2020-07-13  4:22 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-11  3:42 [PR PATCH] [RFC] New numpy build_helper to support cross compilation ahesford
2020-07-11  3:48 ` [PR REVIEW] " ericonr
2020-07-11  4:02 ` sgn
2020-07-11  4:09 ` [PR PATCH] [Updated] " ahesford
2020-07-11  4:11 ` [PR REVIEW] " ahesford
2020-07-11  4:21 ` ahesford
2020-07-11  4:29 ` ericonr
2020-07-11 16:14 ` ahesford
2020-07-11 17:03 ` [PR PATCH] [Updated] " ahesford
2020-07-12 18:30 ` ahesford
2020-07-12 18:30 ` ahesford
2020-07-13  2:45 ` [PR PATCH] [Merged]: " ahesford
2020-07-13  4:13 ` ericonr
2020-07-13  4:17 ` ahesford
2020-07-13  4:22 ` ericonr

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).