* [ISSUE] numpy headers symlink breaks using numpy in a venv
@ 2022-03-09 15:05 tornaria
2022-03-09 15:06 ` tornaria
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: tornaria @ 2022-03-09 15:05 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 2018 bytes --]
New issue by tornaria on void-packages repository
https://github.com/void-linux/void-packages/issues/36062
Description:
For some (unspecified) reason the numpy template contains this in its `post_install()`:
```
# create compat symlinks for .h files
vmkdir ${py3_inc}
ln -sfr ${DESTDIR}/${py3_sitelib}/numpy/core/include/numpy \
${DESTDIR}/${py3_inc}
```
This creates a symlink `/usr/include/python3.10/numpy -> ../../lib/python3.10/site-packages/numpy/core/include/numpy`
In effect this means that whenever one uses `-I/usr/include/python3.10` to add python headers, one is also adding system numpy headers.
The problem arises when one uses a different version of numpy installed in a venv. To compile against numpy headers one would add `-I$VENV/lib/python3.10/site-packages/numpy/core/include` to use those headers. However, when the `CFLAGS` has something like `-I/usr/include/python3.10 -I$VENV/lib/python3.10/site-packages/numpy/core/include` the symlink installed in the python include dir will take precedence.
This gives a runtime error, e.g. when system numpy is 1.22 (as in void) and venv numpy is 1.21 (as in sagemath) since those versions are binary-incompatible (the numpy.ndarray structure changed size).
A workaround is to ensure the numpy include dir always comes before the python include dir, but the issue is bound to resurface since controlling the order of stuff placed on CFLAGS is kind of difficult.
As a concrete example: sage-9.5 (current release) used to build ok on void linux but after numpy upgraded to 1.22 it no longer builds (this does NOT affect our template for sagemath -- we don't use a venv). See: https://trac.sagemath.org/ticket/33473.
Arguably, nothing else than python headers should be installed in `/usr/include/python3.10`; installing headers from other packages there is forcing the use of a particular version of the package.
Note: debian/ubuntu also adds this symlink. I didn't find other distros that do (I only looked at a few).
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: numpy headers symlink breaks using numpy in a venv
2022-03-09 15:05 [ISSUE] numpy headers symlink breaks using numpy in a venv tornaria
@ 2022-03-09 15:06 ` tornaria
2022-03-09 16:27 ` tornaria
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: tornaria @ 2022-03-09 15:06 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 1065 bytes --]
New comment by tornaria on void-packages repository
https://github.com/void-linux/void-packages/issues/36062#issuecomment-1063016163
Comment:
BTW, the numpy include dir should not be hardcoded, it is supposed to be obtained in setup.py with:
```
>>> import numpy; print(numpy.get_include())
/usr/lib/python3.10/site-packages/numpy/core/include
```
(this is when running system python without venv)
```
>>> import numpy; print(numpy.get_include())
/home/tornaria/src/sage/sage-9.5/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/numpy/core/include
```
(this is when running system python with a venv)
If a python package doesn't compile without the symlink, this is a bug in the package which should use `numpy.get_include()` to know which numpy headers to use. This is the only way to make sure the package uses the correct headers when using numpy from a venv (because `numpy.get_include()` should be called from `setup.py` in the venv so it will give the correct answer, meaning the one that matches the numpy package in the venv).
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: numpy headers symlink breaks using numpy in a venv
2022-03-09 15:05 [ISSUE] numpy headers symlink breaks using numpy in a venv tornaria
2022-03-09 15:06 ` tornaria
@ 2022-03-09 16:27 ` tornaria
2022-03-09 17:28 ` tornaria
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: tornaria @ 2022-03-09 16:27 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 550 bytes --]
New comment by tornaria on void-packages repository
https://github.com/void-linux/void-packages/issues/36062#issuecomment-1063111417
Comment:
> Note: debian/ubuntu also adds this symlink. I didn't find other distros that do (I only looked at a few).
Fedora installs the symlink as `/usr/include/numpy`, which I guess is ok for the current issue but would cause a conflict between `python-numpy` and `python3-numpy` (cf https://bugzilla.redhat.com/show_bug.cgi?id=185079 and https://src.fedoraproject.org/rpms/numpy/blob/f36/f/numpy.spec#_142)
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: numpy headers symlink breaks using numpy in a venv
2022-03-09 15:05 [ISSUE] numpy headers symlink breaks using numpy in a venv tornaria
2022-03-09 15:06 ` tornaria
2022-03-09 16:27 ` tornaria
@ 2022-03-09 17:28 ` tornaria
2022-03-09 18:39 ` ahesford
2022-03-09 20:33 ` [ISSUE] [CLOSED] " ahesford
4 siblings, 0 replies; 6+ messages in thread
From: tornaria @ 2022-03-09 17:28 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 211 bytes --]
New comment by tornaria on void-packages repository
https://github.com/void-linux/void-packages/issues/36062#issuecomment-1063176324
Comment:
@ahesford I'd appreciate to know your point of view on this issue.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: numpy headers symlink breaks using numpy in a venv
2022-03-09 15:05 [ISSUE] numpy headers symlink breaks using numpy in a venv tornaria
` (2 preceding siblings ...)
2022-03-09 17:28 ` tornaria
@ 2022-03-09 18:39 ` ahesford
2022-03-09 20:33 ` [ISSUE] [CLOSED] " ahesford
4 siblings, 0 replies; 6+ messages in thread
From: ahesford @ 2022-03-09 18:39 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 618 bytes --]
New comment by ahesford on void-packages repository
https://github.com/void-linux/void-packages/issues/36062#issuecomment-1063239193
Comment:
I agree that these links are problematic.
This is probably a hack around path issues for cross-builds that compile C extensions with Numpy headers that is no longer necessary. I took some care to properly order CFLAGS in the numpy build helper; hopefully we can just drop the links.
I think there is a new Numpy release; we can bump the version, drop the symlinks and test some packages that rely on the build helper to confirm that everything works as expected.
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [ISSUE] [CLOSED] numpy headers symlink breaks using numpy in a venv
2022-03-09 15:05 [ISSUE] numpy headers symlink breaks using numpy in a venv tornaria
` (3 preceding siblings ...)
2022-03-09 18:39 ` ahesford
@ 2022-03-09 20:33 ` ahesford
4 siblings, 0 replies; 6+ messages in thread
From: ahesford @ 2022-03-09 20:33 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 2021 bytes --]
Closed issue by tornaria on void-packages repository
https://github.com/void-linux/void-packages/issues/36062
Description:
For some (unspecified) reason the numpy template contains this in its `post_install()`:
```
# create compat symlinks for .h files
vmkdir ${py3_inc}
ln -sfr ${DESTDIR}/${py3_sitelib}/numpy/core/include/numpy \
${DESTDIR}/${py3_inc}
```
This creates a symlink `/usr/include/python3.10/numpy -> ../../lib/python3.10/site-packages/numpy/core/include/numpy`
In effect this means that whenever one uses `-I/usr/include/python3.10` to add python headers, one is also adding system numpy headers.
The problem arises when one uses a different version of numpy installed in a venv. To compile against numpy headers one would add `-I$VENV/lib/python3.10/site-packages/numpy/core/include` to use those headers. However, when the `CFLAGS` has something like `-I/usr/include/python3.10 -I$VENV/lib/python3.10/site-packages/numpy/core/include` the symlink installed in the python include dir will take precedence.
This gives a runtime error, e.g. when system numpy is 1.22 (as in void) and venv numpy is 1.21 (as in sagemath) since those versions are binary-incompatible (the numpy.ndarray structure changed size).
A workaround is to ensure the numpy include dir always comes before the python include dir, but the issue is bound to resurface since controlling the order of stuff placed on CFLAGS is kind of difficult.
As a concrete example: sage-9.5 (current release) used to build ok on void linux but after numpy upgraded to 1.22 it no longer builds (this does NOT affect our template for sagemath -- we don't use a venv). See: https://trac.sagemath.org/ticket/33473.
Arguably, nothing else than python headers should be installed in `/usr/include/python3.10`; installing headers from other packages there is forcing the use of a particular version of the package.
Note: debian/ubuntu also adds this symlink. I didn't find other distros that do (I only looked at a few).
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-03-09 20:33 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-03-09 15:05 [ISSUE] numpy headers symlink breaks using numpy in a venv tornaria
2022-03-09 15:06 ` tornaria
2022-03-09 16:27 ` tornaria
2022-03-09 17:28 ` tornaria
2022-03-09 18:39 ` ahesford
2022-03-09 20:33 ` [ISSUE] [CLOSED] " ahesford
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).