[ISSUE] Issue with automake 1.16.4+ and python

[ISSUE] Issue with automake 1.16.4+ and python

[newbluemoon]

[-- Attachment #1: Type: text/plain, Size: 2152 bytes --]

New issue by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/33559

Description:
automake 1.16.4 made some changes to `python.m4` which leads to `$PYTHON_PREFIX` instead of `$prefix` ending up in pkg-config .pc files.

So far I have found two:

/usr/lib/pkgconfig/compizconfig-python.pc has a line
`pyexecdir=${PYTHON_EXEC_PREFIX}/lib/python3.10/site-packages`

and 

/usr/lib/pkgconfig/xcb-proto.pc has
`pythondir=${pc_sysrootdir}${PYTHON_PREFIX}/lib/python3.10/site-packages`

xcb-proto is a dependency of polybar which fails to build right now because `$PYTHON_PREFIX` cannot be resolved.

There is a newer automake 1.16.5 where they again made some changes to `python.m4`, but the problem consists.
I found following patch (against 1.16.5) solves it (it actually reverts the lines to what they were before the change):

```
--- a/m4/python.m4	2021-10-04 04:51:12.000000000 +0200
+++ b/m4/python.m4	2021-10-15 08:38:08.575081215 +0200
@@ -263,7 +263,7 @@
    case $am_cv_python_pythondir in
    $am_py_prefix*)
      am__strip_prefix=`echo "$am_py_prefix" | sed 's|.|.|g'`
-     am_cv_python_pythondir=`echo "$am_cv_python_pythondir" | sed "s,^$am__strip_prefix,\\${PYTHON_PREFIX},"`
+     am_cv_python_pythondir=`echo "$am_cv_python_pythondir" | sed "s,^$am__strip_prefix,${PYTHON_PREFIX},"`
      ;;
    *)
      case $am_py_prefix in
@@ -305,7 +305,7 @@
    case $am_cv_python_pyexecdir in
    $am_py_exec_prefix*)
      am__strip_prefix=`echo "$am_py_exec_prefix" | sed 's|.|.|g'`
-     am_cv_python_pyexecdir=`echo "$am_cv_python_pyexecdir" | sed "s,^$am__strip_prefix,\\${PYTHON_EXEC_PREFIX},"`
+     am_cv_python_pyexecdir=`echo "$am_cv_python_pyexecdir" | sed "s,^$am__strip_prefix,${PYTHON_EXEC_PREFIX},"`
      ;;
    *)
      case $am_py_exec_prefix in
```

But I’m not sure about any implications. I filed a bug report upstream and as soon as it appears on their mailing list I will post the link.

For reference:
https://git.savannah.gnu.org/cgit/automake.git/log/m4/python.m4
The problematic commits are the ones from 2021-05-18 and 2021-09-19.

Re: Issue with automake 1.16.4+ and python

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 720 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/issues/33559#issuecomment-944984595

Comment:
I'm not exactly sure what `automake` is trying to accomplish here. If you rebuild `xcb-proto` with your patched `automake`, which of the following lines do you see in `/usr/lib/pkgconfig/xcb-proto.pc`?
- `pythondir=${pc_sysrootdir}/usr/lib/python3.10/site-packages`, or
- `pythondir=${pc_sysrootdir}${prefix}/lib/python3.10/site-packages`

The latter is what currently appears in the `xcb-proto` package.

In `polybar`, what happens if you `export PYTHON_PREFIX=/usr` either outside of any function or in a `pre_configure` function? Will `pkg-config` to the right thing?

Re: Issue with automake 1.16.4+ and python

[newbluemoon]

[-- Attachment #1: Type: text/plain, Size: 564 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/33559#issuecomment-944989841

Comment:
With the patched automake `xcb-proto.pc` contains the line
```
pythondir=${pc_sysrootdir}${prefix}/lib/python3.10/site-packages
```
which is as it was with 1.16.2. But when you build it with Void’s current 1.16.4 it’s
```
pythondir=${pc_sysrootdir}${PYTHON_PREFIX}/lib/python3.10/site-packages
```

Exporting `PYTHON_PREFIX` in the polybar template either inside or outside `pre_configure` has no effect.

Re: Issue with automake 1.16.4+ and python

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 347 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/issues/33559#issuecomment-944990877

Comment:
Seems reasonable. PR a patch (maybe with an update if you want) and please note whether you get a response to your upstream report. We want to consider any upstream justification for the breaking change.

Re: Issue with automake 1.16.4+ and python

[newbluemoon]

[-- Attachment #1: Type: text/plain, Size: 213 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/33559#issuecomment-944994500

Comment:
Done. When I get a response to the bug report I will post it here.

Re: Issue with automake 1.16.4+ and python

[newbluemoon]

[-- Attachment #1: Type: text/plain, Size: 364 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/33559#issuecomment-944995967

Comment:
In 1.16.5 they tried to restore the old behaviour and added a new configure option to activate the new one.
They might just have forgotten to remove the double backslash. But it’s not really clear to me, either. ;)

Re: Issue with automake 1.16.4+ and python

[newbluemoon]

[-- Attachment #1: Type: text/plain, Size: 2153 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/33559#issuecomment-945054931

Comment:
So I got a response :)

>     On 2021-10-17 12:32 , Karl Berry wrote:
>     >      https://git.savannah.gnu.org/cgit/automake.git/commit/m4/python.m4?id=ed8daa069a6c8ed34f7856c42402ec46f305e670
>     > 
>     >      However, this change leads to pkg-config .pc files containing the
>     >      variable name $PYTHON_PREFIX, not its content
>     > 
>     > I admit I didn't test much explicitly, but my understanding was that the
>     > use of ${PYTHON_PREFIX} in various places instead of its value was
>     > intentional, to allow for overriding at make time.
>     > 
>     > Thus, whatever the process is for creating .pc files, I guess it
>     > may need to change. A previous setting in the Makefile.am should yield
>     > the value of PYTHON_PREFIX to be substituted.
>     
>     The documentation mentions in several places that the variables set by 
>     Automake are only really designed to be used in the Makefiles that it 
>     generates. Using AC_SUBST to directly substitute them into other kinds 
>     of files comes with the risk of exactly this kind of thing happening.
>     
>     You could equally complain that some variables you want to use in the 
>     .pc file are defined in terms of $prefix; except it just happens that 
>     most .pc files already contain a prefix variable, so it works.
>     
>     As Karl alluded to, if you want the fully expanded path, the .pc file 
>     needs to be generated by make, not configure. If you don't do it that 
>     way, the way these Makefile variables are defined will change and break 
>     your non-Makefiles sometimes.
> 
>     - Josh

It seems automake is doing the right thing and xcb-proto should create its .pc file differently.
However, I guess this behaviour is hidden in other projects using autotools, too, and will be revealed one after another when they start using automake >=1.16.4.

Maybe a post-install hook could check for `$PYTHON_PREFIX` and print a warning or error out?

Re: Issue with automake 1.16.4+ and python

[newbluemoon]

[-- Attachment #1: Type: text/plain, Size: 336 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/33559#issuecomment-945061537

Comment:
Just now I saw that there is an approved, simple patch for xcb-proto to adjust it to the new automake behaviour already.
With it polybar builds fine now. Will open a PR to patch xcb-proto.

Re: Issue with automake 1.16.4+ and python

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 899 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/issues/33559#issuecomment-945116914

Comment:
The ugliest part about this is that they decided to use variable names that look totally different from other variables in the pc files, which makes the intention harder to gather. Why not `pyexecprefix` and `pyprefix` To make it clear these should be set like others? For that matter, why does `PYTHON_EXEC_PREFIX` even exist? What does this provide that can't be discerned by `pyexecdir`?

At this point, drop the patch in this PR since they clearly don't want the old behavior. I think whatever this breaks should be fixed by attrition; adding an xbps-src hook to scan pc files will add extra time to package building for little benefit. Presumable other upstreams will fix this problem in due time, and we can patch what we identify in the meantime.

Re: Issue with automake 1.16.4+ and python

[newbluemoon]

[-- Attachment #1: Type: text/plain, Size: 883 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/33559#issuecomment-945138636

Comment:
Already removed the patch from #33591 ;)

Regarding the motivation why they changed it I found
https://gitlab.freedesktop.org/xorg/proto/xcbproto/-/merge_requests/25#note_1061544

> The change in automake was motivated by the fact that PYTHON_PREFIX is not necessarily the same as $prefix. It could be a subdirectory (as in framework installations) or somewhere completely different.
> 
> The next automake release [i.e. 1.16.5] will probably change the default PYTHON_PREFIX back to $prefix for the sake of backwards compatibility. But it's not correct to assume that the two should always be the same. With 1.16.4, you can use the --with-python_prefix configure option to make them the same if that's how you want to configure.

Re: Issue with automake 1.16.4+ and python

[github-actions]

[-- Attachment #1: Type: text/plain, Size: 293 bytes --]

New comment by github-actions[bot] on void-packages repository

https://github.com/void-linux/void-packages/issues/33559#issuecomment-1149373804

Comment:
Issues become stale 90 days after last activity and are closed 14 days after that.  If this issue is still relevant bump it or assign it.

Re: [ISSUE] [CLOSED] Issue with automake 1.16.4+ and python

[github-actions]

[-- Attachment #1: Type: text/plain, Size: 2139 bytes --]

Closed issue by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/33559

Description:
automake 1.16.4 made some changes to `python.m4` which leads to `$PYTHON_PREFIX` instead of `$prefix` ending up in pkg-config .pc files.

So far I have found two:

/usr/lib/pkgconfig/compizconfig-python.pc has a line
`pyexecdir=${PYTHON_EXEC_PREFIX}/lib/python3.10/site-packages`

and 

/usr/lib/pkgconfig/xcb-proto.pc has
`pythondir=${pc_sysrootdir}${PYTHON_PREFIX}/lib/python3.10/site-packages`

xcb-proto is a dependency of polybar which fails to build right now because `$PYTHON_PREFIX` cannot be resolved.

There is a newer automake 1.16.5 where they again made some changes to `python.m4`, but the problem consists.
I found following patch (against 1.16.5) solves it (it actually reverts the lines to what they were before the change):

```
--- a/m4/python.m4	2021-10-04 04:51:12.000000000 +0200
+++ b/m4/python.m4	2021-10-15 08:38:08.575081215 +0200
@@ -263,7 +263,7 @@
    case $am_cv_python_pythondir in
    $am_py_prefix*)
      am__strip_prefix=`echo "$am_py_prefix" | sed 's|.|.|g'`
-     am_cv_python_pythondir=`echo "$am_cv_python_pythondir" | sed "s,^$am__strip_prefix,\\${PYTHON_PREFIX},"`
+     am_cv_python_pythondir=`echo "$am_cv_python_pythondir" | sed "s,^$am__strip_prefix,${PYTHON_PREFIX},"`
      ;;
    *)
      case $am_py_prefix in
@@ -305,7 +305,7 @@
    case $am_cv_python_pyexecdir in
    $am_py_exec_prefix*)
      am__strip_prefix=`echo "$am_py_exec_prefix" | sed 's|.|.|g'`
-     am_cv_python_pyexecdir=`echo "$am_cv_python_pyexecdir" | sed "s,^$am__strip_prefix,\\${PYTHON_EXEC_PREFIX},"`
+     am_cv_python_pyexecdir=`echo "$am_cv_python_pyexecdir" | sed "s,^$am__strip_prefix,${PYTHON_EXEC_PREFIX},"`
      ;;
    *)
      case $am_py_exec_prefix in
```

But I’m not sure about any implications. I filed a bug report upstream:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=51225

For reference:
https://git.savannah.gnu.org/cgit/automake.git/log/m4/python.m4
The problematic commits are the ones from 2021-05-18 and 2021-09-19.