Re: [PR PATCH] [Updated] [RFC] Support for PEP517 build systems with new build style

[ahesford <ahesford@users.noreply.github.com>]

[-- Attachment #1: Type: text/plain, Size: 2542 bytes --]

There is an updated pull request by ahesford against master on the void-packages repository

https://github.com/ahesford/void-packages packaging
https://github.com/void-linux/void-packages/pull/26883

[RFC] Support for PEP517 build systems with new build style
I [have been told](https://github.com/pypa/packaging/issues/363) that PEP517 build systems are the way of the future for Python package building and installation, and `setuptools` will become (or is now) disfavored. `python3-packaging` is the first of our packages to drop `setuptools` and specifically require a PEP517 builder.

This is an attempt to support PEP517 builds in our `python3-module` build style. For now, the preferred (only?) way to do PEP517 builds is to rely on `pip` to do the work. Fortunately, because no PEP517 builder supports compiled extensions, we can avoid the pain of trying to force `pip` to behave with cross compilers (for now).

Use of the PEP517 build procedure in a template is enabled by setting `python_pep517=yes`. If this is adopted, we'll have to modify `xlint` as well.

`pip` can do a one-pass build and install, but I figured it was better to split into a build-wheel stage and an install-wheel stage so people can do `./xbps-src build` and investigate the artifacts.

I am not thrilled with the use of globs in `do_install` when setting a default `$python_pep517_wheel` but, according to [PEP 427](https://www.python.org/dev/peps/pep-0427/#file-name-convention) and its referenced [PEP 425](https://www.python.org/dev/peps/pep-0425), the filename components I'm trying to match with the globs are not easily predicted. In any case, if this produces undesirable behavior in specific templates, the author can manually set that variable. Any other ideas are welcome.

Finally, the build process produces `direct_url.json` in the Python `dist-info` directory to comply with [PEP 610](https://www.python.org/dev/peps/pep-0610), which replaces a simple version number in `pip freeze` output with a `file://` URL pointing to the location of the wheel used for installation. (In our case, this will be `/builddir/$wrksrc/$build_wrksrc/$python_pep517_wheel`.) For distribution packages, I do not believe this is desirable. We can manually remove the file, for example in `do_install`, assuming the `dist-info` directory is predictable. Comments about doing this are welcome.

cc: any @void-linux/pkg-committers with a stake in Python packages

A patch file from https://github.com/void-linux/void-packages/pull/26883.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-packaging-26883.patch --]
[-- Type: text/x-diff, Size: 18718 bytes --]

From f4cb9e9526f735bd347725b1d6165e989fb27218 Mon Sep 17 00:00:00 2001
From: "Andrew J. Hesford" <ajh@sideband.org>
Date: Wed, 2 Dec 2020 22:20:18 -0500
Subject: [PATCH 1/3] build-style/python3-pep517: new style for PEP517 Python
 packages

---
 Manual.md                                     | 22 ++++++++-----
 common/build-style/python3-pep517.sh          | 31 +++++++++++++++++++
 .../environment/build-style/python3-pep517.sh |  2 ++
 3 files changed, 48 insertions(+), 7 deletions(-)
 create mode 100644 common/build-style/python3-pep517.sh
 create mode 100644 common/environment/build-style/python3-pep517.sh

diff --git a/Manual.md b/Manual.md
index 7647f957129..3c4807a7c58 100644
--- a/Manual.md
+++ b/Manual.md
@@ -561,17 +561,22 @@ phase if `${build_style}` is set to `configure`, `gnu-configure` or
 `gnu-makefile` build methods. By default set to
 `PREFIX=/usr DESTDIR=${DESTDIR}`.
 
-- `make_build_target` The target to be passed in to `${make_cmd}` at the build phase if
-`${build_style}` is set to `configure`, `gnu-configure` or `gnu-makefile`
-build methods. Unset by default (`all` target).
+- `make_build_target` The build target. If `${build_style}` is set to `configure`, `gnu-configure`
+or `gnu-makefile`, this is the target passed to `${make_cmd}` in the build phase; when unset, it
+defaults to `all`. If `${build_style}` is `python3-pep517`, this is the path of the package
+directory that should be built as a Python wheel; when unset, defaults to `.` (the current
+directory with respect to the build).
 
 - `make_check_target` The target to be passed in to `${make_cmd}` at the check phase if
 `${build_style}` is set to `configure`, `gnu-configure` or `gnu-makefile`
 build methods. By default set to `check`.
 
-- `make_install_target` The target to be passed in to `${make_cmd}` at the `install-destdir` phase
-if `${build_style}` is set to `configure`, `gnu-configure` or `gnu-makefile`
-build methods. By default set to `install`.
+- `make_install_target` The installation target. When `${build_style}` is set to `configure`,
+`gnu-configure` or `gnu-makefile`, this is the target passed to `${make_command}` in the install
+phase; when unset, it defaults to `install`. If `${build_style}` is `python-pep517`, this is the
+path of the Python wheel produced by the build phase that will be installed; when unset, the
+`python-pep517` build style will look for a wheel matching the package name and version in the
+current directory with respect to the install.
 
 - `patch_args` The arguments to be passed in to the `patch(1)` command when applying
 patches to the package sources during `do_patch()`. Patches are stored in
@@ -1481,7 +1486,10 @@ be your guidance to decide whether or not to split off a `-doc` subpackage.
 
 Python packages should be built with the `python{,2,3}-module` build style, if possible.
 This sets some environment variables required to allow cross compilation. Support to allow
-building a python module for multiple versions from a single template is also possible.
+building a python module for multiple versions from a single template is also possible. The
+`python3-pep517` build style provides means to build python packages that provide a build-system
+definition compliant with [PEP 517](https://www.python.org/dev/peps/pep-0517/) without a traditional
+`setup.py` script.
 
 Python packages that rely on `python3-setuptools` should generally map `setup_requires`
 dependencies in `setup.py` to `hostmakedepends` in the template and `install_requires`
diff --git a/common/build-style/python3-pep517.sh b/common/build-style/python3-pep517.sh
new file mode 100644
index 00000000000..1a3c6d31eed
--- /dev/null
+++ b/common/build-style/python3-pep517.sh
@@ -0,0 +1,31 @@
+#
+# This style is for templates installing python3 modules adhering to PEP517
+#
+
+do_build() {
+	# No PEP517 build tool currently supports compiled extensions
+	# Thus, there is no need to accommodate cross compilation here
+	: ${make_build_target:=.}
+
+	mkdir -p build
+	TMPDIR=build python3 -m pip wheel --no-deps --use-pep517 --no-clean \
+		--no-build-isolation ${make_build_args} ${make_build_target}
+}
+
+do_check() {
+	if python3 -m pytest --help >/dev/null 2>&1; then
+		python3 -m pytest ${make_check_args} ${make_check_target}
+	else
+		msg_warn "Unable to determine tests for PEP517 Python templates"
+		return 0
+	fi
+}
+
+do_install() {
+	# As with do_build, no need to accommodate cross compilation here
+	: ${make_install_target:=${pkgname#python3-}-${version}-*-*-*.whl}
+
+	TMPDIR=build python3 -m pip install --use-pep517 --prefix /usr \
+		--root ${DESTDIR} --no-deps --no-build-isolation \
+		--no-clean ${make_install_args} ${make_install_target}
+}
diff --git a/common/environment/build-style/python3-pep517.sh b/common/environment/build-style/python3-pep517.sh
new file mode 100644
index 00000000000..48f0c1b17f4
--- /dev/null
+++ b/common/environment/build-style/python3-pep517.sh
@@ -0,0 +1,2 @@
+hostmakedepends+=" python3-pip"
+lib32disabled=yes

From e60c5bbdaa74b957cdd9c9cdacbf1ef54ef1ca14 Mon Sep 17 00:00:00 2001
From: "Andrew J. Hesford" <ajh@sideband.org>
Date: Wed, 2 Dec 2020 08:08:33 -0500
Subject: [PATCH 2/3] New package: python3-flit_core-3.0.0

---
 .../python3-flit_core/patches/use_toml.patch  | 254 ++++++++++++++++++
 srcpkgs/python3-flit_core/template            |  23 ++
 2 files changed, 277 insertions(+)
 create mode 100644 srcpkgs/python3-flit_core/patches/use_toml.patch
 create mode 100644 srcpkgs/python3-flit_core/template

diff --git a/srcpkgs/python3-flit_core/patches/use_toml.patch b/srcpkgs/python3-flit_core/patches/use_toml.patch
new file mode 100644
index 00000000000..8a0a668f616
--- /dev/null
+++ b/srcpkgs/python3-flit_core/patches/use_toml.patch
@@ -0,0 +1,254 @@
+From b81b1da55ef0f2768413669725d2874fcb0c29fb Mon Sep 17 00:00:00 2001
+From: Kale Kundert <kale@thekunderts.net>
+Date: Sun, 2 Feb 2020 19:22:34 -0500
+Subject: [PATCH] Replace 'pytoml' with 'toml'
+
+The pytoml package is deprecated, and doesn't support the most recent
+TOML standard (e.g. dotted keys).
+---
+ doc/development.rst                  |  2 +-
+ doc/pyproject_toml.rst               |  2 +-
+ flit/init.py                         |  2 +-
+ flit/tomlify.py                      |  8 ++++----
+ flit_core/flit_core/build_thyself.py |  2 +-
+ flit_core/flit_core/config.py        |  2 +-
+ flit_core/flit_core/sdist.py         |  2 +-
+ flit_core/flit_core/wheel.py         |  2 +-
+ pyproject.toml                       |  2 +-
+ tests/test_init.py                   | 12 ++++++------
+ tests/test_tomlify.py                |  4 ++--
+ tox.ini                              |  2 +-
+ 12 files changed, 21 insertions(+), 21 deletions(-)
+
+diff --git doc/development.rst doc/development.rst
+index 9b8dc5e..f714999 100644
+--- doc/development.rst
++++ doc/development.rst
+@@ -5,7 +5,7 @@ To get a development installation of Flit itself::
+ 
+     git clone https://github.com/takluyver/flit.git
+     cd flit
+-    python3 -m pip install docutils requests pytoml
++    python3 -m pip install docutils requests toml
+     python3 bootstrap_dev.py
+ 
+ This links Flit into the current Python environment, so you can make changes
+diff --git doc/pyproject_toml.rst doc/pyproject_toml.rst
+index f9ddc7f..8d38bf9 100644
+--- doc/pyproject_toml.rst
++++ doc/pyproject_toml.rst
+@@ -120,7 +120,7 @@ Here's the full metadata section from flit itself:
+         "flit_core>=2.2.0",
+         "requests",
+         "docutils",
+-        "pytoml",
++        "toml",
+         "zipfile36; python_version in '3.3 3.4 3.5'",
+     ]
+     requires-python=">=3.5"
+diff --git flit/init.py flit/init.py
+index f7ea93a..b0c77c6 100644
+--- flit/init.py
++++ flit/init.py
+@@ -5,7 +5,7 @@
+ from pathlib import Path
+ import re
+ import sys
+-import pytoml as toml
++import toml
+ 
+ def get_data_dir():
+     """Get the directory path for flit user data files.
+diff --git flit/tomlify.py flit/tomlify.py
+index 0c8db69..a9c9c60 100644
+--- flit/tomlify.py
++++ flit/tomlify.py
+@@ -5,7 +5,7 @@
+ import configparser
+ import os
+ from pathlib import Path
+-import pytoml
++import toml
+ 
+ from .config import metadata_list_fields
+ from .init import TEMPLATE
+@@ -40,11 +40,11 @@ def convert(path):
+ 
+     written_entrypoints = False
+     with Path('pyproject.toml').open('w', encoding='utf-8') as f:
+-        f.write(TEMPLATE.format(metadata=pytoml.dumps(metadata)))
++        f.write(TEMPLATE.format(metadata=toml.dumps(metadata)))
+ 
+         if scripts:
+             f.write('\n[tool.flit.scripts]\n')
+-            pytoml.dump(scripts, f)
++            toml.dump(scripts, f)
+ 
+         for groupname, group in entrypoints.items():
+             if not dict(group):
+@@ -53,7 +53,7 @@ def convert(path):
+             if '.' in groupname:
+                 groupname = '"{}"'.format(groupname)
+             f.write('\n[tool.flit.entrypoints.{}]\n'.format(groupname))
+-            pytoml.dump(OrderedDict(group), f)
++            toml.dump(OrderedDict(group), f)
+             written_entrypoints = True
+ 
+     print("Written 'pyproject.toml'")
+diff --git flit_core/flit_core/build_thyself.py flit_core/flit_core/build_thyself.py
+index 7daf67f..f3aaf7c 100644
+--- flit_core/flit_core/build_thyself.py
++++ flit_core/flit_core/build_thyself.py
+@@ -25,7 +25,7 @@
+     'summary': ('Distribution-building parts of Flit. '
+                     'See flit package for more information'),
+     'requires_dist': [
+-        'pytoml',
++        'toml',
+     ],
+     'requires_python': '>=3.4',
+     'classifiers': [
+diff --git flit_core/flit_core/config.py flit_core/flit_core/config.py
+index 0af9c00..6ed20d3 100644
+--- flit_core/flit_core/config.py
++++ flit_core/flit_core/config.py
+@@ -3,7 +3,7 @@
+ import logging
+ import os
+ import os.path as osp
+-import pytoml as toml
++import toml
+ import re
+ 
+ log = logging.getLogger(__name__)
+diff --git flit_core/flit_core/sdist.py flit_core/flit_core/sdist.py
+index 1fe5bb4..963b4e5 100644
+--- flit_core/flit_core/sdist.py
++++ flit_core/flit_core/sdist.py
+@@ -95,7 +95,7 @@ def __init__(self, module, metadata, cfgdir, reqs_by_extra, entrypoints,
+ 
+     @classmethod
+     def from_ini_path(cls, ini_path: Path):
+-        # Local import so bootstrapping doesn't try to load pytoml
++        # Local import so bootstrapping doesn't try to load toml
+         from .config import read_flit_config
+         ini_info = read_flit_config(ini_path)
+         srcdir = ini_path.parent
+diff --git flit_core/flit_core/wheel.py flit_core/flit_core/wheel.py
+index 1550846..2bc55e1 100644
+--- flit_core/flit_core/wheel.py
++++ flit_core/flit_core/wheel.py
+@@ -83,7 +83,7 @@ def __init__(self, directory, module, metadata, entrypoints, target_fp):
+ 
+     @classmethod
+     def from_ini_path(cls, ini_path, target_fp):
+-        # Local import so bootstrapping doesn't try to load pytoml
++        # Local import so bootstrapping doesn't try to load toml
+         from .config import read_flit_config
+         directory = ini_path.parent
+         ini_info = read_flit_config(ini_path)
+diff --git pyproject.toml pyproject.toml
+index ac8d001..0af74b2 100644
+--- pyproject.toml
++++ pyproject.toml
+@@ -11,7 +11,7 @@ requires=[
+     "flit_core>=3.0.0",
+     "requests",
+     "docutils",
+-    "pytoml",
++    "toml",
+     "zipfile36; python_version in '3.3 3.4 3.5'",
+ ]
+ requires-python=">=3.5"
+diff --git tests/test_init.py tests/test_init.py
+index fb8ad4d..7330ff5 100644
+--- tests/test_init.py
++++ tests/test_init.py
+@@ -6,7 +6,7 @@
+ from unittest.mock import patch
+ import pytest
+ 
+-import pytoml
++import toml
+ 
+ from flit import init
+ 
+@@ -107,7 +107,7 @@ def test_init():
+         generated = Path(td) / 'pyproject.toml'
+         assert_isfile(generated)
+         with generated.open() as f:
+-            data = pytoml.load(f)
++            data = toml.load(f)
+         assert data['tool']['flit']['metadata'][
+                    'author-email'] == "test@example.com"
+         license = Path(td) / 'LICENSE'
+@@ -131,7 +131,7 @@ def test_init_homepage_and_license_are_optional():
+         ti = init.TerminalIniter(td)
+         ti.initialise()
+         with Path(td, 'pyproject.toml').open() as f:
+-            data = pytoml.load(f)
++            data = toml.load(f)
+         assert not Path(td, 'LICENSE').exists()
+     metadata = data['tool']['flit']['metadata']
+     assert metadata == {
+@@ -154,7 +154,7 @@ def test_init_homepage_validator():
+         ti = init.TerminalIniter(td)
+         ti.initialise()
+         with Path(td, 'pyproject.toml').open() as f:
+-            data = pytoml.load(f)
++            data = toml.load(f)
+     metadata = data['tool']['flit']['metadata']
+     assert metadata == {
+         'author': 'Test Author',
+@@ -176,7 +176,7 @@ def test_author_email_field_is_optional():
+         ti = init.TerminalIniter(td)
+         ti.initialise()
+         with Path(td, 'pyproject.toml').open() as f:
+-            data = pytoml.load(f)
++            data = toml.load(f)
+         assert not Path(td, 'LICENSE').exists()
+     metadata = data['tool']['flit']['metadata']
+     assert metadata == {
+@@ -216,7 +216,7 @@ def test_init_readme_found_yes_choosen():
+         ti = init.TerminalIniter(td)
+         ti.initialise()
+         with Path(td, 'pyproject.toml').open() as f:
+-            data = pytoml.load(f)
++            data = toml.load(f)
+ 
+     metadata = data['tool']['flit']['metadata']
+     assert metadata == {
+diff --git tests/test_tomlify.py tests/test_tomlify.py
+index 2bd75dc..a7b7978 100644
+--- tests/test_tomlify.py
++++ tests/test_tomlify.py
+@@ -1,6 +1,6 @@
+ import os
+ from pathlib import Path
+-import pytoml
++import toml
+ from shutil import copy
+ from testpath import assert_isfile
+ 
+@@ -18,7 +18,7 @@ def test_tomlify(copy_sample, monkeypatch):
+     assert_isfile(pyproject_toml)
+ 
+     with pyproject_toml.open(encoding='utf-8') as f:
+-        content = pytoml.load(f)
++        content = toml.load(f)
+ 
+     assert 'build-system' in content
+     assert 'tool' in content
+diff --git tox.ini tox.ini
+index 08ce1bf..d1025e1 100644
+--- tox.ini
++++ tox.ini
+@@ -18,7 +18,7 @@ deps =
+     testpath
+     responses
+     docutils
+-    pytoml
++    toml
+     pytest>=2.7.3
+     pytest-cov
+ 
diff --git a/srcpkgs/python3-flit_core/template b/srcpkgs/python3-flit_core/template
new file mode 100644
index 00000000000..f3b166c60ff
--- /dev/null
+++ b/srcpkgs/python3-flit_core/template
@@ -0,0 +1,23 @@
+# Template file for 'python3-flit_core'
+pkgname=python3-flit_core
+version=3.0.0
+revision=1
+wrksrc="flit-${version}"
+build_wrksrc="flit_core"
+build_style=python3-pep517
+depends="python3-toml"
+short_desc="Simplified packaging of Python modules - PEP 517 build backend"
+maintainer="Andrew J. Hesford <ajh@sideband.org>"
+license="BSD-3-Clause"
+homepage="https://flit.readthedocs.io"
+changelog="https://flit.readthedocs.io/en/latest/history.html"
+distfiles="${PYPI_SITE}/f/flit/flit-${version}.tar.gz"
+checksum=b4fe0f84a1ffbf125d003e253ec98c0b6e3e31290b31fba3ad22d28588c20893
+
+do_check() {
+	: provides no tests
+}
+
+post_install() {
+	vlicense ../LICENSE
+}

From 901c4f9ac021eff0fcbd12c69f0a1dc2c902dd1d Mon Sep 17 00:00:00 2001
From: "Andrew J. Hesford" <ajh@sideband.org>
Date: Wed, 2 Dec 2020 10:28:38 -0500
Subject: [PATCH 3/3] python-packaging: split package

New package: python3-packaging-20.7
---
 srcpkgs/python-packaging/template  | 20 +++++---------------
 srcpkgs/python-packaging/update    |  3 +++
 srcpkgs/python3-packaging          |  1 -
 srcpkgs/python3-packaging/template | 23 +++++++++++++++++++++++
 4 files changed, 31 insertions(+), 16 deletions(-)
 create mode 100644 srcpkgs/python-packaging/update
 delete mode 120000 srcpkgs/python3-packaging
 create mode 100644 srcpkgs/python3-packaging/template

diff --git a/srcpkgs/python-packaging/template b/srcpkgs/python-packaging/template
index e36a04a1540..aa1f64182d7 100644
--- a/srcpkgs/python-packaging/template
+++ b/srcpkgs/python-packaging/template
@@ -1,29 +1,19 @@
 # Template file for 'python-packaging'
 pkgname=python-packaging
 version=20.4
-revision=1
+revision=2
 wrksrc="packaging-${version}"
-build_style=python-module
-pycompile_module="packaging"
-hostmakedepends="python-setuptools python3-setuptools"
-depends="python-attrs python-parsing python-six"
+build_style=python2-module
+hostmakedepends="python-setuptools"
+depends="python-parsing python-six"
 short_desc="Core utilities for Python packages (Python2)"
 maintainer="Alessio Sergi <al3hex@gmail.com>"
 license="Apache-2.0, BSD-2-Clause"
 homepage="https://github.com/pypa/packaging"
+changelog="https://raw.githubusercontent.com/pypa/packaging/master/CHANGELOG.rst"
 distfiles="${PYPI_SITE}/p/packaging/packaging-${version}.tar.gz"
 checksum=4357f74f47b9c12db93624a82154e9b120fa8293699949152b22065d556079f8
 
 post_install() {
 	vlicense LICENSE
 }
-
-python3-packaging_package() {
-	pycompile_module="packaging"
-	depends="python3-attrs python3-parsing python3-six"
-	short_desc="${short_desc/Python2/Python3}"
-	pkg_install() {
-		vmove usr/lib/python3*
-		vlicense LICENSE
-	}
-}
diff --git a/srcpkgs/python-packaging/update b/srcpkgs/python-packaging/update
new file mode 100644
index 00000000000..3b1a41fb94a
--- /dev/null
+++ b/srcpkgs/python-packaging/update
@@ -0,0 +1,3 @@
+# packaging >= 20.5 drops setuptools for flit, which is py3-only;
+# prevent update hits on newer versions since they require py3 to build
+pattern="packaging-\K20\.4(\.[0-9.]+)?(?=.tar.gz)"
diff --git a/srcpkgs/python3-packaging b/srcpkgs/python3-packaging
deleted file mode 120000
index 98299fdb1d7..00000000000
--- a/srcpkgs/python3-packaging
+++ /dev/null
@@ -1 +0,0 @@
-python-packaging
\ No newline at end of file
diff --git a/srcpkgs/python3-packaging/template b/srcpkgs/python3-packaging/template
new file mode 100644
index 00000000000..a4323dad4d6
--- /dev/null
+++ b/srcpkgs/python3-packaging/template
@@ -0,0 +1,23 @@
+# Template file for 'python3-packaging'
+pkgname=python3-packaging
+version=20.7
+revision=1
+wrksrc="packaging-${version}"
+build_style=python3-pep517
+hostmakedepends="python3-flit_core"
+depends="python3-parsing"
+checkdepends="python3-pytest"
+short_desc="Core utilities for Python packages (Python3)"
+maintainer="Andrew J. Hesford <ajh@sideband.org>"
+license="Apache-2.0, BSD-2-Clause"
+homepage="https://github.com/pypa/packaging"
+distfiles="${PYPI_SITE}/p/packaging/packaging-${version}.tar.gz"
+checksum=05af3bb85d320377db281cf254ab050e1a7ebcbf5410685a9a407e18a1f81236
+
+do_check() {
+	: provides no tests
+}
+
+post_install() {
+	vlicense LICENSE
+}