Re: [PR PATCH] [Updated] fix sort-dependencies to use checkdepends

[tornaria <tornaria@users.noreply.github.com>]

[-- Attachment #1: Type: text/plain, Size: 1936 bytes --]

There is an updated pull request by tornaria against master on the void-packages repository

https://github.com/tornaria/void-packages sort-deps
https://github.com/void-linux/void-packages/pull/47888

fix sort-dependencies to use checkdepends
As it is now, if `pkgA` checkdepends on `pkgB`, sort-dependencies will still print `pkgA` before `pkgB`. This causes CI to build `pkgB` twice: first build `pkgA` which forces implicit build of `pkgB`; then build of `pkgB` (explicit, so it will ignore the package is already built).

A concrete example:

```
$ ./xbps-src sort-dependencies python3-process-tests python3-pytest-cov
python3-pytest-cov
python3-process-tests
```
After this commit, the order above is the other way around.

The example above causes `python-process-tests` to be built twice, as shown in:
https://github.com/void-linux/void-packages/actions/runs/7107346278/job/19348602412?pr=47610

<!-- Uncomment relevant sections and delete options which are not applicable -->

#### Testing the changes
- I tested the changes in this PR: **YES**

<!--
#### New package
- This new package conforms to the [package requirements](https://github.com/void-linux/void-packages/blob/master/CONTRIBUTING.md#package-requirements): **YES**|**NO**
-->

<!-- Note: If the build is likely to take more than 2 hours, please add ci skip tag as described in
https://github.com/void-linux/void-packages/blob/master/CONTRIBUTING.md#continuous-integration
and test at least one native build and, if supported, at least one cross build.
Ignore this section if this PR is not skipping CI.
-->
<!--
#### Local build testing
- I built this PR locally for my native architecture, (ARCH-LIBC)
- I built this PR locally for these architectures (if supported. mark crossbuilds):
  - aarch64-musl
  - armv7l
  - armv6l-musl
-->


A patch file from https://github.com/void-linux/void-packages/pull/47888.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-sort-deps-47888.patch --]
[-- Type: text/x-diff, Size: 7356 bytes --]

From f15228ae3174e5f1a5ab6db00583cc6ac6f4b7e3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Gonzalo=20Tornar=C3=ADa?= <tornaria@cmat.edu.uy>
Date: Tue, 26 Dec 2023 12:28:39 -0300
Subject: [PATCH 1/4] common/xbps-src/shutils/build_dependencies.sh: implement
 skip_check_step()

This function contains the logic that determines whether the check step
will be skipped for the current pkg, taking into account `make_check`.

Use this new function in `install_pkg_deps()` so that it uses a more
accurate condition to skip installing check dependencies.

For instance, check dependencies for a pkg with `make_check=extended`
will no longer be installed when using `-Q`. Similar for `make_check=ci-skip`.

Replaces: #46207
---
 common/xbps-src/shutils/build_dependencies.sh | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/common/xbps-src/shutils/build_dependencies.sh b/common/xbps-src/shutils/build_dependencies.sh
index 20f61528d14c2..9eadb674b384d 100644
--- a/common/xbps-src/shutils/build_dependencies.sh
+++ b/common/xbps-src/shutils/build_dependencies.sh
@@ -124,6 +124,17 @@ check_installed_pkg() {
     return 1
 }
 
+#
+# Return 0 if we will skip the check step
+#
+skip_check_step() {
+    [ -z "$XBPS_CHECK_PKGS" ] ||
+        [ "$XBPS_CROSS_BUILD" ] ||
+        [ "$make_check" = ci-skip  -a "$XBPS_BUILD_ENVIRONMENT" = void-packages-ci ] ||
+        [ "$make_check" = extended -a "$XBPS_CHECK_PKGS" != full ] ||
+        [ "$make_check" = no ]
+}
+
 #
 # Build all dependencies required to build and run.
 #
@@ -137,7 +148,7 @@ install_pkg_deps() {
     local -a host_missing_deps missing_deps missing_rdeps
 
     [ -z "$pkgname" ] && return 2
-    [ -z "$XBPS_CHECK_PKGS" ] && unset checkdepends
+    skip_check_step && unset checkdepends
 
     if [[ $build_style ]] || [[ $build_helper ]]; then
         style=" with"
@@ -208,7 +219,7 @@ install_pkg_deps() {
     #
     # Host check dependencies.
     #
-    if [[ ${checkdepends} ]] && [[ $XBPS_CHECK_PKGS ]] && [ -z "$XBPS_CROSS_BUILD" ]; then
+    if [[ ${checkdepends} ]]; then
         templates=""
         # check validity
         for f in ${checkdepends}; do

From 7be0492ad0109836b0f5ccdb76db3b622ea41001 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Gonzalo=20Tornar=C3=ADa?= <tornaria@cmat.edu.uy>
Date: Sat, 23 Dec 2023 19:40:53 -0300
Subject: [PATCH 2/4] ./xbps-src show-build-deps: include checkdepends when
 using -Q or -K

Due to this change, `./xbps-src sort-dependencies` will take
checkdepends into account when using -Q or -K.

Before this commit, if `pkgA` checkdepends on `pkgB`, sort-dependencies
could still print `pkgA` before `pkgB`. This causes CI to build `pkgB`
twice: first when building `pkgA`, which forces implicit build of pkgB;
second when building `pkgB` (explicit, so it will ignore the package is
already built).

The implementation uses `skip_check_step()` from previous commit, for
consistency, so checkdepends are only taken into account if the check
step would be enabled.

In particular, nothing is changed unless -Q or -K flag is passed.

EXAMPLE:

Before:
```
$ ./xbps-src -Q sort-dependencies python3-process-tests python3-pytest-cov
python3-pytest-cov
python3-process-tests
```

After:
```
$ ./xbps-src -Q sort-dependencies python3-process-tests python3-pytest-cov
python3-process-tests
python3-pytest-cov
```
---
 common/xbps-src/shutils/show.sh | 8 +++++++-
 xbps-src                        | 7 +++++++
 2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/common/xbps-src/shutils/show.sh b/common/xbps-src/shutils/show.sh
index 606396d2b4b1a..18817f5c619a3 100644
--- a/common/xbps-src/shutils/show.sh
+++ b/common/xbps-src/shutils/show.sh
@@ -117,7 +117,9 @@ show_pkg_build_depends() {
 }
 
 show_pkg_build_deps() {
-    show_pkg_build_depends "${makedepends} $(setup_pkg_depends '' 1 1)" "${hostmakedepends}"
+    local build_depends="${makedepends} $(setup_pkg_depends '' 1 1)"
+    skip_check_step || build_depends+=" ${checkdepends}"
+    show_pkg_build_depends "${build_depends}" "${hostmakedepends}"
 }
 
 show_pkg_hostmakedepends() {
@@ -128,6 +130,10 @@ show_pkg_makedepends() {
     show_pkg_build_depends "${makedepends}" ""
 }
 
+show_pkg_checkdepends() {
+    show_pkg_build_depends "${checkdepends}" ""
+}
+
 show_pkg_build_options() {
     local f
 
diff --git a/xbps-src b/xbps-src
index cdb5f8c6d643e..b8ce3b21b13a4 100755
--- a/xbps-src
+++ b/xbps-src
@@ -91,6 +91,9 @@ show-avail <pkgname>
 show-build-deps <pkgname>
     Show required build dependencies for <pkgname>.
 
+show-check-deps <pkgname>
+    Show required check dependencies for <pkgname>.
+
 show-deps <pkgname>
     Show required run-time dependencies for <pkgname>. Package must be
     installed into destdir.
@@ -869,6 +872,10 @@ case "$XBPS_TARGET" in
         read_pkg ignore-problems
         show_pkg_makedepends
         ;;
+    show-checkdepends)
+        read_pkg ignore-problems
+        show_pkg_checkdepends
+        ;;
     show-pkg-var-dump)
         read_pkg ignore-problems
         for sub_name in $subpackages; do

From 177b6710406b502da2d6c6e49b5335234a61b243 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Gonzalo=20Tornar=C3=ADa?= <tornaria@cmat.edu.uy>
Date: Tue, 26 Dec 2023 12:46:37 -0300
Subject: [PATCH 3/4] common/travis/build.sh: use $test for sort-dependencies

This ensures that checkdepends will be taken into account in the build
order whenever test is enabled.
---
 common/travis/build.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/common/travis/build.sh b/common/travis/build.sh
index 007a61f994d78..b319057344304 100755
--- a/common/travis/build.sh
+++ b/common/travis/build.sh
@@ -10,7 +10,7 @@ if [ "$3" = 1 ]; then
 	test="-Q"
 fi
 
-PKGS=$(/hostrepo/xbps-src sort-dependencies $(cat /tmp/templates))
+PKGS=$(/hostrepo/xbps-src $test sort-dependencies $(cat /tmp/templates))
 
 for pkg in ${PKGS}; do
 	/hostrepo/xbps-src -j$(nproc) -s -H "$HOME"/hostdir $arch $test pkg "$pkg"

From c46793f2daf3a6ab0306702b1c9a402c03ce2e54 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Gonzalo=20Tornar=C3=ADa?= <tornaria@cmat.edu.uy>
Date: Tue, 26 Dec 2023 10:38:49 -0300
Subject: [PATCH 4/4] common/scripts/xbps-cycles.py: add -Q and -K options for
 dependencies

---
 common/scripts/xbps-cycles.py | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/common/scripts/xbps-cycles.py b/common/scripts/xbps-cycles.py
index 7710381eb2925..1f51fe71fd809 100755
--- a/common/scripts/xbps-cycles.py
+++ b/common/scripts/xbps-cycles.py
@@ -101,6 +101,10 @@ def find_cycles(depmap, xbpsdir):
 			help='Directory used to cache build dependencies (must exist)')
 	parser.add_argument('-d', '--directory',
 			default=None, help='Path to void-packages repo')
+	parser.add_argument('-Q', dest='check_pkgs', action='store_const',
+			const='yes', help='Use build dependencies for check -Q')
+	parser.add_argument('-K', dest='check_pkgs', action='store_const',
+			const='full', help='Use build dependencies for check -K')
 
 	args = parser.parse_args()
 
@@ -108,6 +112,9 @@ def find_cycles(depmap, xbpsdir):
 		try: args.directory = os.environ['XBPS_DISTDIR']
 		except KeyError: args.directory = '.'
 
+	if args.check_pkgs:
+		os.environ['XBPS_CHECK_PKGS'] = args.check_pkgs
+
 	pool = multiprocessing.Pool(processes = args.jobs)
 
 	pattern = os.path.join(args.directory, 'srcpkgs', '*')