Re: [PR PATCH] [Updated] [RFC] xbps-src: add -A flag for host arch, create masterdir per host arch

[classabbyamp <classabbyamp@users.noreply.github.com>]

[-- Attachment #1: Type: text/plain, Size: 2494 bytes --]

There is an updated pull request by classabbyamp against master on the void-packages repository

https://github.com/classabbyamp/void-packages host-arch
https://github.com/void-linux/void-packages/pull/46263

[RFC] xbps-src: add -A flag for host arch, create masterdir per host arch
#### Testing the changes
- I tested the changes in this PR: **YES**

### Overview

- default masterdir naming now includes the host `XBPS_ARCH`, e.g. `masterdir-x86_64`
- setting the host `XBPS_ARCH` can be done with `-A <arch>`
    - This will create a new masterdir per host arch automatically
- using an arbitrary masterdir is still supported via `-m <masterdir>`, and overrides whatever `-A` would set
- the invocation of `[binary-]bootstrap` no longer (ab)uses `$XBPS_TARGET_PKG` for setting the masterdir's host architecture

#### Before
*default:*
```
$ ./xbps-src binary-bootstrap
$ ls
... masterdir ...
```
*nondefault masterdir:*
```
$ ./xbps-src -m masterdir-idk binary-bootstrap
$ ls
... masterdir-idk ...
```
*specific host arch:*
```
$ ./xbps-src -m masterdir-musl binary-bootstrap x86_64-musl
$ ls
... masterdir-musl ...
```
*cross with specific host arch:*
```
$ ./xbps-src -m masterdir-musl binary-bootstrap x86_64-musl
$ ./xbps-src -m masterdir-musl -a aarch64-musl pkg foo
```

#### After
*default:*
```
$ ./xbps-src binary-bootstrap
$ ls
... masterdir-x86_64 ...
```
*nondefault masterdir:*
```
$ ./xbps-src -m masterdir-idk binary-bootstrap
$ ls
... masterdir-idk ...
```
*specific host arch:*
```
$ ./xbps-src -A x86_64-musl binary-bootstrap
$ ls
... masterdir-x86_64-musl ...
```
*cross with specific host arch:*
```
$ ./xbps-src -A x86_64-musl binary-bootstrap
$ ./xbps-src -A x86_64-musl -a aarch64-musl pkg foo
```

### Thoughts
- ~~By removing the `[binary-]bootstrap [arch]` invocation style, there will be changes needed to buildbot configuration. If left in, should `-A <arch>` take priority over `[arch]` if both are specified?~~ buildbot config will be changed
- ~~This could be an opportunity to change the name of `masterdir` to something else, like `masterdir` (the containers CI uses are already called `void-buildroot-<libc>`)~~ if this changes, it will be separate; there's too much to change both coherently

### TODO
- [x] various documentation/other xbps-src changes as needed
- [x] buildbot changes if needed


A patch file from https://github.com/void-linux/void-packages/pull/46263.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-host-arch-46263.patch --]
[-- Type: text/x-diff, Size: 12367 bytes --]

From c571198947e8f4343aa82a2a047d573af271ca42 Mon Sep 17 00:00:00 2001
From: classabbyamp <void@placeviolette.net>
Date: Tue, 26 Sep 2023 03:16:39 -0400
Subject: [PATCH 1/3] xbps-src: add -A flag for host arch, create masterdir per
 host arch

---
 xbps-src | 68 +++++++++++++++++++++++++++++++++++---------------------
 1 file changed, 43 insertions(+), 25 deletions(-)

diff --git a/xbps-src b/xbps-src
index 712a578c6f91a..fcc7bb988e12c 100755
--- a/xbps-src
+++ b/xbps-src
@@ -14,14 +14,17 @@ $PROGNAME: [options] <target> [arguments]
 
 Targets: (only one may be specified)
 
-binary-bootstrap [arch]
+binary-bootstrap
     Install bootstrap packages from host repositories into <masterdir>.
-    If the optional 'arch' argument is set, it will install bootstrap packages
-    from this architecture, and its required xbps utilities. The <masterdir>
-    will be initialized for chroot operations.
+    If the optional '-A <arch>' flag is set, it will install bootstrap packages
+    from this architecture, and its required xbps utilities. The <masterdir> will
+    be initialized for chroot operations.
 
 bootstrap
     Build and install from source the bootstrap packages into <masterdir>.
+    If the optional '-A <arch>' flag is set, it will build and install bootstrap
+    packages from this architecture, and its required xbps utilities. The <masterdir>
+    will be initialized for chroot operations.
 
 bootstrap-update
     Updates bootstrap packages with latest versions available from registered
@@ -151,6 +154,13 @@ Options:
 
 -1  If dependencies of target package are missing, fail instead of building them.
 
+-A <host>
+    Use this host machine. Automatically creates masterdir-<host> if it doesn't
+    already exist. Some host machines may require qemu-user-static and
+    binfmt-support if not natively supported by the processor. Supported hosts:
+
+$(print_cross_targets)
+
 -a  <target>
     Cross compile packages for this target machine. Supported targets:
 
@@ -194,7 +204,7 @@ $(print_cross_targets)
 -m  <masterdir>
     Absolute path to a directory to be used as masterdir.
     The masterdir is the main directory to build/store/compile packages.
-    If unset defaults to void-packages/masterdir.
+    If unset defaults to void-packages/masterdir-<host>.
 
 -N  Disable use of remote repositories to resolve dependencies.
 
@@ -290,19 +300,8 @@ check_native_arch() {
         else
             export XBPS_ARCH=$(xbps-uhelper arch)
         fi
-    elif [ "$XBPS_TARGET" = 'binary-bootstrap' ] && [ "$XBPS_TARGET_PKG" ]; then
-        # ^ special case for binary-bootstrap for e.g:
-        # x86_64 -> x86_64-musl
-        # x86_64 -> i686
-        export XBPS_ARCH=$XBPS_TARGET_PKG
     else
-        LDD=$(ldd --version 2>&1|head -1)
-        if [[ $LDD == *musl* ]]; then
-            export XBPS_ARCH=${XBPS_MACHINE%-musl}-musl
-        else
-            # XBPS_ARCH == $(uname -m)
-            export XBPS_ARCH=$(uname -m)
-        fi
+        export XBPS_ARCH="$XBPS_MACHINE"
     fi
 }
 
@@ -372,7 +371,7 @@ readonly XBPS_SRC_VERSION="113"
 export XBPS_MACHINE=$(xbps-uhelper -C /dev/null arch)
 
 XBPS_OPTIONS=
-XBPS_OPTSTRING="1a:bc:CEfgGhH:iIj:Lm:No:p:qsQKr:tV"
+XBPS_OPTSTRING="1A:a:bc:CEfgGhH:iIj:Lm:No:p:qsQKr:tV"
 
 # Preprocess arguments in order to allow options before and after XBPS_TARGET.
 eval set -- $(getopt "$XBPS_OPTSTRING" "$@");
@@ -383,6 +382,7 @@ eval set -- $(getopt "$XBPS_OPTSTRING" "$@");
 while getopts "$XBPS_OPTSTRING" opt; do
     case $opt in
         1) XBPS_ARG_BUILD_ONLY_ONE_PKG=yes; XBPS_OPTIONS+=" -1";;
+        A) XBPS_ARG_MACHINE="$OPTARG"; XBPS_OPTIONS+=" -A $OPTARG";;
         a) XBPS_ARG_CROSS_BUILD="$OPTARG"; XBPS_OPTIONS+=" -a $OPTARG";;
         b) XBPS_ARG_IGNORE_BROKENNESS=yes; XBPS_OPTIONS+=" -b";;
         c) XBPS_ARG_CONFIG="$OPTARG"; XBPS_OPTIONS+=" -c $OPTARG";;
@@ -493,6 +493,7 @@ fi
 [ -n "$XBPS_ARG_PRINT_VARIABLES" ] && XBPS_PRINT_VARIABLES="$XBPS_ARG_PRINT_VARIABLES"
 [ -n "$XBPS_ARG_ALT_REPOSITORY" ] && XBPS_ALT_REPOSITORY="$XBPS_ARG_ALT_REPOSITORY"
 [ -n "$XBPS_ARG_STRICT" ] && XBPS_STRICT="$XBPS_ARG_STRICT"
+[ -n "$XBPS_ARG_MACHINE" ] && XBPS_MACHINE="$XBPS_ARG_MACHINE"
 [ -n "$XBPS_ARG_CROSS_BUILD" ] && XBPS_CROSS_BUILD="$XBPS_ARG_CROSS_BUILD"
 [ -n "$XBPS_ARG_CHECK_PKGS" ] && XBPS_CHECK_PKGS="$XBPS_ARG_CHECK_PKGS"
 [ -n "$XBPS_ARG_MAKEJOBS" ] && XBPS_MAKEJOBS="$XBPS_ARG_MAKEJOBS"
@@ -527,9 +528,18 @@ if [ -z "$IN_CHROOT" -a "$UID" -eq 0 -a -z "$XBPS_ALLOW_CHROOT_BREAKOUT" ]; then
     exit 1
 fi
 
-# if XBPS_MASTERDIR unset, defaults to $XBPS_DISTDIR/masterdir.
-: ${XBPS_MASTERDIR:=$XBPS_DISTDIR/masterdir}
-[ ! -d $XBPS_MASTERDIR ] &&  mkdir -p $XBPS_MASTERDIR
+# if XBPS_MASTERDIR unset, defaults to $XBPS_DISTDIR/masterdir-<host machine>
+# unless in a container for simplicity of container/CI usage
+if [ "$XBPS_CHROOT_CMD" = ethereal ]; then
+    : "${XBPS_MASTERDIR:=$XBPS_DISTDIR/masterdir}"
+else
+    : "${XBPS_MASTERDIR:=$XBPS_DISTDIR/masterdir-$XBPS_MACHINE}"
+    # but use $XBPS_DISTDIR/masterdir if it exists and the new style doesn't
+    if [ ! -d "${XBPS_MASTERDIR}" ] && [ -d "${XBPS_DISTDIR}/masterdir" ]; then
+        XBPS_MASTERDIR="${XBPS_DISTDIR}/masterdir"
+    fi
+fi
+[ ! -d "$XBPS_MASTERDIR" ] && mkdir -p "$XBPS_MASTERDIR"
 
 # if XBPS_HOSTDIR unset, defaults to $XBPS_DISTDIR/hostdir.
 : ${XBPS_HOSTDIR:=$XBPS_DISTDIR/hostdir}
@@ -617,10 +627,10 @@ if [ -z "$XBPS_REINIT" -a -s $XBPS_MASTERDIR/.xbps_chroot_init ]; then
         linux32 xbps-reconfigure -r ${XBPS_MASTERDIR} -a &>/dev/null
         # reexec itself via linux32
         export XBPS_REINIT=1
-        exec linux32 $0 ${XBPS_OPTIONS} $@
+        exec linux32 $0 ${XBPS_OPTIONS} "$@"
     fi
 fi
-if [ "$XBPS_ARCH" ]; then
+if [ -n "$XBPS_ARCH" ]; then
     export XBPS_MACHINE=$XBPS_ARCH
 fi
 # At this point if XBPS_TARGET_MACHINE isn't defined we assume
@@ -743,13 +753,21 @@ reconfigure_base_chroot
 #
 case "$XBPS_TARGET" in
     binary-bootstrap)
-        install_base_chroot ${XBPS_TARGET_PKG:=$XBPS_MACHINE}
+        if [ -n "$XBPS_TARGET_PKG" ]; then
+            msg_error "xbps-src: binary-bootstrap <arch> is no longer supported, use '-A <arch>' instead\n"
+            exit 1
+        fi
+        install_base_chroot "$XBPS_MACHINE"
         ;;
     bootstrap)
         if [ -n "$XBPS_CHECK_PKGS" ]; then
             msg_error "xbps-src: disable tests for bootstrap\n"
             exit 1
         fi
+        if [ -n "$XBPS_TARGET_PKG" ]; then
+            msg_error "xbps-src: bootstrap <arch> is no longer supported, use '-A <arch>' instead\n"
+            exit 1
+        fi
         # base-chroot building on host
         # check for required host utils
         check_reqhost_utils bootstrap
@@ -761,7 +779,7 @@ case "$XBPS_TARGET" in
                 base-chroot base-chroot $XBPS_TARGET || exit 1
         ) || exit 1
         [ -d $XBPS_MASTERDIR ] && rm -rf $XBPS_MASTERDIR
-        install_base_chroot ${XBPS_TARGET_PKG:=$XBPS_MACHINE}
+        install_base_chroot "$XBPS_MACHINE"
         ;;
     bootstrap-update)
         if [ -n "$CHROOT_READY" -a -z "$IN_CHROOT" ]; then

From 3701ccc60c01ccc92424cab06564e963a738038e Mon Sep 17 00:00:00 2001
From: classabbyamp <void@placeviolette.net>
Date: Tue, 26 Sep 2023 03:18:01 -0400
Subject: [PATCH 2/3] xbps: patch xbps-src completions for -A

---
 srcpkgs/xbps/patches/xbps-src-comp.patch | 21 +++++++++++++++++++++
 srcpkgs/xbps/template                    |  2 +-
 2 files changed, 22 insertions(+), 1 deletion(-)
 create mode 100644 srcpkgs/xbps/patches/xbps-src-comp.patch

diff --git a/srcpkgs/xbps/patches/xbps-src-comp.patch b/srcpkgs/xbps/patches/xbps-src-comp.patch
new file mode 100644
index 0000000000000..bb0234f38463d
--- /dev/null
+++ b/srcpkgs/xbps/patches/xbps-src-comp.patch
@@ -0,0 +1,21 @@
+--- a/data/_xbps_src
++++ b/data/_xbps_src
+@@ -26,6 +26,7 @@
+ }
+ 
+ _arguments -s : \
++	'-A[Host architecture]:architecture:($archs)' \
+ 	'-a[Cross compile packages]:architecture:($archs)' \
+ 	'-C[Do not remove build directory/autodeps/destdir]' \
+ 	'-E[Exit immediately when binary package already exists]' \
+@@ -64,9 +65,7 @@
+ 		case $words[1] in
+ 			build|check|configure|extract|fetch|install|patch|pkg|show|show-avail|show-build-deps|show-hostmakedepends|show-makedepends|show-options|update-check)
+ 				_arguments ':package:_xbps_src_all_packages' && ret=0;;
+-			binary-bootstrap)
+-				_arguments '::architecture:($archs)' && ret=0;;
+-			bootstrap|bootstrap-update|chroot|clean-repocache|consistency-check|list|purge-distfiles|remove-autodeps|show-repo-updates|show-sys-updates|update-bulk|update-sys|update-hash-cache)
++			binary-bootstrap|bootstrap|bootstrap-update|chroot|clean-repocache|consistency-check|list|purge-distfiles|remove-autodeps|show-repo-updates|show-sys-updates|update-bulk|update-sys|update-hash-cache)
+ 				# no further arguments
+ 				ret=0;;
+ 			clean)
diff --git a/srcpkgs/xbps/template b/srcpkgs/xbps/template
index 5b34b331d58da..060a70563aeb7 100644
--- a/srcpkgs/xbps/template
+++ b/srcpkgs/xbps/template
@@ -1,7 +1,7 @@
 # Template file for 'xbps'
 pkgname=xbps
 version=0.59.2
-revision=1
+revision=2
 bootstrap=yes
 build_style=configure
 short_desc="XBPS package system utilities"

From af11ee6f1aab718ccfef9cd060b46bcdc7e6ddec Mon Sep 17 00:00:00 2001
From: classabbyamp <void@placeviolette.net>
Date: Tue, 16 Jan 2024 23:33:30 -0500
Subject: [PATCH 3/3] README.md: update documentation for new masterdir
 creation

---
 README.md | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

diff --git a/README.md b/README.md
index 8710f20ae1916..bd7fed879019f 100644
--- a/README.md
+++ b/README.md
@@ -217,7 +217,7 @@ The following directory hierarchy is used with a default configuration file:
             |  |- repocache ...
             |  |- sources ...
             |
-            |- masterdir
+            |- masterdir-<arch>
             |  |- builddir -> ...
             |  |- destdir -> ...
             |  |- host -> bind mounted from <hostdir>
@@ -226,7 +226,7 @@ The following directory hierarchy is used with a default configuration file:
 
 The description of these directories is as follows:
 
- - `masterdir`: master directory to be used as rootfs to build/install packages.
+ - `masterdir-<arch>`: master directory to be used as rootfs to build/install packages.
  - `builddir`: to unpack package source tarballs and where packages are built.
  - `destdir`: to install packages, aka **fake destdir**.
  - `hostdir/ccache`: to store ccache data if the `XBPS_CCACHE` option is enabled.
@@ -450,7 +450,7 @@ and `xbps-src` should be fully functional; just start the `bootstrap` process, i
 
     $ ./xbps-src binary-bootstrap
 
-The default masterdir is created in the current working directory, i.e `void-packages/masterdir`.
+The default masterdir is created in the current working directory, i.e. `void-packages/masterdir-<arch>`, where `<arch>` for the default masterdir is is the native xbps architecture.
 
 <a name="remaking-masterdir"></a>
 ### Remaking the masterdir
@@ -477,20 +477,21 @@ Two ways are available to build 32bit packages on x86\_64:
 
 The canonical mode (native) needs a new x86 `masterdir`:
 
-    $ ./xbps-src -m masterdir-x86 binary-bootstrap i686
-    $ ./xbps-src -m masterdir-x86 ...
+    $ ./xbps-src -A i686 binary-bootstrap
+    $ ./xbps-src -A i686 ...
 
 <a name="building-for-musl"></a>
 ### Building packages natively for the musl C library
 
-Canonical way of building packages for same architecture but different C library is through dedicated masterdir.
+The canonical way of building packages for same architecture but different C library is through a dedicated masterdir by using the host architecture flag `-A`.
 To build for x86_64-musl on glibc x86_64 system, prepare a new masterdir with the musl packages:
 
-    $ ./xbps-src -m masterdir-x86_64-musl binary-bootstrap x86_64-musl
+    $ ./xbps-src -A x86_64-musl binary-bootstrap
 
+This will create and bootstrap a new masterdir called `masterdir-x86_64-musl` that will be used when `-A x86_64-musl` is specified.
 Your new masterdir is now ready to build packages natively for the musl C library:
 
-    $ ./xbps-src -m masterdir-x86_64-musl pkg ...
+    $ ./xbps-src -A x86_64-musl pkg ...
 
 <a name="building-base-system"></a>
 ### Building void base-system from scratch