[PR PATCH] etc/conf: implement XBPS_MIRROR for remote repos

[PR PATCH] etc/conf: implement XBPS_MIRROR for remote repos

[tornaria]

[-- Attachment #1: Type: text/plain, Size: 978 bytes --]

There is a new pull request by tornaria against master on the void-packages repository

https://github.com/tornaria/void-packages xbps-mirror
https://github.com/void-linux/void-packages/pull/34398

etc/conf: implement XBPS_MIRROR for remote repos
Setting this variable will make xbps-src use an alternative mirror for remote repositories.

Using this variable one can change the remote repository for all architectures in a single setting and without the need to modify `etc/xbps.d/repos-remote*.conf`. This is much more convenient as it allows changing remote repos without a dirty worktree.

To use just add a line like the following to `etc/conf`:

    XBPS_MIRROR=https://repo-us.voidlinux.org/

We also disable `00-repository-main.conf` for cross so we don't use the remote repo from the xbps package.

#### Testing the changes
- I tested the changes in this PR: **YES**

A patch file from https://github.com/void-linux/void-packages/pull/34398.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-xbps-mirror-34398.patch --]
[-- Type: text/x-diff, Size: 5556 bytes --]

From 712fba5594ec6e074a33e59843020a64fbb6fa1e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Gonzalo=20Tornar=C3=ADa?= <tornaria@cmat.edu.uy>
Date: Sun, 5 Dec 2021 20:14:14 -0300
Subject: [PATCH] etc/conf: implement XBPS_MIRROR for remote repos

Setting this variable will make xbps-src use an alternative mirror for
remote repositories.

Using this variable one can change the remote repository for all
architectures in a single setting and without the need to modify
`etc/xbps.d/repos-remote*.conf`. This is much more convenient as it
allows changing remote repos without a dirty worktree.

To use just add a line like the following to `etc/conf`:

    XBPS_MIRROR=https://repo-us.voidlinux.org/

We also disable 00-repository-main.conf for cross so we don't use the
remote repo from the xbps package.
---
 common/xbps-src/shutils/chroot.sh | 29 +++++++++++++++++++----------
 etc/defaults.conf                 |  9 +++++++--
 2 files changed, 26 insertions(+), 12 deletions(-)

diff --git a/common/xbps-src/shutils/chroot.sh b/common/xbps-src/shutils/chroot.sh
index 53e3d4a4c70f..99ac57415e2a 100644
--- a/common/xbps-src/shutils/chroot.sh
+++ b/common/xbps-src/shutils/chroot.sh
@@ -237,26 +237,33 @@ chroot_sync_repodata() {
             $confdir/12-repository-local-multilib.conf
     fi
 
+    # mirror_sed is a sed script: nop by default
+    local mirror_sed
+    if [ -n "$XBPS_MIRROR" ]; then
+        # when XBPS_MIRROR is set, mirror_sed rewrites remote repos
+        mirror_sed="s|^repository=http.*/current|repository=${XBPS_MIRROR%/}/current|"
+    fi
+
     if [ "$XBPS_SKIP_REMOTEREPOS" ]; then
         rm -f $confdir/*remote*
     else
         if [ -s "${XBPS_DISTDIR}/etc/xbps.d/repos-remote-${XBPS_MACHINE}.conf" ]; then
             # If per-architecture base remote repo config exists, use that
-            install -Dm644 ${XBPS_DISTDIR}/etc/xbps.d/repos-remote-${XBPS_MACHINE}.conf \
-                $confdir/20-repository-remote.conf
+            sed -e "$mirror_sed" ${XBPS_DISTDIR}/etc/xbps.d/repos-remote-${XBPS_MACHINE}.conf \
+                > $confdir/20-repository-remote.conf
         else
             # Otherwise use generic base for musl or glibc
             local suffix=
             case "$XBPS_MACHINE" in
                 *-musl) suffix="-musl";;
             esac
-            install -Dm644 ${XBPS_DISTDIR}/etc/xbps.d/repos-remote${suffix}.conf \
-                $confdir/20-repository-remote.conf
+            sed -e "$mirror_sed" ${XBPS_DISTDIR}/etc/xbps.d/repos-remote${suffix}.conf \
+                > $confdir/20-repository-remote.conf
         fi
         # Install multilib conf for remote repos if it exists for the architecture
         if [ -s "${XBPS_DISTDIR}/etc/xbps.d/repos-remote-${XBPS_MACHINE}-multilib.conf" ]; then
-            install -Dm644 ${XBPS_DISTDIR}/etc/xbps.d/repos-remote-${XBPS_MACHINE}-multilib.conf \
-                $confdir/22-repository-remote-multilib.conf
+            sed -e "$mirror_sed" ${XBPS_DISTDIR}/etc/xbps.d/repos-remote-${XBPS_MACHINE}-multilib.conf \
+                > $confdir/22-repository-remote-multilib.conf
         fi
     fi
 
@@ -266,6 +273,8 @@ chroot_sync_repodata() {
     if [ -n "$XBPS_CROSS_BUILD" ]; then
         rm -rf $XBPS_MASTERDIR/$XBPS_CROSS_BASE/etc/xbps.d
         mkdir -p $XBPS_MASTERDIR/$XBPS_CROSS_BASE/etc/xbps.d
+        # Disable 00-repository-main.conf from share/xbps.d (part of xbps)
+        ln -s /dev/null $XBPS_MASTERDIR/$XBPS_CROSS_BASE/etc/xbps.d/00-repository-main.conf
         # copy xbps.d files from host for local repos
         cp ${XBPS_MASTERDIR}/etc/xbps.d/*local*.conf \
             $XBPS_MASTERDIR/$XBPS_CROSS_BASE/etc/xbps.d
@@ -274,15 +283,15 @@ chroot_sync_repodata() {
         else
             # Same general logic as above, just into cross root, and no multilib
             if [ -s "${XBPS_DISTDIR}/etc/xbps.d/repos-remote-${XBPS_TARGET_MACHINE}.conf" ]; then
-                install -Dm644 ${XBPS_DISTDIR}/etc/xbps.d/repos-remote-${XBPS_TARGET_MACHINE}.conf \
-                    $crossconfdir/20-repository-remote.conf
+                sed -e "$mirror_sed" ${XBPS_DISTDIR}/etc/xbps.d/repos-remote-${XBPS_TARGET_MACHINE}.conf \
+                    > $crossconfdir/20-repository-remote.conf
             else
                 local suffix=
                 case "$XBPS_TARGET_MACHINE" in
                     *-musl) suffix="-musl"
                 esac
-                install -Dm644 ${XBPS_DISTDIR}/etc/xbps.d/repos-remote${suffix}.conf \
-                    $crossconfdir/20-repository-remote.conf
+                sed -e "$mirror_sed" ${XBPS_DISTDIR}/etc/xbps.d/repos-remote${suffix}.conf \
+                    > $crossconfdir/20-repository-remote.conf
             fi
         fi
 
diff --git a/etc/defaults.conf b/etc/defaults.conf
index 58369e95cbcd..76cea45ddea2 100644
--- a/etc/defaults.conf
+++ b/etc/defaults.conf
@@ -14,13 +14,18 @@
 # rely on this to work properly.
 #
 
+# [OPTIONAL]
+# Use an alternative mirror for remote repositories. This is more
+# convenient than modifying etc/xbps.d/repos-remote*.conf.
+#
+#XBPS_MIRROR=https://repo-us.voidlinux.org/
+
 # [OPTIONAL]
 # Enable optional arguments to xbps-install(1) for the host system.
 # Currently used in the 'binary-bootstrap' and 'bootstrap-update' targets.
 #
 # NOTE: local repositories are handled automatically by xbps-src,
-# but you can modify the default remote repositories at
-# 'etc/xbps.d/repos-remote*.conf'
+# and remote repositories can be changed by setting XBPS_MIRROR
 #
 #XBPS_INSTALL_ARGS=""
 

Re: [PR PATCH] [Updated] etc/conf: implement XBPS_MIRROR for remote repos

[tornaria]

[-- Attachment #1: Type: text/plain, Size: 983 bytes --]

There is an updated pull request by tornaria against master on the void-packages repository

https://github.com/tornaria/void-packages xbps-mirror
https://github.com/void-linux/void-packages/pull/34398

etc/conf: implement XBPS_MIRROR for remote repos
Setting this variable will make xbps-src use an alternative mirror for remote repositories.

Using this variable one can change the remote repository for all architectures in a single setting and without the need to modify `etc/xbps.d/repos-remote*.conf`. This is much more convenient as it allows changing remote repos without a dirty worktree.

To use just add a line like the following to `etc/conf`:

    XBPS_MIRROR=https://repo-us.voidlinux.org/

We also disable `00-repository-main.conf` for cross so we don't use the remote repo from the xbps package.

#### Testing the changes
- I tested the changes in this PR: **YES**

A patch file from https://github.com/void-linux/void-packages/pull/34398.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-xbps-mirror-34398.patch --]
[-- Type: text/x-diff, Size: 5560 bytes --]

From 46210b12e9de4e6e2e3ce05f7d6cf214eefc146a Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Gonzalo=20Tornar=C3=ADa?= <tornaria@cmat.edu.uy>
Date: Sun, 5 Dec 2021 20:14:14 -0300
Subject: [PATCH] etc/conf: implement XBPS_MIRROR for remote repos

Setting this variable will make xbps-src use an alternative mirror for
remote repositories.

Using this variable one can change the remote repository for all
architectures in a single setting and without the need to modify
`etc/xbps.d/repos-remote*.conf`. This is much more convenient as it
allows changing remote repos without a dirty worktree.

To use just add a line like the following to `etc/conf`:

    XBPS_MIRROR=https://repo-us.voidlinux.org/current

We also disable 00-repository-main.conf for cross so we don't use the
remote repo from the xbps package.
---
 common/xbps-src/shutils/chroot.sh | 29 +++++++++++++++++++----------
 etc/defaults.conf                 |  9 +++++++--
 2 files changed, 26 insertions(+), 12 deletions(-)

diff --git a/common/xbps-src/shutils/chroot.sh b/common/xbps-src/shutils/chroot.sh
index 53e3d4a4c70f..a7100d4ea56f 100644
--- a/common/xbps-src/shutils/chroot.sh
+++ b/common/xbps-src/shutils/chroot.sh
@@ -237,26 +237,33 @@ chroot_sync_repodata() {
             $confdir/12-repository-local-multilib.conf
     fi
 
+    # mirror_sed is a sed script: nop by default
+    local mirror_sed
+    if [ -n "$XBPS_MIRROR" ]; then
+        # when XBPS_MIRROR is set, mirror_sed rewrites remote repos
+        mirror_sed="s|^repository=http.*/current|repository=${XBPS_MIRROR}|"
+    fi
+
     if [ "$XBPS_SKIP_REMOTEREPOS" ]; then
         rm -f $confdir/*remote*
     else
         if [ -s "${XBPS_DISTDIR}/etc/xbps.d/repos-remote-${XBPS_MACHINE}.conf" ]; then
             # If per-architecture base remote repo config exists, use that
-            install -Dm644 ${XBPS_DISTDIR}/etc/xbps.d/repos-remote-${XBPS_MACHINE}.conf \
-                $confdir/20-repository-remote.conf
+            sed -e "$mirror_sed" ${XBPS_DISTDIR}/etc/xbps.d/repos-remote-${XBPS_MACHINE}.conf \
+                > $confdir/20-repository-remote.conf
         else
             # Otherwise use generic base for musl or glibc
             local suffix=
             case "$XBPS_MACHINE" in
                 *-musl) suffix="-musl";;
             esac
-            install -Dm644 ${XBPS_DISTDIR}/etc/xbps.d/repos-remote${suffix}.conf \
-                $confdir/20-repository-remote.conf
+            sed -e "$mirror_sed" ${XBPS_DISTDIR}/etc/xbps.d/repos-remote${suffix}.conf \
+                > $confdir/20-repository-remote.conf
         fi
         # Install multilib conf for remote repos if it exists for the architecture
         if [ -s "${XBPS_DISTDIR}/etc/xbps.d/repos-remote-${XBPS_MACHINE}-multilib.conf" ]; then
-            install -Dm644 ${XBPS_DISTDIR}/etc/xbps.d/repos-remote-${XBPS_MACHINE}-multilib.conf \
-                $confdir/22-repository-remote-multilib.conf
+            sed -e "$mirror_sed" ${XBPS_DISTDIR}/etc/xbps.d/repos-remote-${XBPS_MACHINE}-multilib.conf \
+                > $confdir/22-repository-remote-multilib.conf
         fi
     fi
 
@@ -266,6 +273,8 @@ chroot_sync_repodata() {
     if [ -n "$XBPS_CROSS_BUILD" ]; then
         rm -rf $XBPS_MASTERDIR/$XBPS_CROSS_BASE/etc/xbps.d
         mkdir -p $XBPS_MASTERDIR/$XBPS_CROSS_BASE/etc/xbps.d
+        # Disable 00-repository-main.conf from share/xbps.d (part of xbps)
+        ln -s /dev/null $XBPS_MASTERDIR/$XBPS_CROSS_BASE/etc/xbps.d/00-repository-main.conf
         # copy xbps.d files from host for local repos
         cp ${XBPS_MASTERDIR}/etc/xbps.d/*local*.conf \
             $XBPS_MASTERDIR/$XBPS_CROSS_BASE/etc/xbps.d
@@ -274,15 +283,15 @@ chroot_sync_repodata() {
         else
             # Same general logic as above, just into cross root, and no multilib
             if [ -s "${XBPS_DISTDIR}/etc/xbps.d/repos-remote-${XBPS_TARGET_MACHINE}.conf" ]; then
-                install -Dm644 ${XBPS_DISTDIR}/etc/xbps.d/repos-remote-${XBPS_TARGET_MACHINE}.conf \
-                    $crossconfdir/20-repository-remote.conf
+                sed -e "$mirror_sed" ${XBPS_DISTDIR}/etc/xbps.d/repos-remote-${XBPS_TARGET_MACHINE}.conf \
+                    > $crossconfdir/20-repository-remote.conf
             else
                 local suffix=
                 case "$XBPS_TARGET_MACHINE" in
                     *-musl) suffix="-musl"
                 esac
-                install -Dm644 ${XBPS_DISTDIR}/etc/xbps.d/repos-remote${suffix}.conf \
-                    $crossconfdir/20-repository-remote.conf
+                sed -e "$mirror_sed" ${XBPS_DISTDIR}/etc/xbps.d/repos-remote${suffix}.conf \
+                    > $crossconfdir/20-repository-remote.conf
             fi
         fi
 
diff --git a/etc/defaults.conf b/etc/defaults.conf
index 58369e95cbcd..e3876cdc7d28 100644
--- a/etc/defaults.conf
+++ b/etc/defaults.conf
@@ -14,13 +14,18 @@
 # rely on this to work properly.
 #
 
+# [OPTIONAL]
+# Use an alternative mirror for remote repositories. This is more
+# convenient than modifying etc/xbps.d/repos-remote*.conf.
+#
+#XBPS_MIRROR=https://repo-us.voidlinux.org/current
+
 # [OPTIONAL]
 # Enable optional arguments to xbps-install(1) for the host system.
 # Currently used in the 'binary-bootstrap' and 'bootstrap-update' targets.
 #
 # NOTE: local repositories are handled automatically by xbps-src,
-# but you can modify the default remote repositories at
-# 'etc/xbps.d/repos-remote*.conf'
+# and remote repositories can be changed by setting XBPS_MIRROR
 #
 #XBPS_INSTALL_ARGS=""
 

Re: etc/conf: implement XBPS_MIRROR for remote repos

[tornaria]

[-- Attachment #1: Type: text/plain, Size: 337 bytes --]

New comment by tornaria on void-packages repository

https://github.com/void-linux/void-packages/pull/34398#issuecomment-986386651

Comment:
Force push: based on irc feedback, the new version doesn't force `current` to the mirror url so now a possible configuration could  be
```
XBPS_MIRROR=https://repo-us.voidlinux.org/current
```

Re: etc/conf: implement XBPS_MIRROR for remote repos

[tornaria]

[-- Attachment #1: Type: text/plain, Size: 1068 bytes --]

New comment by tornaria on void-packages repository

https://github.com/void-linux/void-packages/pull/34398#issuecomment-986389904

Comment:
Note: this allows to use one of the standard mirrors since it expects the directory layout for different archs to be the same as the main repo. Implementing this would need more variables and IMO it would be a mess.

For more fine-grained control of the repos without disturbing the git worktree my suggestion would be to postpone the block
```
    if [ -d $XBPS_DISTDIR/etc/xbps.d/custom ]; then
        mkdir -p $confdir $crossconfdir
        cp -f $XBPS_DISTDIR/etc/xbps.d/custom/*.conf $confdir
        cp -f $XBPS_DISTDIR/etc/xbps.d/custom/*.conf $crossconfdir
    fi
```
to later, right before the line `# Copy xbps repository keys to the masterdir.`

In this way the all the conf files (`20-repository-remote.conf`, etc) will be generated first, and then we get a chance to overwrite them with custom files. Right now is the other way around and there's no way to eliminate the configuration in these files.

Re: [PR PATCH] [Merged]: etc/conf: implement XBPS_MIRROR for remote repos

[Chocimier]

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]

There's a merged pull request on the void-packages repository

etc/conf: implement XBPS_MIRROR for remote repos
https://github.com/void-linux/void-packages/pull/34398

Description:
Setting this variable will make xbps-src use an alternative mirror for remote repositories.

Using this variable one can change the remote repository for all architectures in a single setting and without the need to modify `etc/xbps.d/repos-remote*.conf`. This is much more convenient as it allows changing remote repos without a dirty worktree.

To use just add a line like the following to `etc/conf`:

    XBPS_MIRROR=https://repo-us.voidlinux.org/

We also disable `00-repository-main.conf` for cross so we don't use the remote repo from the xbps package.

#### Testing the changes
- I tested the changes in this PR: **YES**