[PR PATCH] Add Option to use Distfiles Mirror as a Fallback

[PR PATCH] Add Option to use Distfiles Mirror as a Fallback

[Calandracas606]

[-- Attachment #1: Type: text/plain, Size: 1919 bytes --]

There is a new pull request by Calandracas606 against master on the void-packages repository

https://github.com/Calandracas606/void-packages fallback_distfiles
https://github.com/void-linux/void-packages/pull/50577

Add Option to use Distfiles Mirror as a Fallback
This adds an optional configuration to add a fallback distfiles mirror

The idea is that normally, distfiles should be fetched from the urls in
the template. However, occasionally the urls have rotted, or are simply
down temporairily, in which case the fetch will fail. A workaround to
this has been to use the $XBPS_DISTFILES_MIRROR option, however the
downside to this is that it costs bandwidth, and 99% of the time isn't
needed.

This adds the $XBPS_DISTFILES_FALLBACK option, which can be set to an
addiontional list of mirrors which will be attempted after both
$XBPS_DISTFILES_MIRROR, and the urls in the template have failed to
fetch the distfiles

<!-- Uncomment relevant sections and delete options which are not applicable -->

#### Testing the changes
- I tested the changes in this PR: **YES**

<!--
#### New package
- This new package conforms to the [package requirements](https://github.com/void-linux/void-packages/blob/master/CONTRIBUTING.md#package-requirements): **YES**|**NO**
-->

<!-- Note: If the build is likely to take more than 2 hours, please add ci skip tag as described in
https://github.com/void-linux/void-packages/blob/master/CONTRIBUTING.md#continuous-integration
and test at least one native build and, if supported, at least one cross build.
Ignore this section if this PR is not skipping CI.
-->
<!--
#### Local build testing
- I built this PR locally for my native architecture, (ARCH-LIBC)
- I built this PR locally for these architectures (if supported. mark crossbuilds):
  - aarch64-musl
  - armv7l
  - armv6l-musl
-->


A patch file from https://github.com/void-linux/void-packages/pull/50577.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-fallback_distfiles-50577.patch --]
[-- Type: text/x-diff, Size: 4291 bytes --]

From b1faeabb419a3f51c685f97db6de31b61570b61b Mon Sep 17 00:00:00 2001
From: Daniel Martinez <danielmartinez@cock.li>
Date: Tue, 28 May 2024 15:31:49 -0400
Subject: [PATCH] Add Option to use Distfiles Mirror as a Fallback

This adds an optional configuration to add a fallback distfiles mirror

The idea is that normally, distfiles should be fetched from the urls in
the template. However, occasionally the urls have rotted, or are simply
down temporairily, in which case the fetch will fail. A workaround to
this has been to use the $XBPS_DISTFILES_MIRROR option, however the
downside to this is that it costs bandwidth, and 99% of the time isn't
needed.

This adds the $XBPS_DISTFILES_FALLBACK option, which can be set to an
addiontional list of mirrors which will be attempted after both
$XBPS_DISTFILES_MIRROR, and the urls in the template have failed to
fetch the distfiles
---
 common/hooks/do-fetch/00-distfiles.sh | 16 +++++++++++-----
 etc/defaults.conf                     |  5 +++++
 xbps-src                              | 10 +++++++---
 3 files changed, 23 insertions(+), 8 deletions(-)

diff --git a/common/hooks/do-fetch/00-distfiles.sh b/common/hooks/do-fetch/00-distfiles.sh
index 1adb86a5167823..ec4c3b07a281a8 100644
--- a/common/hooks/do-fetch/00-distfiles.sh
+++ b/common/hooks/do-fetch/00-distfiles.sh
@@ -128,11 +128,11 @@ link_cksum() {
 }
 
 try_mirrors() {
-	local curfile="$1" distfile="$2" cksum="$3" f="$4"
+	local curfile="$1" distfile="$2" cksum="$3" f="$4" mirror_list="$5"
 	local filesum basefile mirror path scheme good
-	[ -z "$XBPS_DISTFILES_MIRROR" ] && return 1
+	[ -z "$mirror_list" ] && return 1
 	basefile="${f##*/}"
-	for mirror in $XBPS_DISTFILES_MIRROR; do
+	for mirror in $mirror_list; do
 		scheme="file"
 		if [[ $mirror == *://* ]]; then
 			scheme="${mirror%%:/*}"
@@ -286,11 +286,17 @@ hook() {
 		fi
 
 		# If distfile does not exist, download it from a mirror location.
-		if try_mirrors "$curfile" "$distfile" "${_checksums[$i]}" "${_distfiles[$i]}"; then
+		if try_mirrors "$curfile" "$distfile" "${_checksums[$i]}" "${_distfiles[$i]}" "$XBPS_DISTFILES_MIRROR"; then
 			continue
 		fi
 
-		if ! try_urls "$curfile"; then
+		# Try the urls in the template
+		if try_urls "$curfile"; then
+			continue
+		fi
+
+		# finally, try the fallback mirrors
+		if ! try_mirrors "$curfile" "$distfile" "${_checksums[$i]}" "${_distfiles[$i]}" "$XBPS_DISTFILES_FALLBACK"; then
 			msg_error "$pkgver: failed to fetch '$curfile'.\n"
 		fi
 	done
diff --git a/etc/defaults.conf b/etc/defaults.conf
index f70e4ea3d0067e..f2e1b7f5c1ce4d 100644
--- a/etc/defaults.conf
+++ b/etc/defaults.conf
@@ -165,3 +165,8 @@ XBPS_SUCMD="sudo /bin/sh -c"
 # This can also be set or exported as a regular environment variable.
 #
 #XBPS_UPDATE_CHECK_VERBOSE=yes
+
+# [OPTIONAL]
+# Use this mirror as a fallback for when the distfiles specified in the template are not found.
+#
+#XBPS_DISTFILES_FALLBACK="https://sources.voidlinux.org/"
diff --git a/xbps-src b/xbps-src
index f8c21b3b92738d..c4fa3ad8714f62 100755
--- a/xbps-src
+++ b/xbps-src
@@ -343,10 +343,11 @@ read_pkg() {
 }
 
 setup_distfiles_mirror() {
+    local mirror_list="$1"
     local mirror scheme path
 
     # Scheme file:// mirror locations only work with uchroot
-    for mirror in $XBPS_DISTFILES_MIRROR; do
+    for mirror in $mirror_list; do
         scheme="file"
         if [[ "$mirror" == *://* ]]; then
             scheme="${mirror%%://*}"
@@ -696,7 +697,7 @@ export XBPS_SHUTILSDIR XBPS_CROSSPFDIR XBPS_TRIGGERSDIR \
     XBPS_LIBEXECDIR XBPS_DISTDIR XBPS_DISTFILES_MIRROR XBPS_ALLOW_RESTRICTED \
     XBPS_USE_GIT_COMMIT_DATE XBPS_PKG_COMPTYPE XBPS_REPO_COMPTYPE \
     XBPS_BUILDHELPERDIR XBPS_USE_BUILD_MTIME XBPS_BUILD_ENVIRONMENT \
-    XBPS_PRESERVE_PKGS XBPS_IGNORE_BROKENNESS
+    XBPS_PRESERVE_PKGS XBPS_IGNORE_BROKENNESS XBPS_DISTFILES_FALLBACK
 
 for i in REPOSITORY DESTDIR BUILDDIR SRCDISTDIR; do
     eval val="\$XBPS_$i"
@@ -760,7 +761,10 @@ done
 if [ -z "$IN_CHROOT" ]; then
     trap 'exit_func' INT TERM
     if [ -n "$XBPS_DISTFILES_MIRROR" ]; then
-        setup_distfiles_mirror
+        setup_distfiles_mirror "$XBPS_DISTFILES_MIRROR"
+    fi
+    if [ -n "$XBPS_DISTFILES_FALLBACK" ]; then
+        setup_distfiles_mirror "$XBPS_DISTFILES_FALLBACK"
     fi
 fi
 

Re: Add Option to use Distfiles Mirror as a Fallback

[Calandracas606]

[-- Attachment #1: Type: text/plain, Size: 453 bytes --]

New comment by Calandracas606 on void-packages repository

https://github.com/void-linux/void-packages/pull/50577#issuecomment-2136291664

Comment:
this PR was tested using the `fbreader` package

url in the template is broken: 
- http://www.fbreader.org/files/desktop/fbreader-sources-0.99.4.tgz


but the distfiles are available at `https://sources.voidlinux.org/` : 
- https://sources.voidlinux.org/fbreader-0.99.4/fbreader-sources-0.99.4.tgz

Re: Add Option to use Distfiles Mirror as a Fallback

[github-actions]

[-- Attachment #1: Type: text/plain, Size: 305 bytes --]

New comment by github-actions[bot] on void-packages repository

https://github.com/void-linux/void-packages/pull/50577#issuecomment-2311417149

Comment:
Pull Requests become stale 90 days after last activity and are closed 14 days after that.  If this pull request is still relevant bump it or assign it.

Re: Add Option to use Distfiles Mirror as a Fallback

[Calandracas606]

[-- Attachment #1: Type: text/plain, Size: 153 bytes --]

New comment by Calandracas606 on void-packages repository

https://github.com/void-linux/void-packages/pull/50577#issuecomment-2313934923

Comment:
bump