Github messages for voidlinux
 help / color / mirror / Atom feed
* [PR PATCH] common: retry ftp download during CI
@ 2020-01-29 22:38 voidlinux-github
  2020-02-11 22:05 ` [PR PATCH] [Merged]: " voidlinux-github
  2020-02-12 18:08 ` voidlinux-github
  0 siblings, 2 replies; 3+ messages in thread
From: voidlinux-github @ 2020-01-29 22:38 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 542 bytes --]

There is a new pull request by Chocimier against master on the void-packages repository

https://github.com/Chocimier/void-packages-org file
https://github.com/void-linux/void-packages/pull/18646

common: retry ftp download during CI
Build will [eventually](https://travis-ci.org/Chocimier/void-packages-org/builds/643581137) pass, but may take too long for now, as retry takes 5 minutes. Patch for xbps to customize timeout may decrease needed time.

A patch file from https://github.com/void-linux/void-packages/pull/18646.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-file-18646.patch --]
[-- Type: text/x-diff, Size: 2826 bytes --]

From ac64046408d324262bdc027cdff451364c1c2552 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Piotr=20W=C3=B3jcik?= <chocimier@tlen.pl>
Date: Wed, 29 Jan 2020 23:28:40 +0100
Subject: [PATCH] common: retry ftp download during CI

---
 common/hooks/do-fetch/00-distfiles.sh | 19 +++++++++++++++----
 common/travis/build.sh                |  2 ++
 common/xbps-src/shutils/chroot.sh     |  1 +
 3 files changed, 18 insertions(+), 4 deletions(-)

diff --git a/common/hooks/do-fetch/00-distfiles.sh b/common/hooks/do-fetch/00-distfiles.sh
index 922245fcd21..528eab8896c 100644
--- a/common/hooks/do-fetch/00-distfiles.sh
+++ b/common/hooks/do-fetch/00-distfiles.sh
@@ -198,7 +198,7 @@ try_mirrors() {
 
 hook() {
 	local srcdir="$XBPS_SRCDISTDIR/$pkgname-$version"
-	local dfcount=0 dfgood=0 errors=0
+	local dfcount=0 dfgood=0 errors=0 max_retries
 
 	if [ ! -d "$srcdir" ]; then
 		mkdir -p -m775 "$srcdir"
@@ -266,10 +266,21 @@ hook() {
 			try_mirrors $curfile $distfile $dfcount $pkgname-$version $f
 		fi
 		# If distfile does not exist, download it from the original location.
-		if [ ! -f "$distfile" ]; then
-			msg_normal "$pkgver: fetching distfile '$curfile'...\n"
-			flock "${distfile}.part" $fetch_cmd "$f"
+		if [[ "$FTP_RETRIES" && "${f}" =~ ^ftp:// ]]; then
+			max_retries="$FTP_RETRIES"
+		else
+			max_retries=1
 		fi
+		for retry in $(seq 1 1 $max_retries); do
+			if [ ! -f "$distfile" ]; then
+				if [ "$retry" == 1 ]; then
+					msg_normal "$pkgver: fetching distfile '$curfile'...\n"
+				else
+					msg_normal "$pkgver: fetch attempt $retry of $max_retries...\n"
+				fi
+				flock "${distfile}.part" $fetch_cmd "$f"
+			fi
+		done
 		if [ ! -f "$distfile" ]; then
 			msg_error "$pkgver: failed to fetch $curfile.\n"
 		fi
diff --git a/common/travis/build.sh b/common/travis/build.sh
index 7732211a59e..1c45910ea69 100755
--- a/common/travis/build.sh
+++ b/common/travis/build.sh
@@ -13,6 +13,8 @@ if [ -r /proc/cpuinfo ]; then
         NPROCS=$(grep ^proc /proc/cpuinfo|wc -l)
 fi
 
+export FTP_RETRIES=10
+
 for pkg in ${PKGS}; do
 	/hostrepo/xbps-src -j$NPROCS -H "$HOME"/hostdir $arch pkg "$pkg"
 	[ $? -eq 1 ] && exit 1
diff --git a/common/xbps-src/shutils/chroot.sh b/common/xbps-src/shutils/chroot.sh
index 228dd2e0399..4f543bf2a8e 100644
--- a/common/xbps-src/shutils/chroot.sh
+++ b/common/xbps-src/shutils/chroot.sh
@@ -173,6 +173,7 @@ chroot_handler() {
             ${SOCKS_PROXY:+SOCKS_PROXY="${SOCKS_PROXY}"} \
             ${NO_PROXY:+NO_PROXY="${NO_PROXY}"} \
             ${HTTP_PROXY_AUTH:+HTTP_PROXY_AUTH="${HTTP_PROXY_AUTH}"} \
+            ${FTP_RETRIES:+FTP_RETRIES="${FTP_RETRIES}"} \
             SOURCE_DATE_EPOCH="$SOURCE_DATE_EPOCH" \
             XBPS_ALLOW_CHROOT_BREAKOUT="$XBPS_ALLOW_CHROOT_BREAKOUT" \
             $XBPS_COMMONDIR/chroot-style/${XBPS_CHROOT_CMD:=uunshare}.sh \

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PR PATCH] [Merged]: common: retry ftp download during CI
  2020-01-29 22:38 [PR PATCH] common: retry ftp download during CI voidlinux-github
@ 2020-02-11 22:05 ` voidlinux-github
  2020-02-12 18:08 ` voidlinux-github
  1 sibling, 0 replies; 3+ messages in thread
From: voidlinux-github @ 2020-02-11 22:05 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 386 bytes --]

There's a merged pull request on the void-packages repository

common: retry ftp download during CI
https://github.com/void-linux/void-packages/pull/18646

Description:
Build will [eventually](https://travis-ci.org/Chocimier/void-packages-org/builds/643581137) pass, but may take too long for now, as retry takes 5 minutes. Patch for xbps to customize timeout may decrease needed time.

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: common: retry ftp download during CI
  2020-01-29 22:38 [PR PATCH] common: retry ftp download during CI voidlinux-github
  2020-02-11 22:05 ` [PR PATCH] [Merged]: " voidlinux-github
@ 2020-02-12 18:08 ` voidlinux-github
  1 sibling, 0 replies; 3+ messages in thread
From: voidlinux-github @ 2020-02-12 18:08 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 439 bytes --]

New comment by Chocimier on void-packages repository

https://github.com/void-linux/void-packages/pull/18646#issuecomment-585339505

Comment:
> I see you've ignored my comment...

@xtraeme, which comment?

> Why does it need to be specific to FTP and travis?

This change aims to handle specific configuration of network for travis making ftp connection there poor. https://blog.travis-ci.com/2018-07-23-the-tale-of-ftp-at-travis-ci

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2020-02-12 18:08 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-01-29 22:38 [PR PATCH] common: retry ftp download during CI voidlinux-github
2020-02-11 22:05 ` [PR PATCH] [Merged]: " voidlinux-github
2020-02-12 18:08 ` voidlinux-github

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).