* [PR PATCH] move update-check out of xbps-src
@ 2021-06-22 1:11 sgn
2021-06-22 2:16 ` the-maldridge
` (7 more replies)
0 siblings, 8 replies; 9+ messages in thread
From: sgn @ 2021-06-22 1:11 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 831 bytes --]
There is a new pull request by sgn against master on the void-packages repository
https://github.com/sgn/void-packages move-update-check-out-of-xbps-src
https://github.com/void-linux/void-packages/pull/31602
move update-check out of xbps-src
Continue discussion from #31546
It's better to rewrite in a real programming language, however this is enough for now.
TODO: check only distfiles.
The `eval` step should be removed when we can remove all variable except `pkgname` and `version` from `distfiles`, and merge the first `distfiles` to the same line of `distfiles=`
@void-linux/pkg-committers
@the-maldridge moving out of `xbps-src`, we can drop the weird `XBPS_ALLOW_CHROOT_BREAKOUT=yes` from `update-check` container.
A patch file from https://github.com/void-linux/void-packages/pull/31602.patch is attached
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-move-update-check-out-of-xbps-src-31602.patch --]
[-- Type: text/x-diff, Size: 20488 bytes --]
From 5422dc71da86fbadd5c24f7371944606455019f8 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=C4=90o=C3=A0n=20Tr=E1=BA=A7n=20C=C3=B4ng=20Danh?=
<congdanhqx@gmail.com>
Date: Tue, 22 Jun 2021 08:00:02 +0700
Subject: [PATCH 1/3] common/scripts: add equivalent script for update-check
This script will read necessary variables from template, then calling
the old update_check method.
---
common/scripts/update-check | 46 +++++++++++++++++++++++++++++++++++++
1 file changed, 46 insertions(+)
create mode 100755 common/scripts/update-check
diff --git a/common/scripts/update-check b/common/scripts/update-check
new file mode 100755
index 000000000000..dbbae3428c70
--- /dev/null
+++ b/common/scripts/update-check
@@ -0,0 +1,46 @@
+#!/bin/bash
+# vim: set ts=4 sw=4 et:
+
+die() {
+ printf '%s\n' "$*" >&2
+ exit 1
+}
+
+pkgname=$1
+
+if [ -z "$pkgname" ]; then
+ die "pkgname is required"
+fi
+
+XBPS_DISTDIR="$(cd "${0%/*}/../.." && pwd)"
+XBPS_SRCPKGDIR="$XBPS_DISTDIR/srcpkgs"
+
+if [ ! -f "$XBPS_SRCPKGDIR/$1/template" ]; then
+ die "$pkgname: not found"
+fi
+
+source "$XBPS_DISTDIR/common/environment/setup/misc.sh"
+source "$XBPS_DISTDIR/common/xbps-src/shutils/update_check.sh"
+
+vars="pkgname|version|homepage|distfiles"
+template="$(
+ sed -E -n "
+ /^[a-z_]*($vars)=[^\"']*\$/p
+ /^[a-z_]*($vars)=\"[^\"]*\"\$/p
+ /^[a-z_]*($vars)='[^']*'\$/p
+ /^[a-z_]*($vars)=\"[^\"]*\$/,/^[^\"]*\"\$/p
+ /^[a-z_]*($vars)='[^']*\$/,/^[^']*'\$/p
+ " "$XBPS_SRCPKGDIR/$1/template"
+)"
+
+# Not really safe
+case "$template" in
+ *'$('* | *'`'*)
+ die "command substitution in pkgname|version|homepage|distfiles"
+ ;;
+esac
+
+eval "$template"
+XBPS_TARGET_PKG=$pkgname
+
+update_check
From cde200d193588eee33180b25886190ed2ccec957 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=C4=90o=C3=A0n=20Tr=E1=BA=A7n=20C=C3=B4ng=20Danh?=
<congdanhqx@gmail.com>
Date: Tue, 22 Jun 2021 08:02:05 +0700
Subject: [PATCH 2/3] xbps-src: call external script for update-check
---
xbps-src | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/xbps-src b/xbps-src
index 806c6480f21c..a21d2aa7fb8a 100755
--- a/xbps-src
+++ b/xbps-src
@@ -959,8 +959,7 @@ case "$XBPS_TARGET" in
bulk_update -I
;;
update-check)
- read_pkg ignore-problems
- update_check
+ ${XBPS_COMMONDIR}/scripts/update-check "$XBPS_TARGET_PKG"
;;
update-hash-cache)
update_hash_cache
From d332ba77b6c6940318f887d0e16ebce6103a9a4c Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=C4=90o=C3=A0n=20Tr=E1=BA=A7n=20C=C3=B4ng=20Danh?=
<congdanhqx@gmail.com>
Date: Tue, 22 Jun 2021 08:03:27 +0700
Subject: [PATCH 3/3] update-check: move the real check to external script
The change is best-viewed with git diff --color-moved
---
common/scripts/update-check | 208 +++++++++++++++++++++++-
common/xbps-src/shutils/update_check.sh | 208 ------------------------
2 files changed, 207 insertions(+), 209 deletions(-)
delete mode 100644 common/xbps-src/shutils/update_check.sh
diff --git a/common/scripts/update-check b/common/scripts/update-check
index dbbae3428c70..f8be72b6265a 100755
--- a/common/scripts/update-check
+++ b/common/scripts/update-check
@@ -20,7 +20,213 @@ if [ ! -f "$XBPS_SRCPKGDIR/$1/template" ]; then
fi
source "$XBPS_DISTDIR/common/environment/setup/misc.sh"
-source "$XBPS_DISTDIR/common/xbps-src/shutils/update_check.sh"
+
+update_check() {
+ local i p url pkgurlname rx found_version consider
+ local update_override=$XBPS_SRCPKGDIR/$XBPS_TARGET_PKG/update
+ local original_pkgname=$pkgname
+ local urlpfx urlsfx
+ local -A fetchedurls
+
+ if [ -r $update_override ]; then
+ . $update_override
+ if [ "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+ echo "using $XBPS_TARGET_PKG/update overrides" 1>&2
+ fi
+ fi
+
+ if ! type curl >/dev/null 2>&1; then
+ echo "ERROR: cannot find \`curl' executable!"
+ return 1
+ fi
+
+ export LC_ALL=C
+
+ if [ -z "$site" ]; then
+ case "$distfiles" in
+ # only consider versions those exist in ftp.gnome.org
+ *ftp.gnome.org*) ;;
+ *)
+ printf '%s\n' "$homepage" ;;
+ esac
+ for i in $distfiles; do
+ printf '%s\n' "${i%/*}/"
+ done
+ else
+ printf '%s\n' "$site"
+ fi |
+ # filter loop: if version are "folder" name based,
+ # substitute original url by every folder based ones (expand)
+ while IFS= read -r url; do
+ # default case: don't rewrite url
+ printf '%s\n' "$url"
+ if [ "$single_directory" ]; then
+ continue
+ fi
+ rx=
+ urlpfx="${url}"
+ urlsfx=
+ dirpfx=
+ case "$url" in
+ *.voidlinux.*|\
+ *sourceforge.net/sourceforge*|\
+ *code.google.com*|*googlecode*|\
+ *launchpad.net*|\
+ *cpan.*|\
+ *pythonhosted.org*|\
+ *github.com*|\
+ *//gitlab.*|\
+ *bitbucket.org*|\
+ *ftp.gnome.org*|\
+ *kernel.org/pub/linux/kernel/*|\
+ *cran.r-project.org/src/contrib*|\
+ *rubygems.org*|\
+ *crates.io*|\
+ *codeberg.org*|\
+ *hg.sr.ht*|\
+ *git.sr.ht*)
+ continue
+ ;;
+ *)
+ vdpfx=${vdprefix:-"|v|\\Q$pkgname\\E"}
+ vdsfx=${vdsuffix:-"|\\.x"}
+ match=$(grep -Po "^[^/]+//[^/]+(/.+)?/($vdpfx)(?=[-_.0-9]*[0-9](?<!\\Q$pkgname\\E)($vdsfx)/)" <<< "$url")
+ if [ "$?" = 0 ]; then
+ urlpfx="${match%/*}/"
+ dirpfx="${match##*/}"
+ urlsfx="${url#$urlpfx}"
+ urlsfx="${urlsfx#*/}"
+ rx="href=[\"']?(\\Q$urlpfx\\E)?\\.?/?\\K\\Q$dirpfx\\E[-_.0-9]*[0-9]($vdsfx)[\"'/]"
+ fi
+ ;;
+ esac
+ if [ "$rx" ]; then
+ # substitute url if needed
+ if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+ echo "(folder) fetching $urlpfx and scanning with $rx" 1>&2
+ fi
+ skipdirs=
+ curl -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$urlpfx" |
+ grep -Po -i "$rx" |
+ # sort -V places 1.1/ before 1/, but 1A/ before 1.1A/
+ sed -e 's:$:A:' -e 's:/A$:A/:' | sort -Vru | sed -e 's:A/$:/A:' -e 's:A$::' |
+ while IFS= read -r newver; do
+ newurl="${urlpfx}${newver}${urlsfx}"
+ if [ "$newurl" = "$url" ]; then
+ skipdirs=yes
+ fi
+ if [ -z "$skipdirs" ]; then
+ printf '%s\n' "$newurl"
+ fi
+ done
+ fi
+ done |
+ while IFS= read -r url; do
+ rx=
+ if [ -z "$site" ]; then
+ case "$url" in
+ *sourceforge.net/sourceforge*)
+ pkgurlname="$(printf %s "$url" | cut -d/ -f5)"
+ url="https://sourceforge.net/projects/$pkgurlname/rss?limit=200";;
+ *code.google.com*|*googlecode*)
+ url="http://code.google.com/p/$pkgname/downloads/list";;
+ *launchpad.net*)
+ pkgurlname="$(printf %s "$url" | cut -d/ -f4)"
+ url="https://launchpad.net/$pkgurlname/+download";;
+ *cpan.*)
+ pkgname=${pkgname#perl-};;
+ *pythonhosted.org*)
+ pkgname=${pkgname#python-}
+ pkgname=${pkgname#python3-}
+ url="https://pypi.org/simple/$pkgname";;
+ *github.com*)
+ pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+ url="https://github.com/$pkgurlname/tags"
+ rx='/archive/refs/tags/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
+ *//gitlab.*)
+ pkgurlname="$(printf %s "$url" | cut -d/ -f1-5)"
+ url="$pkgurlname/tags"
+ rx='/archive/[^/]+/\Q'"$pkgname"'\E-v?\K[\d.]+(?=\.tar\.gz")';;
+ *bitbucket.org*)
+ pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+ url="https://bitbucket.org/$pkgurlname/downloads"
+ rx='/(get|downloads)/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar)';;
+ *ftp.gnome.org*|*download.gnome.org*)
+ : ${pattern="\Q$pkgname\E-\K(0|[13]\.[0-9]*[02468]|[4-9][0-9]+)\.[0-9.]*[0-9](?=)"}
+ url="https://download.gnome.org/sources/$pkgname/cache.json";;
+ *kernel.org/pub/linux/kernel/*)
+ rx=linux-'\K'${version%.*}'[\d.]+(?=\.tar\.xz)';;
+ *cran.r-project.org/src/contrib*)
+ rx='\b\Q'"${pkgname#R-cran-}"'\E_\K\d+(\.\d+)*(-\d+)?(?=\.tar)';;
+ *rubygems.org*)
+ url="https://rubygems.org/gems/${pkgname#ruby-}"
+ rx='href="/gems/'${pkgname#ruby-}'/versions/\K[\d.]*(?=")' ;;
+ *crates.io*)
+ url="https://crates.io/api/v1/crates/${pkgname#rust-}"
+ rx='/crates/'${pkgname#rust-}'/\K[0-9.]*(?=/download)' ;;
+ *codeberg.org*)
+ pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+ url="https://codeberg.org/$pkgurlname/releases"
+ rx='/archive/\K[\d.]+(?=\.tar\.gz)' ;;
+ *hg.sr.ht*)
+ pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+ url="https://hg.sr.ht/$pkgurlname/tags"
+ rx='/archive/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
+ *git.sr.ht*)
+ pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+ url="https://git.sr.ht/$pkgurlname/refs"
+ rx='/archive/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
+ esac
+ fi
+
+ rx=${pattern:-$rx}
+ rx=${rx:-'(?<!-)\b\Q'"$pkgname"'\E[-_]?((src|source)[-_])?v?\K([^-/_\s]*?\d[^-/_\s]*?)(?=(?:[-_.](?:src|source|orig))?\.(?:[jt]ar|shar|t[bglx]z|tbz2|zip))\b'}
+
+ if [ "${fetchedurls[$url]}" ]; then
+ if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+ echo "already fetched $url" 1>&2
+ fi
+ continue
+ fi
+
+ if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+ echo "fetching $url and scanning with $rx" 1>&2
+ fi
+ curl -H 'Accept: text/html,application/xhtml+xml,application/xml,text/plain,application/rss+xml' -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$url" |
+ grep -Po -i "$rx"
+ fetchedurls[$url]=yes
+ done |
+ tr _ . |
+ sort -Vu |
+ {
+ grep . || echo "NO VERSION found for $original_pkgname" 1>&2
+ } |
+ while IFS= read -r found_version; do
+ if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+ echo "found version $found_version"
+ fi
+ consider=true
+ p="$ignore "
+ while [ -n "$p" ]; do
+ i=${p%% *}
+ p=${p#* }
+ case "$found_version" in
+ $i)
+ consider=false
+ if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+ echo "ignored $found_version due to $i"
+ fi
+ esac
+ done
+ if $consider; then
+ xbps-uhelper cmpver "$original_pkgname-${version}_1" \
+ "$original_pkgname-$(printf %s "$found_version" | tr - .)_1"
+ if [ $? = 255 ]; then
+ echo "${original_pkgname}-${version} -> ${original_pkgname}-${found_version}"
+ fi
+ fi
+ done
+}
vars="pkgname|version|homepage|distfiles"
template="$(
diff --git a/common/xbps-src/shutils/update_check.sh b/common/xbps-src/shutils/update_check.sh
deleted file mode 100644
index fcfbf31759d6..000000000000
--- a/common/xbps-src/shutils/update_check.sh
+++ /dev/null
@@ -1,208 +0,0 @@
-# vim: set ts=4 sw=4 et:
-
-update_check() {
- local i p url pkgurlname rx found_version consider
- local update_override=$XBPS_SRCPKGDIR/$XBPS_TARGET_PKG/update
- local original_pkgname=$pkgname
- local urlpfx urlsfx
- local -A fetchedurls
-
- if [ -r $update_override ]; then
- . $update_override
- if [ "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
- echo "using $XBPS_TARGET_PKG/update overrides" 1>&2
- fi
- fi
-
- if ! type curl >/dev/null 2>&1; then
- echo "ERROR: cannot find \`curl' executable!"
- return 1
- fi
-
- export LC_ALL=C
-
- if [ -z "$site" ]; then
- case "$distfiles" in
- # only consider versions those exist in ftp.gnome.org
- *ftp.gnome.org*) ;;
- *)
- printf '%s\n' "$homepage" ;;
- esac
- for i in $distfiles; do
- printf '%s\n' "${i%/*}/"
- done
- else
- printf '%s\n' "$site"
- fi |
- # filter loop: if version are "folder" name based,
- # substitute original url by every folder based ones (expand)
- while IFS= read -r url; do
- # default case: don't rewrite url
- printf '%s\n' "$url"
- if [ "$single_directory" ]; then
- continue
- fi
- rx=
- urlpfx="${url}"
- urlsfx=
- dirpfx=
- case "$url" in
- *.voidlinux.*|\
- *sourceforge.net/sourceforge*|\
- *code.google.com*|*googlecode*|\
- *launchpad.net*|\
- *cpan.*|\
- *pythonhosted.org*|\
- *github.com*|\
- *//gitlab.*|\
- *bitbucket.org*|\
- *ftp.gnome.org*|\
- *kernel.org/pub/linux/kernel/*|\
- *cran.r-project.org/src/contrib*|\
- *rubygems.org*|\
- *crates.io*|\
- *codeberg.org*|\
- *hg.sr.ht*|\
- *git.sr.ht*)
- continue
- ;;
- *)
- vdpfx=${vdprefix:-"|v|\\Q$pkgname\\E"}
- vdsfx=${vdsuffix:-"|\\.x"}
- match=$(grep -Po "^[^/]+//[^/]+(/.+)?/($vdpfx)(?=[-_.0-9]*[0-9](?<!\\Q$pkgname\\E)($vdsfx)/)" <<< "$url")
- if [ "$?" = 0 ]; then
- urlpfx="${match%/*}/"
- dirpfx="${match##*/}"
- urlsfx="${url#$urlpfx}"
- urlsfx="${urlsfx#*/}"
- rx="href=[\"']?(\\Q$urlpfx\\E)?\\.?/?\\K\\Q$dirpfx\\E[-_.0-9]*[0-9]($vdsfx)[\"'/]"
- fi
- ;;
- esac
- if [ "$rx" ]; then
- # substitute url if needed
- if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
- echo "(folder) fetching $urlpfx and scanning with $rx" 1>&2
- fi
- skipdirs=
- curl -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$urlpfx" |
- grep -Po -i "$rx" |
- # sort -V places 1.1/ before 1/, but 1A/ before 1.1A/
- sed -e 's:$:A:' -e 's:/A$:A/:' | sort -Vru | sed -e 's:A/$:/A:' -e 's:A$::' |
- while IFS= read -r newver; do
- newurl="${urlpfx}${newver}${urlsfx}"
- if [ "$newurl" = "$url" ]; then
- skipdirs=yes
- fi
- if [ -z "$skipdirs" ]; then
- printf '%s\n' "$newurl"
- fi
- done
- fi
- done |
- while IFS= read -r url; do
- rx=
- if [ -z "$site" ]; then
- case "$url" in
- *sourceforge.net/sourceforge*)
- pkgurlname="$(printf %s "$url" | cut -d/ -f5)"
- url="https://sourceforge.net/projects/$pkgurlname/rss?limit=200";;
- *code.google.com*|*googlecode*)
- url="http://code.google.com/p/$pkgname/downloads/list";;
- *launchpad.net*)
- pkgurlname="$(printf %s "$url" | cut -d/ -f4)"
- url="https://launchpad.net/$pkgurlname/+download";;
- *cpan.*)
- pkgname=${pkgname#perl-};;
- *pythonhosted.org*)
- pkgname=${pkgname#python-}
- pkgname=${pkgname#python3-}
- url="https://pypi.org/simple/$pkgname";;
- *github.com*)
- pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
- url="https://github.com/$pkgurlname/tags"
- rx='/archive/refs/tags/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
- *//gitlab.*)
- pkgurlname="$(printf %s "$url" | cut -d/ -f1-5)"
- url="$pkgurlname/tags"
- rx='/archive/[^/]+/\Q'"$pkgname"'\E-v?\K[\d.]+(?=\.tar\.gz")';;
- *bitbucket.org*)
- pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
- url="https://bitbucket.org/$pkgurlname/downloads"
- rx='/(get|downloads)/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar)';;
- *ftp.gnome.org*|*download.gnome.org*)
- : ${pattern="\Q$pkgname\E-\K(0|[13]\.[0-9]*[02468]|[4-9][0-9]+)\.[0-9.]*[0-9](?=)"}
- url="https://download.gnome.org/sources/$pkgname/cache.json";;
- *kernel.org/pub/linux/kernel/*)
- rx=linux-'\K'${version%.*}'[\d.]+(?=\.tar\.xz)';;
- *cran.r-project.org/src/contrib*)
- rx='\b\Q'"${pkgname#R-cran-}"'\E_\K\d+(\.\d+)*(-\d+)?(?=\.tar)';;
- *rubygems.org*)
- url="https://rubygems.org/gems/${pkgname#ruby-}"
- rx='href="/gems/'${pkgname#ruby-}'/versions/\K[\d.]*(?=")' ;;
- *crates.io*)
- url="https://crates.io/api/v1/crates/${pkgname#rust-}"
- rx='/crates/'${pkgname#rust-}'/\K[0-9.]*(?=/download)' ;;
- *codeberg.org*)
- pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
- url="https://codeberg.org/$pkgurlname/releases"
- rx='/archive/\K[\d.]+(?=\.tar\.gz)' ;;
- *hg.sr.ht*)
- pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
- url="https://hg.sr.ht/$pkgurlname/tags"
- rx='/archive/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
- *git.sr.ht*)
- pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
- url="https://git.sr.ht/$pkgurlname/refs"
- rx='/archive/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
- esac
- fi
-
- rx=${pattern:-$rx}
- rx=${rx:-'(?<!-)\b\Q'"$pkgname"'\E[-_]?((src|source)[-_])?v?\K([^-/_\s]*?\d[^-/_\s]*?)(?=(?:[-_.](?:src|source|orig))?\.(?:[jt]ar|shar|t[bglx]z|tbz2|zip))\b'}
-
- if [ "${fetchedurls[$url]}" ]; then
- if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
- echo "already fetched $url" 1>&2
- fi
- continue
- fi
-
- if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
- echo "fetching $url and scanning with $rx" 1>&2
- fi
- curl -H 'Accept: text/html,application/xhtml+xml,application/xml,text/plain,application/rss+xml' -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$url" |
- grep -Po -i "$rx"
- fetchedurls[$url]=yes
- done |
- tr _ . |
- sort -Vu |
- {
- grep . || echo "NO VERSION found for $original_pkgname" 1>&2
- } |
- while IFS= read -r found_version; do
- if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
- echo "found version $found_version"
- fi
- consider=true
- p="$ignore "
- while [ -n "$p" ]; do
- i=${p%% *}
- p=${p#* }
- case "$found_version" in
- $i)
- consider=false
- if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
- echo "ignored $found_version due to $i"
- fi
- esac
- done
- if $consider; then
- xbps-uhelper cmpver "$original_pkgname-${version}_1" \
- "$original_pkgname-$(printf %s "$found_version" | tr - .)_1"
- if [ $? = 255 ]; then
- echo "${original_pkgname}-${version} -> ${original_pkgname}-${found_version}"
- fi
- fi
- done
-}
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: move update-check out of xbps-src
2021-06-22 1:11 [PR PATCH] move update-check out of xbps-src sgn
@ 2021-06-22 2:16 ` the-maldridge
2021-06-22 7:13 ` Gottox
` (6 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: the-maldridge @ 2021-06-22 2:16 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 395 bytes --]
New comment by the-maldridge on void-packages repository
https://github.com/void-linux/void-packages/pull/31602#issuecomment-865468643
Comment:
I had started a rewrite in Go, but our regexes require backtracking (`\K`) , which prevented this from being reasonable. If we had a mechanical way to remove our need for backtracking, it would be fairly easy to pick that back up and make it work.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: move update-check out of xbps-src
2021-06-22 1:11 [PR PATCH] move update-check out of xbps-src sgn
2021-06-22 2:16 ` the-maldridge
@ 2021-06-22 7:13 ` Gottox
2021-06-22 7:42 ` the-maldridge
` (5 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Gottox @ 2021-06-22 7:13 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 215 bytes --]
New comment by Gottox on void-packages repository
https://github.com/void-linux/void-packages/pull/31602#issuecomment-865661051
Comment:
Can you elaborate on the reason? 'It's better' is a bit vague for my taste.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: move update-check out of xbps-src
2021-06-22 1:11 [PR PATCH] move update-check out of xbps-src sgn
2021-06-22 2:16 ` the-maldridge
2021-06-22 7:13 ` Gottox
@ 2021-06-22 7:42 ` the-maldridge
2021-06-22 11:05 ` sgn
` (4 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: the-maldridge @ 2021-06-22 7:42 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 358 bytes --]
New comment by the-maldridge on void-packages repository
https://github.com/void-linux/void-packages/pull/31602#issuecomment-865678643
Comment:
@Gottox a primary reason is that shell lacks high parallelism and its difficult to extend. There's a lot more we could be doing with this data but because its in an awkward format its difficult to actually use.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: move update-check out of xbps-src
2021-06-22 1:11 [PR PATCH] move update-check out of xbps-src sgn
` (2 preceding siblings ...)
2021-06-22 7:42 ` the-maldridge
@ 2021-06-22 11:05 ` sgn
2021-06-23 16:36 ` [PR PATCH] [Updated] " sgn
` (3 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: sgn @ 2021-06-22 11:05 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 424 bytes --]
New comment by sgn on void-packages repository
https://github.com/void-linux/void-packages/pull/31602#issuecomment-865887586
Comment:
The reason to move out of `xbps-src` is:
- update-check doesn't need xbps
- running inside xbps-src requires xbps
- running inside xbps-src requires either chrooting (which requires xbps) or XBPS_ALLOW_CHROOT_BREAKOUT
- rewrite in a real programming language could speed things up a bit
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PR PATCH] [Updated] move update-check out of xbps-src
2021-06-22 1:11 [PR PATCH] move update-check out of xbps-src sgn
` (3 preceding siblings ...)
2021-06-22 11:05 ` sgn
@ 2021-06-23 16:36 ` sgn
2021-06-23 17:35 ` Chocimier
` (2 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: sgn @ 2021-06-23 16:36 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 836 bytes --]
There is an updated pull request by sgn against master on the void-packages repository
https://github.com/sgn/void-packages move-update-check-out-of-xbps-src
https://github.com/void-linux/void-packages/pull/31602
move update-check out of xbps-src
Continue discussion from #31546
It's better to rewrite in a real programming language, however this is enough for now.
TODO: check only distfiles.
The `eval` step should be removed when we can remove all variable except `pkgname` and `version` from `distfiles`, and merge the first `distfiles` to the same line of `distfiles=`
@void-linux/pkg-committers
@the-maldridge moving out of `xbps-src`, we can drop the weird `XBPS_ALLOW_CHROOT_BREAKOUT=yes` from `update-check` container.
A patch file from https://github.com/void-linux/void-packages/pull/31602.patch is attached
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-move-update-check-out-of-xbps-src-31602.patch --]
[-- Type: text/x-diff, Size: 20718 bytes --]
From 625eeb2be243fbd6a11a82209c21fc8c354270ed Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=C4=90o=C3=A0n=20Tr=E1=BA=A7n=20C=C3=B4ng=20Danh?=
<congdanhqx@gmail.com>
Date: Tue, 22 Jun 2021 08:00:02 +0700
Subject: [PATCH 1/3] common/scripts: add equivalent script for update-check
This script will read necessary variables from template, then calling
the old update_check method.
---
common/scripts/update-check | 46 +++++++++++++++++++++++++++++++++++++
1 file changed, 46 insertions(+)
create mode 100755 common/scripts/update-check
diff --git a/common/scripts/update-check b/common/scripts/update-check
new file mode 100755
index 000000000000..dbbae3428c70
--- /dev/null
+++ b/common/scripts/update-check
@@ -0,0 +1,46 @@
+#!/bin/bash
+# vim: set ts=4 sw=4 et:
+
+die() {
+ printf '%s\n' "$*" >&2
+ exit 1
+}
+
+pkgname=$1
+
+if [ -z "$pkgname" ]; then
+ die "pkgname is required"
+fi
+
+XBPS_DISTDIR="$(cd "${0%/*}/../.." && pwd)"
+XBPS_SRCPKGDIR="$XBPS_DISTDIR/srcpkgs"
+
+if [ ! -f "$XBPS_SRCPKGDIR/$1/template" ]; then
+ die "$pkgname: not found"
+fi
+
+source "$XBPS_DISTDIR/common/environment/setup/misc.sh"
+source "$XBPS_DISTDIR/common/xbps-src/shutils/update_check.sh"
+
+vars="pkgname|version|homepage|distfiles"
+template="$(
+ sed -E -n "
+ /^[a-z_]*($vars)=[^\"']*\$/p
+ /^[a-z_]*($vars)=\"[^\"]*\"\$/p
+ /^[a-z_]*($vars)='[^']*'\$/p
+ /^[a-z_]*($vars)=\"[^\"]*\$/,/^[^\"]*\"\$/p
+ /^[a-z_]*($vars)='[^']*\$/,/^[^']*'\$/p
+ " "$XBPS_SRCPKGDIR/$1/template"
+)"
+
+# Not really safe
+case "$template" in
+ *'$('* | *'`'*)
+ die "command substitution in pkgname|version|homepage|distfiles"
+ ;;
+esac
+
+eval "$template"
+XBPS_TARGET_PKG=$pkgname
+
+update_check
From dc5214b6205fd64d653102763579a95b0cc5014c Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=C4=90o=C3=A0n=20Tr=E1=BA=A7n=20C=C3=B4ng=20Danh?=
<congdanhqx@gmail.com>
Date: Tue, 22 Jun 2021 08:02:05 +0700
Subject: [PATCH 2/3] xbps-src: call external script for update-check
---
xbps-src | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/xbps-src b/xbps-src
index 806c6480f21c..a21d2aa7fb8a 100755
--- a/xbps-src
+++ b/xbps-src
@@ -959,8 +959,7 @@ case "$XBPS_TARGET" in
bulk_update -I
;;
update-check)
- read_pkg ignore-problems
- update_check
+ ${XBPS_COMMONDIR}/scripts/update-check "$XBPS_TARGET_PKG"
;;
update-hash-cache)
update_hash_cache
From 3a53591a1c287c578eea558d866b664fa7755975 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=C4=90o=C3=A0n=20Tr=E1=BA=A7n=20C=C3=B4ng=20Danh?=
<congdanhqx@gmail.com>
Date: Tue, 22 Jun 2021 08:03:27 +0700
Subject: [PATCH 3/3] update-check: move the real check to external script
The change is best-viewed with git diff --color-moved
---
common/scripts/update-check | 210 +++++++++++++++++++++++-
common/xbps-src/shutils/update_check.sh | 210 ------------------------
2 files changed, 209 insertions(+), 211 deletions(-)
delete mode 100644 common/xbps-src/shutils/update_check.sh
diff --git a/common/scripts/update-check b/common/scripts/update-check
index dbbae3428c70..1d25a7a631eb 100755
--- a/common/scripts/update-check
+++ b/common/scripts/update-check
@@ -20,7 +20,215 @@ if [ ! -f "$XBPS_SRCPKGDIR/$1/template" ]; then
fi
source "$XBPS_DISTDIR/common/environment/setup/misc.sh"
-source "$XBPS_DISTDIR/common/xbps-src/shutils/update_check.sh"
+
+update_check() {
+ local i p url pkgurlname rx found_version consider
+ local update_override=$XBPS_SRCPKGDIR/$XBPS_TARGET_PKG/update
+ local original_pkgname=$pkgname
+ local urlpfx urlsfx
+ local -A fetchedurls
+
+ if [ -r $update_override ]; then
+ . $update_override
+ if [ "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+ echo "using $XBPS_TARGET_PKG/update overrides" 1>&2
+ fi
+ fi
+
+ if ! type curl >/dev/null 2>&1; then
+ echo "ERROR: cannot find \`curl' executable!"
+ return 1
+ fi
+
+ export LC_ALL=C
+
+ if [ -z "$site" ]; then
+ case "$distfiles" in
+ # only consider versions those exist in ftp.gnome.org
+ *ftp.gnome.org*) ;;
+ *)
+ printf '%s\n' "$homepage" ;;
+ esac
+ for i in $distfiles; do
+ printf '%s\n' "${i%/*}/"
+ done
+ else
+ printf '%s\n' "$site"
+ fi |
+ # filter loop: if version are "folder" name based,
+ # substitute original url by every folder based ones (expand)
+ while IFS= read -r url; do
+ # default case: don't rewrite url
+ printf '%s\n' "$url"
+ if [ "$single_directory" ]; then
+ continue
+ fi
+ rx=
+ urlpfx="${url}"
+ urlsfx=
+ dirpfx=
+ case "$url" in
+ *.voidlinux.*|\
+ *sourceforge.net/sourceforge*|\
+ *code.google.com*|*googlecode*|\
+ *launchpad.net*|\
+ *cpan.*|\
+ *pythonhosted.org*|\
+ *github.com*|\
+ *//gitlab.*|\
+ *bitbucket.org*|\
+ *ftp.gnome.org*|\
+ *kernel.org/pub/linux/kernel/*|\
+ *cran.r-project.org/src/contrib*|\
+ *rubygems.org*|\
+ *crates.io*|\
+ *codeberg.org*|\
+ *hg.sr.ht*|\
+ *git.sr.ht*)
+ continue
+ ;;
+ *)
+ vdpfx=${vdprefix:-"|v|\\Q$pkgname\\E"}
+ vdsfx=${vdsuffix:-"|\\.x"}
+ match=$(grep -Po "^[^/]+//[^/]+(/.+)?/($vdpfx)(?=[-_.0-9]*[0-9](?<!\\Q$pkgname\\E)($vdsfx)/)" <<< "$url")
+ if [ "$?" = 0 ]; then
+ urlpfx="${match%/*}/"
+ dirpfx="${match##*/}"
+ urlsfx="${url#$urlpfx}"
+ urlsfx="${urlsfx#*/}"
+ rx="href=[\"']?(\\Q$urlpfx\\E)?\\.?/?\\K\\Q$dirpfx\\E[-_.0-9]*[0-9]($vdsfx)[\"'/]"
+ fi
+ ;;
+ esac
+ if [ "$rx" ]; then
+ # substitute url if needed
+ if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+ echo "(folder) fetching $urlpfx and scanning with $rx" 1>&2
+ fi
+ skipdirs=
+ curl -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$urlpfx" |
+ grep -Po -i "$rx" |
+ # sort -V places 1.1/ before 1/, but 1A/ before 1.1A/
+ sed -e 's:$:A:' -e 's:/A$:A/:' | sort -Vru | sed -e 's:A/$:/A:' -e 's:A$::' |
+ while IFS= read -r newver; do
+ newurl="${urlpfx}${newver}${urlsfx}"
+ if [ "$newurl" = "$url" ]; then
+ skipdirs=yes
+ fi
+ if [ -z "$skipdirs" ]; then
+ printf '%s\n' "$newurl"
+ fi
+ done
+ fi
+ done |
+ while IFS= read -r url; do
+ rx=
+ if [ -z "$site" ]; then
+ case "$url" in
+ *sourceforge.net/sourceforge*)
+ pkgurlname="$(printf %s "$url" | cut -d/ -f5)"
+ url="https://sourceforge.net/projects/$pkgurlname/rss?limit=200";;
+ *code.google.com*|*googlecode*)
+ url="http://code.google.com/p/$pkgname/downloads/list";;
+ *launchpad.net*)
+ pkgurlname="$(printf %s "$url" | cut -d/ -f4)"
+ url="https://launchpad.net/$pkgurlname/+download";;
+ *cpan.*)
+ pkgname=${pkgname#perl-};;
+ *pythonhosted.org*)
+ pkgname=${pkgname#python-}
+ pkgname=${pkgname#python3-}
+ url="https://pypi.org/simple/$pkgname";;
+ *github.com*)
+ pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+ url="https://github.com/$pkgurlname/tags"
+ rx='/archive/refs/tags/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
+ *//gitlab.*)
+ pkgurlname="$(printf %s "$url" | cut -d/ -f1-5)"
+ url="$pkgurlname/tags"
+ rx='/archive/[^/]+/\Q'"$pkgname"'\E-v?\K[\d.]+(?=\.tar\.gz")';;
+ *bitbucket.org*)
+ pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+ url="https://bitbucket.org/$pkgurlname/downloads"
+ rx='/(get|downloads)/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar)';;
+ *ftp.gnome.org*|*download.gnome.org*)
+ : ${pattern="\Q$pkgname\E-\K(0|[13]\.[0-9]*[02468]|[4-9][0-9]+)\.[0-9.]*[0-9](?=)"}
+ url="https://download.gnome.org/sources/$pkgname/cache.json";;
+ *kernel.org/pub/linux/kernel/*)
+ rx=linux-'\K'${version%.*}'[\d.]+(?=\.tar\.xz)';;
+ *cran.r-project.org/src/contrib*)
+ rx='\b\Q'"${pkgname#R-cran-}"'\E_\K\d+(\.\d+)*(-\d+)?(?=\.tar)';;
+ *rubygems.org*)
+ url="https://rubygems.org/gems/${pkgname#ruby-}"
+ rx='href="/gems/'${pkgname#ruby-}'/versions/\K[\d.]*(?=")' ;;
+ *crates.io*)
+ url="https://crates.io/api/v1/crates/${pkgname#rust-}"
+ rx='/crates/'${pkgname#rust-}'/\K[0-9.]*(?=/download)' ;;
+ *codeberg.org*)
+ pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+ url="https://codeberg.org/$pkgurlname/releases"
+ rx='/archive/\K[\d.]+(?=\.tar\.gz)' ;;
+ *hg.sr.ht*)
+ pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+ url="https://hg.sr.ht/$pkgurlname/tags"
+ rx='/archive/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
+ *git.sr.ht*)
+ pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+ url="https://git.sr.ht/$pkgurlname/refs"
+ rx='/archive/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
+ *pkgs.fedoraproject.org*)
+ url="https://pkgs.fedoraproject.org/repo/pkgs/$pkgname" ;;
+ esac
+ fi
+
+ rx=${pattern:-$rx}
+ rx=${rx:-'(?<!-)\b\Q'"$pkgname"'\E[-_]?((src|source)[-_])?v?\K([^-/_\s]*?\d[^-/_\s]*?)(?=(?:[-_.](?:src|source|orig))?\.(?:[jt]ar|shar|t[bglx]z|tbz2|zip))\b'}
+
+ if [ "${fetchedurls[$url]}" ]; then
+ if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+ echo "already fetched $url" 1>&2
+ fi
+ continue
+ fi
+
+ if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+ echo "fetching $url and scanning with $rx" 1>&2
+ fi
+ curl -H 'Accept: text/html,application/xhtml+xml,application/xml,text/plain,application/rss+xml' -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$url" |
+ grep -Po -i "$rx"
+ fetchedurls[$url]=yes
+ done |
+ tr _ . |
+ sort -Vu |
+ {
+ grep . || echo "NO VERSION found for $original_pkgname" 1>&2
+ } |
+ while IFS= read -r found_version; do
+ if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+ echo "found version $found_version"
+ fi
+ consider=true
+ p="$ignore "
+ while [ -n "$p" ]; do
+ i=${p%% *}
+ p=${p#* }
+ case "$found_version" in
+ $i)
+ consider=false
+ if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+ echo "ignored $found_version due to $i"
+ fi
+ esac
+ done
+ if $consider; then
+ xbps-uhelper cmpver "$original_pkgname-${version}_1" \
+ "$original_pkgname-$(printf %s "$found_version" | tr - .)_1"
+ if [ $? = 255 ]; then
+ echo "${original_pkgname}-${version} -> ${original_pkgname}-${found_version}"
+ fi
+ fi
+ done
+}
vars="pkgname|version|homepage|distfiles"
template="$(
diff --git a/common/xbps-src/shutils/update_check.sh b/common/xbps-src/shutils/update_check.sh
deleted file mode 100644
index 0a624274f500..000000000000
--- a/common/xbps-src/shutils/update_check.sh
+++ /dev/null
@@ -1,210 +0,0 @@
-# vim: set ts=4 sw=4 et:
-
-update_check() {
- local i p url pkgurlname rx found_version consider
- local update_override=$XBPS_SRCPKGDIR/$XBPS_TARGET_PKG/update
- local original_pkgname=$pkgname
- local urlpfx urlsfx
- local -A fetchedurls
-
- if [ -r $update_override ]; then
- . $update_override
- if [ "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
- echo "using $XBPS_TARGET_PKG/update overrides" 1>&2
- fi
- fi
-
- if ! type curl >/dev/null 2>&1; then
- echo "ERROR: cannot find \`curl' executable!"
- return 1
- fi
-
- export LC_ALL=C
-
- if [ -z "$site" ]; then
- case "$distfiles" in
- # only consider versions those exist in ftp.gnome.org
- *ftp.gnome.org*) ;;
- *)
- printf '%s\n' "$homepage" ;;
- esac
- for i in $distfiles; do
- printf '%s\n' "${i%/*}/"
- done
- else
- printf '%s\n' "$site"
- fi |
- # filter loop: if version are "folder" name based,
- # substitute original url by every folder based ones (expand)
- while IFS= read -r url; do
- # default case: don't rewrite url
- printf '%s\n' "$url"
- if [ "$single_directory" ]; then
- continue
- fi
- rx=
- urlpfx="${url}"
- urlsfx=
- dirpfx=
- case "$url" in
- *.voidlinux.*|\
- *sourceforge.net/sourceforge*|\
- *code.google.com*|*googlecode*|\
- *launchpad.net*|\
- *cpan.*|\
- *pythonhosted.org*|\
- *github.com*|\
- *//gitlab.*|\
- *bitbucket.org*|\
- *ftp.gnome.org*|\
- *kernel.org/pub/linux/kernel/*|\
- *cran.r-project.org/src/contrib*|\
- *rubygems.org*|\
- *crates.io*|\
- *codeberg.org*|\
- *hg.sr.ht*|\
- *git.sr.ht*)
- continue
- ;;
- *)
- vdpfx=${vdprefix:-"|v|\\Q$pkgname\\E"}
- vdsfx=${vdsuffix:-"|\\.x"}
- match=$(grep -Po "^[^/]+//[^/]+(/.+)?/($vdpfx)(?=[-_.0-9]*[0-9](?<!\\Q$pkgname\\E)($vdsfx)/)" <<< "$url")
- if [ "$?" = 0 ]; then
- urlpfx="${match%/*}/"
- dirpfx="${match##*/}"
- urlsfx="${url#$urlpfx}"
- urlsfx="${urlsfx#*/}"
- rx="href=[\"']?(\\Q$urlpfx\\E)?\\.?/?\\K\\Q$dirpfx\\E[-_.0-9]*[0-9]($vdsfx)[\"'/]"
- fi
- ;;
- esac
- if [ "$rx" ]; then
- # substitute url if needed
- if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
- echo "(folder) fetching $urlpfx and scanning with $rx" 1>&2
- fi
- skipdirs=
- curl -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$urlpfx" |
- grep -Po -i "$rx" |
- # sort -V places 1.1/ before 1/, but 1A/ before 1.1A/
- sed -e 's:$:A:' -e 's:/A$:A/:' | sort -Vru | sed -e 's:A/$:/A:' -e 's:A$::' |
- while IFS= read -r newver; do
- newurl="${urlpfx}${newver}${urlsfx}"
- if [ "$newurl" = "$url" ]; then
- skipdirs=yes
- fi
- if [ -z "$skipdirs" ]; then
- printf '%s\n' "$newurl"
- fi
- done
- fi
- done |
- while IFS= read -r url; do
- rx=
- if [ -z "$site" ]; then
- case "$url" in
- *sourceforge.net/sourceforge*)
- pkgurlname="$(printf %s "$url" | cut -d/ -f5)"
- url="https://sourceforge.net/projects/$pkgurlname/rss?limit=200";;
- *code.google.com*|*googlecode*)
- url="http://code.google.com/p/$pkgname/downloads/list";;
- *launchpad.net*)
- pkgurlname="$(printf %s "$url" | cut -d/ -f4)"
- url="https://launchpad.net/$pkgurlname/+download";;
- *cpan.*)
- pkgname=${pkgname#perl-};;
- *pythonhosted.org*)
- pkgname=${pkgname#python-}
- pkgname=${pkgname#python3-}
- url="https://pypi.org/simple/$pkgname";;
- *github.com*)
- pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
- url="https://github.com/$pkgurlname/tags"
- rx='/archive/refs/tags/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
- *//gitlab.*)
- pkgurlname="$(printf %s "$url" | cut -d/ -f1-5)"
- url="$pkgurlname/tags"
- rx='/archive/[^/]+/\Q'"$pkgname"'\E-v?\K[\d.]+(?=\.tar\.gz")';;
- *bitbucket.org*)
- pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
- url="https://bitbucket.org/$pkgurlname/downloads"
- rx='/(get|downloads)/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar)';;
- *ftp.gnome.org*|*download.gnome.org*)
- : ${pattern="\Q$pkgname\E-\K(0|[13]\.[0-9]*[02468]|[4-9][0-9]+)\.[0-9.]*[0-9](?=)"}
- url="https://download.gnome.org/sources/$pkgname/cache.json";;
- *kernel.org/pub/linux/kernel/*)
- rx=linux-'\K'${version%.*}'[\d.]+(?=\.tar\.xz)';;
- *cran.r-project.org/src/contrib*)
- rx='\b\Q'"${pkgname#R-cran-}"'\E_\K\d+(\.\d+)*(-\d+)?(?=\.tar)';;
- *rubygems.org*)
- url="https://rubygems.org/gems/${pkgname#ruby-}"
- rx='href="/gems/'${pkgname#ruby-}'/versions/\K[\d.]*(?=")' ;;
- *crates.io*)
- url="https://crates.io/api/v1/crates/${pkgname#rust-}"
- rx='/crates/'${pkgname#rust-}'/\K[0-9.]*(?=/download)' ;;
- *codeberg.org*)
- pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
- url="https://codeberg.org/$pkgurlname/releases"
- rx='/archive/\K[\d.]+(?=\.tar\.gz)' ;;
- *hg.sr.ht*)
- pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
- url="https://hg.sr.ht/$pkgurlname/tags"
- rx='/archive/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
- *git.sr.ht*)
- pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
- url="https://git.sr.ht/$pkgurlname/refs"
- rx='/archive/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
- *pkgs.fedoraproject.org*)
- url="https://pkgs.fedoraproject.org/repo/pkgs/$pkgname" ;;
- esac
- fi
-
- rx=${pattern:-$rx}
- rx=${rx:-'(?<!-)\b\Q'"$pkgname"'\E[-_]?((src|source)[-_])?v?\K([^-/_\s]*?\d[^-/_\s]*?)(?=(?:[-_.](?:src|source|orig))?\.(?:[jt]ar|shar|t[bglx]z|tbz2|zip))\b'}
-
- if [ "${fetchedurls[$url]}" ]; then
- if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
- echo "already fetched $url" 1>&2
- fi
- continue
- fi
-
- if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
- echo "fetching $url and scanning with $rx" 1>&2
- fi
- curl -H 'Accept: text/html,application/xhtml+xml,application/xml,text/plain,application/rss+xml' -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$url" |
- grep -Po -i "$rx"
- fetchedurls[$url]=yes
- done |
- tr _ . |
- sort -Vu |
- {
- grep . || echo "NO VERSION found for $original_pkgname" 1>&2
- } |
- while IFS= read -r found_version; do
- if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
- echo "found version $found_version"
- fi
- consider=true
- p="$ignore "
- while [ -n "$p" ]; do
- i=${p%% *}
- p=${p#* }
- case "$found_version" in
- $i)
- consider=false
- if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
- echo "ignored $found_version due to $i"
- fi
- esac
- done
- if $consider; then
- xbps-uhelper cmpver "$original_pkgname-${version}_1" \
- "$original_pkgname-$(printf %s "$found_version" | tr - .)_1"
- if [ $? = 255 ]; then
- echo "${original_pkgname}-${version} -> ${original_pkgname}-${found_version}"
- fi
- fi
- done
-}
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: move update-check out of xbps-src
2021-06-22 1:11 [PR PATCH] move update-check out of xbps-src sgn
` (4 preceding siblings ...)
2021-06-23 16:36 ` [PR PATCH] [Updated] " sgn
@ 2021-06-23 17:35 ` Chocimier
2022-05-23 2:12 ` github-actions
2022-06-06 2:14 ` [PR PATCH] [Closed]: " github-actions
7 siblings, 0 replies; 9+ messages in thread
From: Chocimier @ 2021-06-23 17:35 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 1770 bytes --]
New comment by Chocimier on void-packages repository
https://github.com/void-linux/void-packages/pull/31602#issuecomment-867029335
Comment:
> check only distfiles
homepage and site are important too
> remove all variable except pkgname and version from distfiles
There are templates that use wrksrc, _distver being custom transformation of version and more.
> update-check doesn't need xbps
It uses dewey version sort with `xbps-uhelper cmpver` and needs to use it more. Current [sort -Vu](https://github.com/void-linux/void-packages/blob/471c2d1dd4af758dcd5a450f498439dc25045e34/common/xbps-src/shutils/update_check.sh#L179) sorts beta releases as greater than final releases - so that xupdate updates templates that for some reasons are in `2beta1` version to `2beta2` rather than `2`.
> requires [...] XBPS_ALLOW_CHROOT_BREAKOUT
Is this problem?
> could speed things up a bit
Test was done on templates with `0.` prepended to version to force matches.
There are around 60 packages for which new versions are not detected after that change as for now, not counting R and ruby packages defining distfiles in build style.
Run time of void-updates -p 5 went down by 5min, from 2h 25min to 2h 20min. On same machine, `xbps-src show` for all packages takes around 10 min - limit of what can be expected to improve.
Roughly half of execution time was spent in void-updates' `get_date` - but this is not representative, as there are more found versions that usually.
My conclusion is that this is micro-optimization that doesn't touch real problem pointed by maldridge - templates being bash programs depending on lot of xbps-src code that are complex to really understand, hard or slow to process for all usecases, like cycle detection.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: move update-check out of xbps-src
2021-06-22 1:11 [PR PATCH] move update-check out of xbps-src sgn
` (5 preceding siblings ...)
2021-06-23 17:35 ` Chocimier
@ 2022-05-23 2:12 ` github-actions
2022-06-06 2:14 ` [PR PATCH] [Closed]: " github-actions
7 siblings, 0 replies; 9+ messages in thread
From: github-actions @ 2022-05-23 2:12 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 305 bytes --]
New comment by github-actions[bot] on void-packages repository
https://github.com/void-linux/void-packages/pull/31602#issuecomment-1134091762
Comment:
Pull Requests become stale 90 days after last activity and are closed 14 days after that. If this pull request is still relevant bump it or assign it.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PR PATCH] [Closed]: move update-check out of xbps-src
2021-06-22 1:11 [PR PATCH] move update-check out of xbps-src sgn
` (6 preceding siblings ...)
2022-05-23 2:12 ` github-actions
@ 2022-06-06 2:14 ` github-actions
7 siblings, 0 replies; 9+ messages in thread
From: github-actions @ 2022-06-06 2:14 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 662 bytes --]
There's a closed pull request on the void-packages repository
move update-check out of xbps-src
https://github.com/void-linux/void-packages/pull/31602
Description:
Continue discussion from #31546
It's better to rewrite in a real programming language, however this is enough for now.
TODO: check only distfiles.
The `eval` step should be removed when we can remove all variable except `pkgname` and `version` from `distfiles`, and merge the first `distfiles` to the same line of `distfiles=`
@void-linux/pkg-committers
@the-maldridge moving out of `xbps-src`, we can drop the weird `XBPS_ALLOW_CHROOT_BREAKOUT=yes` from `update-check` container.
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2022-06-06 2:14 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-22 1:11 [PR PATCH] move update-check out of xbps-src sgn
2021-06-22 2:16 ` the-maldridge
2021-06-22 7:13 ` Gottox
2021-06-22 7:42 ` the-maldridge
2021-06-22 11:05 ` sgn
2021-06-23 16:36 ` [PR PATCH] [Updated] " sgn
2021-06-23 17:35 ` Chocimier
2022-05-23 2:12 ` github-actions
2022-06-06 2:14 ` [PR PATCH] [Closed]: " github-actions
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).