Github messages for voidlinux
 help / color / mirror / Atom feed
* [PR PATCH] move update-check out of xbps-src
@ 2021-06-22  1:11 sgn
  2021-06-22  2:16 ` the-maldridge
                   ` (7 more replies)
  0 siblings, 8 replies; 9+ messages in thread
From: sgn @ 2021-06-22  1:11 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 831 bytes --]

There is a new pull request by sgn against master on the void-packages repository

https://github.com/sgn/void-packages move-update-check-out-of-xbps-src
https://github.com/void-linux/void-packages/pull/31602

move update-check out of xbps-src
Continue discussion from #31546 

It's better to rewrite in a real programming language, however this is enough for now.

TODO: check only distfiles.

The `eval` step should be removed when we can remove all variable except `pkgname` and `version` from `distfiles`, and merge the first `distfiles` to the same line of `distfiles=`

@void-linux/pkg-committers 
@the-maldridge moving out of `xbps-src`, we can drop the weird `XBPS_ALLOW_CHROOT_BREAKOUT=yes` from `update-check` container.

A patch file from https://github.com/void-linux/void-packages/pull/31602.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-move-update-check-out-of-xbps-src-31602.patch --]
[-- Type: text/x-diff, Size: 20488 bytes --]

From 5422dc71da86fbadd5c24f7371944606455019f8 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=C4=90o=C3=A0n=20Tr=E1=BA=A7n=20C=C3=B4ng=20Danh?=
 <congdanhqx@gmail.com>
Date: Tue, 22 Jun 2021 08:00:02 +0700
Subject: [PATCH 1/3] common/scripts: add equivalent script for update-check

This script will read necessary variables from template, then calling
the old update_check method.
---
 common/scripts/update-check | 46 +++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100755 common/scripts/update-check

diff --git a/common/scripts/update-check b/common/scripts/update-check
new file mode 100755
index 000000000000..dbbae3428c70
--- /dev/null
+++ b/common/scripts/update-check
@@ -0,0 +1,46 @@
+#!/bin/bash
+# vim: set ts=4 sw=4 et:
+
+die() {
+    printf '%s\n' "$*" >&2
+    exit 1
+}
+
+pkgname=$1
+
+if [ -z "$pkgname" ]; then
+    die "pkgname is required"
+fi
+
+XBPS_DISTDIR="$(cd "${0%/*}/../.." && pwd)"
+XBPS_SRCPKGDIR="$XBPS_DISTDIR/srcpkgs"
+
+if [ ! -f "$XBPS_SRCPKGDIR/$1/template" ]; then
+    die "$pkgname: not found"
+fi
+
+source "$XBPS_DISTDIR/common/environment/setup/misc.sh"
+source "$XBPS_DISTDIR/common/xbps-src/shutils/update_check.sh"
+
+vars="pkgname|version|homepage|distfiles"
+template="$(
+    sed -E -n "
+        /^[a-z_]*($vars)=[^\"']*\$/p
+        /^[a-z_]*($vars)=\"[^\"]*\"\$/p
+        /^[a-z_]*($vars)='[^']*'\$/p
+        /^[a-z_]*($vars)=\"[^\"]*\$/,/^[^\"]*\"\$/p
+        /^[a-z_]*($vars)='[^']*\$/,/^[^']*'\$/p
+    " "$XBPS_SRCPKGDIR/$1/template"
+)"
+
+# Not really safe
+case "$template" in
+    *'$('* | *'`'*)
+        die "command substitution in pkgname|version|homepage|distfiles"
+        ;;
+esac
+
+eval "$template"
+XBPS_TARGET_PKG=$pkgname
+
+update_check

From cde200d193588eee33180b25886190ed2ccec957 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=C4=90o=C3=A0n=20Tr=E1=BA=A7n=20C=C3=B4ng=20Danh?=
 <congdanhqx@gmail.com>
Date: Tue, 22 Jun 2021 08:02:05 +0700
Subject: [PATCH 2/3] xbps-src: call external script for update-check

---
 xbps-src | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/xbps-src b/xbps-src
index 806c6480f21c..a21d2aa7fb8a 100755
--- a/xbps-src
+++ b/xbps-src
@@ -959,8 +959,7 @@ case "$XBPS_TARGET" in
         bulk_update -I
         ;;
     update-check)
-        read_pkg ignore-problems
-        update_check
+        ${XBPS_COMMONDIR}/scripts/update-check "$XBPS_TARGET_PKG"
         ;;
     update-hash-cache)
         update_hash_cache

From d332ba77b6c6940318f887d0e16ebce6103a9a4c Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=C4=90o=C3=A0n=20Tr=E1=BA=A7n=20C=C3=B4ng=20Danh?=
 <congdanhqx@gmail.com>
Date: Tue, 22 Jun 2021 08:03:27 +0700
Subject: [PATCH 3/3] update-check: move the real check to external script

The change is best-viewed with git diff --color-moved
---
 common/scripts/update-check             | 208 +++++++++++++++++++++++-
 common/xbps-src/shutils/update_check.sh | 208 ------------------------
 2 files changed, 207 insertions(+), 209 deletions(-)
 delete mode 100644 common/xbps-src/shutils/update_check.sh

diff --git a/common/scripts/update-check b/common/scripts/update-check
index dbbae3428c70..f8be72b6265a 100755
--- a/common/scripts/update-check
+++ b/common/scripts/update-check
@@ -20,7 +20,213 @@ if [ ! -f "$XBPS_SRCPKGDIR/$1/template" ]; then
 fi
 
 source "$XBPS_DISTDIR/common/environment/setup/misc.sh"
-source "$XBPS_DISTDIR/common/xbps-src/shutils/update_check.sh"
+
+update_check() {
+    local i p url pkgurlname rx found_version consider
+    local update_override=$XBPS_SRCPKGDIR/$XBPS_TARGET_PKG/update
+    local original_pkgname=$pkgname
+    local urlpfx urlsfx
+    local -A fetchedurls
+
+    if [ -r $update_override ]; then
+        . $update_override
+        if [ "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+            echo "using $XBPS_TARGET_PKG/update overrides" 1>&2
+        fi
+    fi
+
+    if ! type curl >/dev/null 2>&1; then
+        echo "ERROR: cannot find \`curl' executable!"
+        return 1
+    fi
+
+    export LC_ALL=C
+
+    if [ -z "$site" ]; then
+        case "$distfiles" in
+            # only consider versions those exist in ftp.gnome.org
+            *ftp.gnome.org*) ;;
+            *)
+                printf '%s\n' "$homepage" ;;
+        esac
+        for i in $distfiles; do
+            printf '%s\n' "${i%/*}/"
+        done
+    else
+        printf '%s\n' "$site"
+    fi |
+    # filter loop: if version are "folder" name based,
+    # substitute original url by every folder based ones (expand)
+    while IFS= read -r url; do
+        # default case: don't rewrite url
+        printf '%s\n' "$url"
+        if [ "$single_directory" ]; then
+            continue
+        fi
+        rx=
+        urlpfx="${url}"
+        urlsfx=
+        dirpfx=
+        case "$url" in
+            *.voidlinux.*|\
+              *sourceforge.net/sourceforge*|\
+              *code.google.com*|*googlecode*|\
+              *launchpad.net*|\
+              *cpan.*|\
+              *pythonhosted.org*|\
+              *github.com*|\
+              *//gitlab.*|\
+              *bitbucket.org*|\
+              *ftp.gnome.org*|\
+              *kernel.org/pub/linux/kernel/*|\
+              *cran.r-project.org/src/contrib*|\
+              *rubygems.org*|\
+              *crates.io*|\
+              *codeberg.org*|\
+              *hg.sr.ht*|\
+              *git.sr.ht*)
+                continue
+                ;;
+            *)
+                vdpfx=${vdprefix:-"|v|\\Q$pkgname\\E"}
+                vdsfx=${vdsuffix:-"|\\.x"}
+                match=$(grep -Po "^[^/]+//[^/]+(/.+)?/($vdpfx)(?=[-_.0-9]*[0-9](?<!\\Q$pkgname\\E)($vdsfx)/)" <<< "$url")
+                if [ "$?" = 0 ]; then
+                    urlpfx="${match%/*}/"
+                    dirpfx="${match##*/}"
+                    urlsfx="${url#$urlpfx}"
+                    urlsfx="${urlsfx#*/}"
+                    rx="href=[\"']?(\\Q$urlpfx\\E)?\\.?/?\\K\\Q$dirpfx\\E[-_.0-9]*[0-9]($vdsfx)[\"'/]"
+                fi
+                ;;
+        esac
+        if [ "$rx" ]; then
+            # substitute url if needed
+            if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+                echo "(folder) fetching $urlpfx and scanning with $rx" 1>&2
+            fi
+            skipdirs=
+            curl -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$urlpfx" |
+                grep -Po -i "$rx" |
+                # sort -V places 1.1/ before 1/, but 1A/ before 1.1A/
+                sed -e 's:$:A:' -e 's:/A$:A/:' | sort -Vru | sed -e 's:A/$:/A:' -e 's:A$::' |
+                while IFS= read -r newver; do
+                    newurl="${urlpfx}${newver}${urlsfx}"
+                    if [ "$newurl" = "$url" ]; then
+                        skipdirs=yes
+                    fi
+                    if [ -z "$skipdirs" ]; then
+                        printf '%s\n' "$newurl"
+                    fi
+                done
+        fi
+    done |
+    while IFS= read -r url; do
+        rx=
+        if [ -z "$site" ]; then
+            case "$url" in
+            *sourceforge.net/sourceforge*)
+                pkgurlname="$(printf %s "$url" | cut -d/ -f5)"
+                url="https://sourceforge.net/projects/$pkgurlname/rss?limit=200";;
+            *code.google.com*|*googlecode*)
+                url="http://code.google.com/p/$pkgname/downloads/list";;
+            *launchpad.net*)
+                pkgurlname="$(printf %s "$url" | cut -d/ -f4)"
+                url="https://launchpad.net/$pkgurlname/+download";;
+            *cpan.*)
+                pkgname=${pkgname#perl-};;
+            *pythonhosted.org*)
+                pkgname=${pkgname#python-}
+                pkgname=${pkgname#python3-}
+                url="https://pypi.org/simple/$pkgname";;
+            *github.com*)
+                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+                url="https://github.com/$pkgurlname/tags"
+                rx='/archive/refs/tags/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
+            *//gitlab.*)
+                pkgurlname="$(printf %s "$url" | cut -d/ -f1-5)"
+                url="$pkgurlname/tags"
+                rx='/archive/[^/]+/\Q'"$pkgname"'\E-v?\K[\d.]+(?=\.tar\.gz")';;
+            *bitbucket.org*)
+                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+                url="https://bitbucket.org/$pkgurlname/downloads"
+                rx='/(get|downloads)/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar)';;
+            *ftp.gnome.org*|*download.gnome.org*)
+                : ${pattern="\Q$pkgname\E-\K(0|[13]\.[0-9]*[02468]|[4-9][0-9]+)\.[0-9.]*[0-9](?=)"}
+                url="https://download.gnome.org/sources/$pkgname/cache.json";;
+            *kernel.org/pub/linux/kernel/*)
+                rx=linux-'\K'${version%.*}'[\d.]+(?=\.tar\.xz)';;
+            *cran.r-project.org/src/contrib*)
+                rx='\b\Q'"${pkgname#R-cran-}"'\E_\K\d+(\.\d+)*(-\d+)?(?=\.tar)';;
+            *rubygems.org*)
+                url="https://rubygems.org/gems/${pkgname#ruby-}"
+                rx='href="/gems/'${pkgname#ruby-}'/versions/\K[\d.]*(?=")' ;;
+            *crates.io*)
+                url="https://crates.io/api/v1/crates/${pkgname#rust-}"
+                rx='/crates/'${pkgname#rust-}'/\K[0-9.]*(?=/download)' ;;
+            *codeberg.org*)
+                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+                url="https://codeberg.org/$pkgurlname/releases"
+                rx='/archive/\K[\d.]+(?=\.tar\.gz)' ;;
+            *hg.sr.ht*)
+                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+                url="https://hg.sr.ht/$pkgurlname/tags"
+                rx='/archive/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
+            *git.sr.ht*)
+                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+                url="https://git.sr.ht/$pkgurlname/refs"
+                rx='/archive/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
+            esac
+        fi
+
+        rx=${pattern:-$rx}
+        rx=${rx:-'(?<!-)\b\Q'"$pkgname"'\E[-_]?((src|source)[-_])?v?\K([^-/_\s]*?\d[^-/_\s]*?)(?=(?:[-_.](?:src|source|orig))?\.(?:[jt]ar|shar|t[bglx]z|tbz2|zip))\b'}
+
+        if [ "${fetchedurls[$url]}" ]; then
+            if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+                echo "already fetched $url" 1>&2
+            fi
+            continue
+        fi
+
+        if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+            echo "fetching $url and scanning with $rx" 1>&2
+        fi
+        curl -H 'Accept: text/html,application/xhtml+xml,application/xml,text/plain,application/rss+xml' -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$url" |
+            grep -Po -i "$rx"
+        fetchedurls[$url]=yes
+    done |
+    tr _ . |
+    sort -Vu |
+    {
+        grep . || echo "NO VERSION found for $original_pkgname" 1>&2
+    } |
+    while IFS= read -r found_version; do
+        if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+            echo "found version $found_version"
+        fi
+        consider=true
+        p="$ignore "
+        while [ -n "$p" ]; do
+            i=${p%% *}
+            p=${p#* }
+            case "$found_version" in
+            $i)
+                consider=false
+                if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+                    echo "ignored $found_version due to $i"
+                fi
+            esac
+        done
+        if $consider; then
+            xbps-uhelper cmpver "$original_pkgname-${version}_1" \
+                "$original_pkgname-$(printf %s "$found_version" | tr - .)_1"
+            if [ $? = 255 ]; then
+                echo "${original_pkgname}-${version} -> ${original_pkgname}-${found_version}"
+            fi
+        fi
+    done
+}
 
 vars="pkgname|version|homepage|distfiles"
 template="$(
diff --git a/common/xbps-src/shutils/update_check.sh b/common/xbps-src/shutils/update_check.sh
deleted file mode 100644
index fcfbf31759d6..000000000000
--- a/common/xbps-src/shutils/update_check.sh
+++ /dev/null
@@ -1,208 +0,0 @@
-# vim: set ts=4 sw=4 et:
-
-update_check() {
-    local i p url pkgurlname rx found_version consider
-    local update_override=$XBPS_SRCPKGDIR/$XBPS_TARGET_PKG/update
-    local original_pkgname=$pkgname
-    local urlpfx urlsfx
-    local -A fetchedurls
-
-    if [ -r $update_override ]; then
-        . $update_override
-        if [ "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
-            echo "using $XBPS_TARGET_PKG/update overrides" 1>&2
-        fi
-    fi
-
-    if ! type curl >/dev/null 2>&1; then
-        echo "ERROR: cannot find \`curl' executable!"
-        return 1
-    fi
-
-    export LC_ALL=C
-
-    if [ -z "$site" ]; then
-        case "$distfiles" in
-            # only consider versions those exist in ftp.gnome.org
-            *ftp.gnome.org*) ;;
-            *)
-                printf '%s\n' "$homepage" ;;
-        esac
-        for i in $distfiles; do
-            printf '%s\n' "${i%/*}/"
-        done
-    else
-        printf '%s\n' "$site"
-    fi |
-    # filter loop: if version are "folder" name based,
-    # substitute original url by every folder based ones (expand)
-    while IFS= read -r url; do
-        # default case: don't rewrite url
-        printf '%s\n' "$url"
-        if [ "$single_directory" ]; then
-            continue
-        fi
-        rx=
-        urlpfx="${url}"
-        urlsfx=
-        dirpfx=
-        case "$url" in
-            *.voidlinux.*|\
-              *sourceforge.net/sourceforge*|\
-              *code.google.com*|*googlecode*|\
-              *launchpad.net*|\
-              *cpan.*|\
-              *pythonhosted.org*|\
-              *github.com*|\
-              *//gitlab.*|\
-              *bitbucket.org*|\
-              *ftp.gnome.org*|\
-              *kernel.org/pub/linux/kernel/*|\
-              *cran.r-project.org/src/contrib*|\
-              *rubygems.org*|\
-              *crates.io*|\
-              *codeberg.org*|\
-              *hg.sr.ht*|\
-              *git.sr.ht*)
-                continue
-                ;;
-            *)
-                vdpfx=${vdprefix:-"|v|\\Q$pkgname\\E"}
-                vdsfx=${vdsuffix:-"|\\.x"}
-                match=$(grep -Po "^[^/]+//[^/]+(/.+)?/($vdpfx)(?=[-_.0-9]*[0-9](?<!\\Q$pkgname\\E)($vdsfx)/)" <<< "$url")
-                if [ "$?" = 0 ]; then
-                    urlpfx="${match%/*}/"
-                    dirpfx="${match##*/}"
-                    urlsfx="${url#$urlpfx}"
-                    urlsfx="${urlsfx#*/}"
-                    rx="href=[\"']?(\\Q$urlpfx\\E)?\\.?/?\\K\\Q$dirpfx\\E[-_.0-9]*[0-9]($vdsfx)[\"'/]"
-                fi
-                ;;
-        esac
-        if [ "$rx" ]; then
-            # substitute url if needed
-            if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
-                echo "(folder) fetching $urlpfx and scanning with $rx" 1>&2
-            fi
-            skipdirs=
-            curl -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$urlpfx" |
-                grep -Po -i "$rx" |
-                # sort -V places 1.1/ before 1/, but 1A/ before 1.1A/
-                sed -e 's:$:A:' -e 's:/A$:A/:' | sort -Vru | sed -e 's:A/$:/A:' -e 's:A$::' |
-                while IFS= read -r newver; do
-                    newurl="${urlpfx}${newver}${urlsfx}"
-                    if [ "$newurl" = "$url" ]; then
-                        skipdirs=yes
-                    fi
-                    if [ -z "$skipdirs" ]; then
-                        printf '%s\n' "$newurl"
-                    fi
-                done
-        fi
-    done |
-    while IFS= read -r url; do
-        rx=
-        if [ -z "$site" ]; then
-            case "$url" in
-            *sourceforge.net/sourceforge*)
-                pkgurlname="$(printf %s "$url" | cut -d/ -f5)"
-                url="https://sourceforge.net/projects/$pkgurlname/rss?limit=200";;
-            *code.google.com*|*googlecode*)
-                url="http://code.google.com/p/$pkgname/downloads/list";;
-            *launchpad.net*)
-                pkgurlname="$(printf %s "$url" | cut -d/ -f4)"
-                url="https://launchpad.net/$pkgurlname/+download";;
-            *cpan.*)
-                pkgname=${pkgname#perl-};;
-            *pythonhosted.org*)
-                pkgname=${pkgname#python-}
-                pkgname=${pkgname#python3-}
-                url="https://pypi.org/simple/$pkgname";;
-            *github.com*)
-                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
-                url="https://github.com/$pkgurlname/tags"
-                rx='/archive/refs/tags/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
-            *//gitlab.*)
-                pkgurlname="$(printf %s "$url" | cut -d/ -f1-5)"
-                url="$pkgurlname/tags"
-                rx='/archive/[^/]+/\Q'"$pkgname"'\E-v?\K[\d.]+(?=\.tar\.gz")';;
-            *bitbucket.org*)
-                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
-                url="https://bitbucket.org/$pkgurlname/downloads"
-                rx='/(get|downloads)/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar)';;
-            *ftp.gnome.org*|*download.gnome.org*)
-                : ${pattern="\Q$pkgname\E-\K(0|[13]\.[0-9]*[02468]|[4-9][0-9]+)\.[0-9.]*[0-9](?=)"}
-                url="https://download.gnome.org/sources/$pkgname/cache.json";;
-            *kernel.org/pub/linux/kernel/*)
-                rx=linux-'\K'${version%.*}'[\d.]+(?=\.tar\.xz)';;
-            *cran.r-project.org/src/contrib*)
-                rx='\b\Q'"${pkgname#R-cran-}"'\E_\K\d+(\.\d+)*(-\d+)?(?=\.tar)';;
-            *rubygems.org*)
-                url="https://rubygems.org/gems/${pkgname#ruby-}"
-                rx='href="/gems/'${pkgname#ruby-}'/versions/\K[\d.]*(?=")' ;;
-            *crates.io*)
-                url="https://crates.io/api/v1/crates/${pkgname#rust-}"
-                rx='/crates/'${pkgname#rust-}'/\K[0-9.]*(?=/download)' ;;
-            *codeberg.org*)
-                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
-                url="https://codeberg.org/$pkgurlname/releases"
-                rx='/archive/\K[\d.]+(?=\.tar\.gz)' ;;
-            *hg.sr.ht*)
-                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
-                url="https://hg.sr.ht/$pkgurlname/tags"
-                rx='/archive/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
-            *git.sr.ht*)
-                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
-                url="https://git.sr.ht/$pkgurlname/refs"
-                rx='/archive/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
-            esac
-        fi
-
-        rx=${pattern:-$rx}
-        rx=${rx:-'(?<!-)\b\Q'"$pkgname"'\E[-_]?((src|source)[-_])?v?\K([^-/_\s]*?\d[^-/_\s]*?)(?=(?:[-_.](?:src|source|orig))?\.(?:[jt]ar|shar|t[bglx]z|tbz2|zip))\b'}
-
-        if [ "${fetchedurls[$url]}" ]; then
-            if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
-                echo "already fetched $url" 1>&2
-            fi
-            continue
-        fi
-
-        if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
-            echo "fetching $url and scanning with $rx" 1>&2
-        fi
-        curl -H 'Accept: text/html,application/xhtml+xml,application/xml,text/plain,application/rss+xml' -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$url" |
-            grep -Po -i "$rx"
-        fetchedurls[$url]=yes
-    done |
-    tr _ . |
-    sort -Vu |
-    {
-        grep . || echo "NO VERSION found for $original_pkgname" 1>&2
-    } |
-    while IFS= read -r found_version; do
-        if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
-            echo "found version $found_version"
-        fi
-        consider=true
-        p="$ignore "
-        while [ -n "$p" ]; do
-            i=${p%% *}
-            p=${p#* }
-            case "$found_version" in
-            $i)
-                consider=false
-                if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
-                    echo "ignored $found_version due to $i"
-                fi
-            esac
-        done
-        if $consider; then
-            xbps-uhelper cmpver "$original_pkgname-${version}_1" \
-                "$original_pkgname-$(printf %s "$found_version" | tr - .)_1"
-            if [ $? = 255 ]; then
-                echo "${original_pkgname}-${version} -> ${original_pkgname}-${found_version}"
-            fi
-        fi
-    done
-}

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: move update-check out of xbps-src
  2021-06-22  1:11 [PR PATCH] move update-check out of xbps-src sgn
@ 2021-06-22  2:16 ` the-maldridge
  2021-06-22  7:13 ` Gottox
                   ` (6 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: the-maldridge @ 2021-06-22  2:16 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 395 bytes --]

New comment by the-maldridge on void-packages repository

https://github.com/void-linux/void-packages/pull/31602#issuecomment-865468643

Comment:
I had started a rewrite in Go, but our regexes require backtracking (`\K`) , which prevented this from being reasonable.  If we had a mechanical way to remove our need for backtracking, it would be fairly easy to pick that back up and make it work.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: move update-check out of xbps-src
  2021-06-22  1:11 [PR PATCH] move update-check out of xbps-src sgn
  2021-06-22  2:16 ` the-maldridge
@ 2021-06-22  7:13 ` Gottox
  2021-06-22  7:42 ` the-maldridge
                   ` (5 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Gottox @ 2021-06-22  7:13 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 215 bytes --]

New comment by Gottox on void-packages repository

https://github.com/void-linux/void-packages/pull/31602#issuecomment-865661051

Comment:
Can you elaborate on the reason? 'It's better' is a bit vague for my taste.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: move update-check out of xbps-src
  2021-06-22  1:11 [PR PATCH] move update-check out of xbps-src sgn
  2021-06-22  2:16 ` the-maldridge
  2021-06-22  7:13 ` Gottox
@ 2021-06-22  7:42 ` the-maldridge
  2021-06-22 11:05 ` sgn
                   ` (4 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: the-maldridge @ 2021-06-22  7:42 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 358 bytes --]

New comment by the-maldridge on void-packages repository

https://github.com/void-linux/void-packages/pull/31602#issuecomment-865678643

Comment:
@Gottox a primary reason is that shell lacks high parallelism and its difficult to extend.  There's a lot more we could be doing with this data but because its in an awkward format its difficult to actually use.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: move update-check out of xbps-src
  2021-06-22  1:11 [PR PATCH] move update-check out of xbps-src sgn
                   ` (2 preceding siblings ...)
  2021-06-22  7:42 ` the-maldridge
@ 2021-06-22 11:05 ` sgn
  2021-06-23 16:36 ` [PR PATCH] [Updated] " sgn
                   ` (3 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: sgn @ 2021-06-22 11:05 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 424 bytes --]

New comment by sgn on void-packages repository

https://github.com/void-linux/void-packages/pull/31602#issuecomment-865887586

Comment:
The reason to move out of `xbps-src` is:

- update-check doesn't need xbps
- running inside xbps-src requires xbps
- running inside xbps-src requires either chrooting (which requires xbps) or XBPS_ALLOW_CHROOT_BREAKOUT
- rewrite in a real programming language could speed things up a bit

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PR PATCH] [Updated] move update-check out of xbps-src
  2021-06-22  1:11 [PR PATCH] move update-check out of xbps-src sgn
                   ` (3 preceding siblings ...)
  2021-06-22 11:05 ` sgn
@ 2021-06-23 16:36 ` sgn
  2021-06-23 17:35 ` Chocimier
                   ` (2 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: sgn @ 2021-06-23 16:36 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 836 bytes --]

There is an updated pull request by sgn against master on the void-packages repository

https://github.com/sgn/void-packages move-update-check-out-of-xbps-src
https://github.com/void-linux/void-packages/pull/31602

move update-check out of xbps-src
Continue discussion from #31546 

It's better to rewrite in a real programming language, however this is enough for now.

TODO: check only distfiles.

The `eval` step should be removed when we can remove all variable except `pkgname` and `version` from `distfiles`, and merge the first `distfiles` to the same line of `distfiles=`

@void-linux/pkg-committers 
@the-maldridge moving out of `xbps-src`, we can drop the weird `XBPS_ALLOW_CHROOT_BREAKOUT=yes` from `update-check` container.

A patch file from https://github.com/void-linux/void-packages/pull/31602.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-move-update-check-out-of-xbps-src-31602.patch --]
[-- Type: text/x-diff, Size: 20718 bytes --]

From 625eeb2be243fbd6a11a82209c21fc8c354270ed Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=C4=90o=C3=A0n=20Tr=E1=BA=A7n=20C=C3=B4ng=20Danh?=
 <congdanhqx@gmail.com>
Date: Tue, 22 Jun 2021 08:00:02 +0700
Subject: [PATCH 1/3] common/scripts: add equivalent script for update-check

This script will read necessary variables from template, then calling
the old update_check method.
---
 common/scripts/update-check | 46 +++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100755 common/scripts/update-check

diff --git a/common/scripts/update-check b/common/scripts/update-check
new file mode 100755
index 000000000000..dbbae3428c70
--- /dev/null
+++ b/common/scripts/update-check
@@ -0,0 +1,46 @@
+#!/bin/bash
+# vim: set ts=4 sw=4 et:
+
+die() {
+    printf '%s\n' "$*" >&2
+    exit 1
+}
+
+pkgname=$1
+
+if [ -z "$pkgname" ]; then
+    die "pkgname is required"
+fi
+
+XBPS_DISTDIR="$(cd "${0%/*}/../.." && pwd)"
+XBPS_SRCPKGDIR="$XBPS_DISTDIR/srcpkgs"
+
+if [ ! -f "$XBPS_SRCPKGDIR/$1/template" ]; then
+    die "$pkgname: not found"
+fi
+
+source "$XBPS_DISTDIR/common/environment/setup/misc.sh"
+source "$XBPS_DISTDIR/common/xbps-src/shutils/update_check.sh"
+
+vars="pkgname|version|homepage|distfiles"
+template="$(
+    sed -E -n "
+        /^[a-z_]*($vars)=[^\"']*\$/p
+        /^[a-z_]*($vars)=\"[^\"]*\"\$/p
+        /^[a-z_]*($vars)='[^']*'\$/p
+        /^[a-z_]*($vars)=\"[^\"]*\$/,/^[^\"]*\"\$/p
+        /^[a-z_]*($vars)='[^']*\$/,/^[^']*'\$/p
+    " "$XBPS_SRCPKGDIR/$1/template"
+)"
+
+# Not really safe
+case "$template" in
+    *'$('* | *'`'*)
+        die "command substitution in pkgname|version|homepage|distfiles"
+        ;;
+esac
+
+eval "$template"
+XBPS_TARGET_PKG=$pkgname
+
+update_check

From dc5214b6205fd64d653102763579a95b0cc5014c Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=C4=90o=C3=A0n=20Tr=E1=BA=A7n=20C=C3=B4ng=20Danh?=
 <congdanhqx@gmail.com>
Date: Tue, 22 Jun 2021 08:02:05 +0700
Subject: [PATCH 2/3] xbps-src: call external script for update-check

---
 xbps-src | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/xbps-src b/xbps-src
index 806c6480f21c..a21d2aa7fb8a 100755
--- a/xbps-src
+++ b/xbps-src
@@ -959,8 +959,7 @@ case "$XBPS_TARGET" in
         bulk_update -I
         ;;
     update-check)
-        read_pkg ignore-problems
-        update_check
+        ${XBPS_COMMONDIR}/scripts/update-check "$XBPS_TARGET_PKG"
         ;;
     update-hash-cache)
         update_hash_cache

From 3a53591a1c287c578eea558d866b664fa7755975 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=C4=90o=C3=A0n=20Tr=E1=BA=A7n=20C=C3=B4ng=20Danh?=
 <congdanhqx@gmail.com>
Date: Tue, 22 Jun 2021 08:03:27 +0700
Subject: [PATCH 3/3] update-check: move the real check to external script

The change is best-viewed with git diff --color-moved
---
 common/scripts/update-check             | 210 +++++++++++++++++++++++-
 common/xbps-src/shutils/update_check.sh | 210 ------------------------
 2 files changed, 209 insertions(+), 211 deletions(-)
 delete mode 100644 common/xbps-src/shutils/update_check.sh

diff --git a/common/scripts/update-check b/common/scripts/update-check
index dbbae3428c70..1d25a7a631eb 100755
--- a/common/scripts/update-check
+++ b/common/scripts/update-check
@@ -20,7 +20,215 @@ if [ ! -f "$XBPS_SRCPKGDIR/$1/template" ]; then
 fi
 
 source "$XBPS_DISTDIR/common/environment/setup/misc.sh"
-source "$XBPS_DISTDIR/common/xbps-src/shutils/update_check.sh"
+
+update_check() {
+    local i p url pkgurlname rx found_version consider
+    local update_override=$XBPS_SRCPKGDIR/$XBPS_TARGET_PKG/update
+    local original_pkgname=$pkgname
+    local urlpfx urlsfx
+    local -A fetchedurls
+
+    if [ -r $update_override ]; then
+        . $update_override
+        if [ "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+            echo "using $XBPS_TARGET_PKG/update overrides" 1>&2
+        fi
+    fi
+
+    if ! type curl >/dev/null 2>&1; then
+        echo "ERROR: cannot find \`curl' executable!"
+        return 1
+    fi
+
+    export LC_ALL=C
+
+    if [ -z "$site" ]; then
+        case "$distfiles" in
+            # only consider versions those exist in ftp.gnome.org
+            *ftp.gnome.org*) ;;
+            *)
+                printf '%s\n' "$homepage" ;;
+        esac
+        for i in $distfiles; do
+            printf '%s\n' "${i%/*}/"
+        done
+    else
+        printf '%s\n' "$site"
+    fi |
+    # filter loop: if version are "folder" name based,
+    # substitute original url by every folder based ones (expand)
+    while IFS= read -r url; do
+        # default case: don't rewrite url
+        printf '%s\n' "$url"
+        if [ "$single_directory" ]; then
+            continue
+        fi
+        rx=
+        urlpfx="${url}"
+        urlsfx=
+        dirpfx=
+        case "$url" in
+            *.voidlinux.*|\
+              *sourceforge.net/sourceforge*|\
+              *code.google.com*|*googlecode*|\
+              *launchpad.net*|\
+              *cpan.*|\
+              *pythonhosted.org*|\
+              *github.com*|\
+              *//gitlab.*|\
+              *bitbucket.org*|\
+              *ftp.gnome.org*|\
+              *kernel.org/pub/linux/kernel/*|\
+              *cran.r-project.org/src/contrib*|\
+              *rubygems.org*|\
+              *crates.io*|\
+              *codeberg.org*|\
+              *hg.sr.ht*|\
+              *git.sr.ht*)
+                continue
+                ;;
+            *)
+                vdpfx=${vdprefix:-"|v|\\Q$pkgname\\E"}
+                vdsfx=${vdsuffix:-"|\\.x"}
+                match=$(grep -Po "^[^/]+//[^/]+(/.+)?/($vdpfx)(?=[-_.0-9]*[0-9](?<!\\Q$pkgname\\E)($vdsfx)/)" <<< "$url")
+                if [ "$?" = 0 ]; then
+                    urlpfx="${match%/*}/"
+                    dirpfx="${match##*/}"
+                    urlsfx="${url#$urlpfx}"
+                    urlsfx="${urlsfx#*/}"
+                    rx="href=[\"']?(\\Q$urlpfx\\E)?\\.?/?\\K\\Q$dirpfx\\E[-_.0-9]*[0-9]($vdsfx)[\"'/]"
+                fi
+                ;;
+        esac
+        if [ "$rx" ]; then
+            # substitute url if needed
+            if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+                echo "(folder) fetching $urlpfx and scanning with $rx" 1>&2
+            fi
+            skipdirs=
+            curl -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$urlpfx" |
+                grep -Po -i "$rx" |
+                # sort -V places 1.1/ before 1/, but 1A/ before 1.1A/
+                sed -e 's:$:A:' -e 's:/A$:A/:' | sort -Vru | sed -e 's:A/$:/A:' -e 's:A$::' |
+                while IFS= read -r newver; do
+                    newurl="${urlpfx}${newver}${urlsfx}"
+                    if [ "$newurl" = "$url" ]; then
+                        skipdirs=yes
+                    fi
+                    if [ -z "$skipdirs" ]; then
+                        printf '%s\n' "$newurl"
+                    fi
+                done
+        fi
+    done |
+    while IFS= read -r url; do
+        rx=
+        if [ -z "$site" ]; then
+            case "$url" in
+            *sourceforge.net/sourceforge*)
+                pkgurlname="$(printf %s "$url" | cut -d/ -f5)"
+                url="https://sourceforge.net/projects/$pkgurlname/rss?limit=200";;
+            *code.google.com*|*googlecode*)
+                url="http://code.google.com/p/$pkgname/downloads/list";;
+            *launchpad.net*)
+                pkgurlname="$(printf %s "$url" | cut -d/ -f4)"
+                url="https://launchpad.net/$pkgurlname/+download";;
+            *cpan.*)
+                pkgname=${pkgname#perl-};;
+            *pythonhosted.org*)
+                pkgname=${pkgname#python-}
+                pkgname=${pkgname#python3-}
+                url="https://pypi.org/simple/$pkgname";;
+            *github.com*)
+                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+                url="https://github.com/$pkgurlname/tags"
+                rx='/archive/refs/tags/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
+            *//gitlab.*)
+                pkgurlname="$(printf %s "$url" | cut -d/ -f1-5)"
+                url="$pkgurlname/tags"
+                rx='/archive/[^/]+/\Q'"$pkgname"'\E-v?\K[\d.]+(?=\.tar\.gz")';;
+            *bitbucket.org*)
+                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+                url="https://bitbucket.org/$pkgurlname/downloads"
+                rx='/(get|downloads)/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar)';;
+            *ftp.gnome.org*|*download.gnome.org*)
+                : ${pattern="\Q$pkgname\E-\K(0|[13]\.[0-9]*[02468]|[4-9][0-9]+)\.[0-9.]*[0-9](?=)"}
+                url="https://download.gnome.org/sources/$pkgname/cache.json";;
+            *kernel.org/pub/linux/kernel/*)
+                rx=linux-'\K'${version%.*}'[\d.]+(?=\.tar\.xz)';;
+            *cran.r-project.org/src/contrib*)
+                rx='\b\Q'"${pkgname#R-cran-}"'\E_\K\d+(\.\d+)*(-\d+)?(?=\.tar)';;
+            *rubygems.org*)
+                url="https://rubygems.org/gems/${pkgname#ruby-}"
+                rx='href="/gems/'${pkgname#ruby-}'/versions/\K[\d.]*(?=")' ;;
+            *crates.io*)
+                url="https://crates.io/api/v1/crates/${pkgname#rust-}"
+                rx='/crates/'${pkgname#rust-}'/\K[0-9.]*(?=/download)' ;;
+            *codeberg.org*)
+                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+                url="https://codeberg.org/$pkgurlname/releases"
+                rx='/archive/\K[\d.]+(?=\.tar\.gz)' ;;
+            *hg.sr.ht*)
+                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+                url="https://hg.sr.ht/$pkgurlname/tags"
+                rx='/archive/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
+            *git.sr.ht*)
+                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
+                url="https://git.sr.ht/$pkgurlname/refs"
+                rx='/archive/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
+            *pkgs.fedoraproject.org*)
+                url="https://pkgs.fedoraproject.org/repo/pkgs/$pkgname" ;;
+            esac
+        fi
+
+        rx=${pattern:-$rx}
+        rx=${rx:-'(?<!-)\b\Q'"$pkgname"'\E[-_]?((src|source)[-_])?v?\K([^-/_\s]*?\d[^-/_\s]*?)(?=(?:[-_.](?:src|source|orig))?\.(?:[jt]ar|shar|t[bglx]z|tbz2|zip))\b'}
+
+        if [ "${fetchedurls[$url]}" ]; then
+            if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+                echo "already fetched $url" 1>&2
+            fi
+            continue
+        fi
+
+        if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+            echo "fetching $url and scanning with $rx" 1>&2
+        fi
+        curl -H 'Accept: text/html,application/xhtml+xml,application/xml,text/plain,application/rss+xml' -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$url" |
+            grep -Po -i "$rx"
+        fetchedurls[$url]=yes
+    done |
+    tr _ . |
+    sort -Vu |
+    {
+        grep . || echo "NO VERSION found for $original_pkgname" 1>&2
+    } |
+    while IFS= read -r found_version; do
+        if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+            echo "found version $found_version"
+        fi
+        consider=true
+        p="$ignore "
+        while [ -n "$p" ]; do
+            i=${p%% *}
+            p=${p#* }
+            case "$found_version" in
+            $i)
+                consider=false
+                if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
+                    echo "ignored $found_version due to $i"
+                fi
+            esac
+        done
+        if $consider; then
+            xbps-uhelper cmpver "$original_pkgname-${version}_1" \
+                "$original_pkgname-$(printf %s "$found_version" | tr - .)_1"
+            if [ $? = 255 ]; then
+                echo "${original_pkgname}-${version} -> ${original_pkgname}-${found_version}"
+            fi
+        fi
+    done
+}
 
 vars="pkgname|version|homepage|distfiles"
 template="$(
diff --git a/common/xbps-src/shutils/update_check.sh b/common/xbps-src/shutils/update_check.sh
deleted file mode 100644
index 0a624274f500..000000000000
--- a/common/xbps-src/shutils/update_check.sh
+++ /dev/null
@@ -1,210 +0,0 @@
-# vim: set ts=4 sw=4 et:
-
-update_check() {
-    local i p url pkgurlname rx found_version consider
-    local update_override=$XBPS_SRCPKGDIR/$XBPS_TARGET_PKG/update
-    local original_pkgname=$pkgname
-    local urlpfx urlsfx
-    local -A fetchedurls
-
-    if [ -r $update_override ]; then
-        . $update_override
-        if [ "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
-            echo "using $XBPS_TARGET_PKG/update overrides" 1>&2
-        fi
-    fi
-
-    if ! type curl >/dev/null 2>&1; then
-        echo "ERROR: cannot find \`curl' executable!"
-        return 1
-    fi
-
-    export LC_ALL=C
-
-    if [ -z "$site" ]; then
-        case "$distfiles" in
-            # only consider versions those exist in ftp.gnome.org
-            *ftp.gnome.org*) ;;
-            *)
-                printf '%s\n' "$homepage" ;;
-        esac
-        for i in $distfiles; do
-            printf '%s\n' "${i%/*}/"
-        done
-    else
-        printf '%s\n' "$site"
-    fi |
-    # filter loop: if version are "folder" name based,
-    # substitute original url by every folder based ones (expand)
-    while IFS= read -r url; do
-        # default case: don't rewrite url
-        printf '%s\n' "$url"
-        if [ "$single_directory" ]; then
-            continue
-        fi
-        rx=
-        urlpfx="${url}"
-        urlsfx=
-        dirpfx=
-        case "$url" in
-            *.voidlinux.*|\
-              *sourceforge.net/sourceforge*|\
-              *code.google.com*|*googlecode*|\
-              *launchpad.net*|\
-              *cpan.*|\
-              *pythonhosted.org*|\
-              *github.com*|\
-              *//gitlab.*|\
-              *bitbucket.org*|\
-              *ftp.gnome.org*|\
-              *kernel.org/pub/linux/kernel/*|\
-              *cran.r-project.org/src/contrib*|\
-              *rubygems.org*|\
-              *crates.io*|\
-              *codeberg.org*|\
-              *hg.sr.ht*|\
-              *git.sr.ht*)
-                continue
-                ;;
-            *)
-                vdpfx=${vdprefix:-"|v|\\Q$pkgname\\E"}
-                vdsfx=${vdsuffix:-"|\\.x"}
-                match=$(grep -Po "^[^/]+//[^/]+(/.+)?/($vdpfx)(?=[-_.0-9]*[0-9](?<!\\Q$pkgname\\E)($vdsfx)/)" <<< "$url")
-                if [ "$?" = 0 ]; then
-                    urlpfx="${match%/*}/"
-                    dirpfx="${match##*/}"
-                    urlsfx="${url#$urlpfx}"
-                    urlsfx="${urlsfx#*/}"
-                    rx="href=[\"']?(\\Q$urlpfx\\E)?\\.?/?\\K\\Q$dirpfx\\E[-_.0-9]*[0-9]($vdsfx)[\"'/]"
-                fi
-                ;;
-        esac
-        if [ "$rx" ]; then
-            # substitute url if needed
-            if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
-                echo "(folder) fetching $urlpfx and scanning with $rx" 1>&2
-            fi
-            skipdirs=
-            curl -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$urlpfx" |
-                grep -Po -i "$rx" |
-                # sort -V places 1.1/ before 1/, but 1A/ before 1.1A/
-                sed -e 's:$:A:' -e 's:/A$:A/:' | sort -Vru | sed -e 's:A/$:/A:' -e 's:A$::' |
-                while IFS= read -r newver; do
-                    newurl="${urlpfx}${newver}${urlsfx}"
-                    if [ "$newurl" = "$url" ]; then
-                        skipdirs=yes
-                    fi
-                    if [ -z "$skipdirs" ]; then
-                        printf '%s\n' "$newurl"
-                    fi
-                done
-        fi
-    done |
-    while IFS= read -r url; do
-        rx=
-        if [ -z "$site" ]; then
-            case "$url" in
-            *sourceforge.net/sourceforge*)
-                pkgurlname="$(printf %s "$url" | cut -d/ -f5)"
-                url="https://sourceforge.net/projects/$pkgurlname/rss?limit=200";;
-            *code.google.com*|*googlecode*)
-                url="http://code.google.com/p/$pkgname/downloads/list";;
-            *launchpad.net*)
-                pkgurlname="$(printf %s "$url" | cut -d/ -f4)"
-                url="https://launchpad.net/$pkgurlname/+download";;
-            *cpan.*)
-                pkgname=${pkgname#perl-};;
-            *pythonhosted.org*)
-                pkgname=${pkgname#python-}
-                pkgname=${pkgname#python3-}
-                url="https://pypi.org/simple/$pkgname";;
-            *github.com*)
-                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
-                url="https://github.com/$pkgurlname/tags"
-                rx='/archive/refs/tags/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
-            *//gitlab.*)
-                pkgurlname="$(printf %s "$url" | cut -d/ -f1-5)"
-                url="$pkgurlname/tags"
-                rx='/archive/[^/]+/\Q'"$pkgname"'\E-v?\K[\d.]+(?=\.tar\.gz")';;
-            *bitbucket.org*)
-                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
-                url="https://bitbucket.org/$pkgurlname/downloads"
-                rx='/(get|downloads)/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar)';;
-            *ftp.gnome.org*|*download.gnome.org*)
-                : ${pattern="\Q$pkgname\E-\K(0|[13]\.[0-9]*[02468]|[4-9][0-9]+)\.[0-9.]*[0-9](?=)"}
-                url="https://download.gnome.org/sources/$pkgname/cache.json";;
-            *kernel.org/pub/linux/kernel/*)
-                rx=linux-'\K'${version%.*}'[\d.]+(?=\.tar\.xz)';;
-            *cran.r-project.org/src/contrib*)
-                rx='\b\Q'"${pkgname#R-cran-}"'\E_\K\d+(\.\d+)*(-\d+)?(?=\.tar)';;
-            *rubygems.org*)
-                url="https://rubygems.org/gems/${pkgname#ruby-}"
-                rx='href="/gems/'${pkgname#ruby-}'/versions/\K[\d.]*(?=")' ;;
-            *crates.io*)
-                url="https://crates.io/api/v1/crates/${pkgname#rust-}"
-                rx='/crates/'${pkgname#rust-}'/\K[0-9.]*(?=/download)' ;;
-            *codeberg.org*)
-                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
-                url="https://codeberg.org/$pkgurlname/releases"
-                rx='/archive/\K[\d.]+(?=\.tar\.gz)' ;;
-            *hg.sr.ht*)
-                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
-                url="https://hg.sr.ht/$pkgurlname/tags"
-                rx='/archive/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
-            *git.sr.ht*)
-                pkgurlname="$(printf %s "$url" | cut -d/ -f4,5)"
-                url="https://git.sr.ht/$pkgurlname/refs"
-                rx='/archive/(v?|\Q'"$pkgname"'\E-)?\K[\d.]+(?=\.tar\.gz")';;
-            *pkgs.fedoraproject.org*)
-                url="https://pkgs.fedoraproject.org/repo/pkgs/$pkgname" ;;
-            esac
-        fi
-
-        rx=${pattern:-$rx}
-        rx=${rx:-'(?<!-)\b\Q'"$pkgname"'\E[-_]?((src|source)[-_])?v?\K([^-/_\s]*?\d[^-/_\s]*?)(?=(?:[-_.](?:src|source|orig))?\.(?:[jt]ar|shar|t[bglx]z|tbz2|zip))\b'}
-
-        if [ "${fetchedurls[$url]}" ]; then
-            if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
-                echo "already fetched $url" 1>&2
-            fi
-            continue
-        fi
-
-        if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
-            echo "fetching $url and scanning with $rx" 1>&2
-        fi
-        curl -H 'Accept: text/html,application/xhtml+xml,application/xml,text/plain,application/rss+xml' -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$url" |
-            grep -Po -i "$rx"
-        fetchedurls[$url]=yes
-    done |
-    tr _ . |
-    sort -Vu |
-    {
-        grep . || echo "NO VERSION found for $original_pkgname" 1>&2
-    } |
-    while IFS= read -r found_version; do
-        if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
-            echo "found version $found_version"
-        fi
-        consider=true
-        p="$ignore "
-        while [ -n "$p" ]; do
-            i=${p%% *}
-            p=${p#* }
-            case "$found_version" in
-            $i)
-                consider=false
-                if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
-                    echo "ignored $found_version due to $i"
-                fi
-            esac
-        done
-        if $consider; then
-            xbps-uhelper cmpver "$original_pkgname-${version}_1" \
-                "$original_pkgname-$(printf %s "$found_version" | tr - .)_1"
-            if [ $? = 255 ]; then
-                echo "${original_pkgname}-${version} -> ${original_pkgname}-${found_version}"
-            fi
-        fi
-    done
-}

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: move update-check out of xbps-src
  2021-06-22  1:11 [PR PATCH] move update-check out of xbps-src sgn
                   ` (4 preceding siblings ...)
  2021-06-23 16:36 ` [PR PATCH] [Updated] " sgn
@ 2021-06-23 17:35 ` Chocimier
  2022-05-23  2:12 ` github-actions
  2022-06-06  2:14 ` [PR PATCH] [Closed]: " github-actions
  7 siblings, 0 replies; 9+ messages in thread
From: Chocimier @ 2021-06-23 17:35 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 1770 bytes --]

New comment by Chocimier on void-packages repository

https://github.com/void-linux/void-packages/pull/31602#issuecomment-867029335

Comment:
> check only distfiles

homepage and site are important too

> remove all variable except pkgname and version from distfiles

There are templates that use wrksrc, _distver being custom transformation of version and more.

> update-check doesn't need xbps 

It uses dewey version sort with `xbps-uhelper cmpver` and needs to use it more. Current [sort -Vu](https://github.com/void-linux/void-packages/blob/471c2d1dd4af758dcd5a450f498439dc25045e34/common/xbps-src/shutils/update_check.sh#L179) sorts beta releases as greater than final releases - so that xupdate updates templates that for some reasons are in `2beta1` version to `2beta2` rather than `2`.

> requires [...] XBPS_ALLOW_CHROOT_BREAKOUT

Is this problem?

> could speed things up a bit

Test was done on templates with `0.` prepended to version to force matches.
There are around 60 packages for which new versions are not detected after that change as for now, not counting R and ruby packages defining distfiles in build style.
Run time of void-updates -p 5 went down by 5min, from 2h 25min to 2h 20min. On same machine, `xbps-src show` for all packages takes around 10 min - limit of what can be expected to improve.
Roughly half of execution time was spent in void-updates' `get_date` - but this is not representative, as there are more found versions that usually.

My conclusion is that this is micro-optimization that doesn't touch real problem pointed by maldridge  - templates being bash programs depending on lot of xbps-src code that are complex to really understand, hard or slow to process for all usecases, like cycle detection.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: move update-check out of xbps-src
  2021-06-22  1:11 [PR PATCH] move update-check out of xbps-src sgn
                   ` (5 preceding siblings ...)
  2021-06-23 17:35 ` Chocimier
@ 2022-05-23  2:12 ` github-actions
  2022-06-06  2:14 ` [PR PATCH] [Closed]: " github-actions
  7 siblings, 0 replies; 9+ messages in thread
From: github-actions @ 2022-05-23  2:12 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 305 bytes --]

New comment by github-actions[bot] on void-packages repository

https://github.com/void-linux/void-packages/pull/31602#issuecomment-1134091762

Comment:
Pull Requests become stale 90 days after last activity and are closed 14 days after that.  If this pull request is still relevant bump it or assign it.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PR PATCH] [Closed]: move update-check out of xbps-src
  2021-06-22  1:11 [PR PATCH] move update-check out of xbps-src sgn
                   ` (6 preceding siblings ...)
  2022-05-23  2:12 ` github-actions
@ 2022-06-06  2:14 ` github-actions
  7 siblings, 0 replies; 9+ messages in thread
From: github-actions @ 2022-06-06  2:14 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]

There's a closed pull request on the void-packages repository

move update-check out of xbps-src
https://github.com/void-linux/void-packages/pull/31602

Description:
Continue discussion from #31546 

It's better to rewrite in a real programming language, however this is enough for now.

TODO: check only distfiles.

The `eval` step should be removed when we can remove all variable except `pkgname` and `version` from `distfiles`, and merge the first `distfiles` to the same line of `distfiles=`

@void-linux/pkg-committers 
@the-maldridge moving out of `xbps-src`, we can drop the weird `XBPS_ALLOW_CHROOT_BREAKOUT=yes` from `update-check` container.

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2022-06-06  2:14 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-22  1:11 [PR PATCH] move update-check out of xbps-src sgn
2021-06-22  2:16 ` the-maldridge
2021-06-22  7:13 ` Gottox
2021-06-22  7:42 ` the-maldridge
2021-06-22 11:05 ` sgn
2021-06-23 16:36 ` [PR PATCH] [Updated] " sgn
2021-06-23 17:35 ` Chocimier
2022-05-23  2:12 ` github-actions
2022-06-06  2:14 ` [PR PATCH] [Closed]: " github-actions

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).