[PR PATCH] [RFC] xbps-src: allow JSON in update_check

[PR PATCH] [RFC] xbps-src: allow JSON in update_check

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 923 bytes --]

There is a new pull request by ahesford against master on the void-packages repository

https://github.com/ahesford/void-packages json
https://github.com/void-linux/void-packages/pull/24456

[RFC] xbps-src: allow JSON in update_check
The `update_check` function restricts content types from sites that provide version lists. Adding `application/json` as an accepted type allows querying sites like

https://api.github.com/repos/<org>/<repo>/tags

which provide more complete listings of tags than the default locations that return HTML. This can be important for projects that provide a large number of prerelease tarballs, because they can flood the default tag or release pages and prevent proper version detection.

The `python3-bokeh` `update` definition takes advantage of this new capability to properly track versions.

A patch file from https://github.com/void-linux/void-packages/pull/24456.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-json-24456.patch --]
[-- Type: text/x-diff, Size: 2259 bytes --]

From 96abdaf3a88b1c6df6d0c5c3ff87bdc7c0eb58ef Mon Sep 17 00:00:00 2001
From: "Andrew J. Hesford" <ajh@sideband.org>
Date: Mon, 24 Aug 2020 11:16:11 -0400
Subject: [PATCH 1/2] xbps-src: allow JSON in update_check

The `update_check` function restricts content types from sites that
provide version lists. Adding `application/json` as an accepted type
allows querying sites like

https://api.github.com/repos/<org>/<repo>/tags

which provide more complete listings of tags than the default locations
that return HTML. This can be important for projects that provide a
large number of prerelease tarballs, because they can flood the default
tag or release pages and prevent proper version detection.
---
 common/xbps-src/shutils/update_check.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/common/xbps-src/shutils/update_check.sh b/common/xbps-src/shutils/update_check.sh
index b450c57beea..7112c2de8b4 100644
--- a/common/xbps-src/shutils/update_check.sh
+++ b/common/xbps-src/shutils/update_check.sh
@@ -161,7 +161,7 @@ update_check() {
         if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
             echo "fetching $url" 1>&2
         fi
-        curl -H 'Accept: text/html,application/xhtml+xml,application/xml,text/plain,application/rss+xml' -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$url" |
+        curl -H 'Accept: text/html,application/xhtml+xml,application/xml,text/plain,application/rss+xml,application/json' -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$url" |
             grep -Po -i "$rx"
         fetchedurls[$url]=yes
     done |

From a0fd88a0ee4d2148288be96fe489dc7e3742c25a Mon Sep 17 00:00:00 2001
From: "Andrew J. Hesford" <ajh@sideband.org>
Date: Mon, 24 Aug 2020 11:19:58 -0400
Subject: [PATCH 2/2] python3-bokeh: use api.github.com for complete version
 listing

---
 srcpkgs/python3-bokeh/update | 2 ++
 1 file changed, 2 insertions(+)
 create mode 100644 srcpkgs/python3-bokeh/update

diff --git a/srcpkgs/python3-bokeh/update b/srcpkgs/python3-bokeh/update
new file mode 100644
index 00000000000..4d9db4290e9
--- /dev/null
+++ b/srcpkgs/python3-bokeh/update
@@ -0,0 +1,2 @@
+site="https://api.github.com/repos/bokeh/bokeh/tags"
+pattern='"name":\s*"\K[0-9.]+(?=")'

Re: [RFC] xbps-src: allow JSON in update_check

[leahneukirchen]

[-- Attachment #1: Type: text/plain, Size: 268 bytes --]

New comment by leahneukirchen on void-packages repository

https://github.com/void-linux/void-packages/pull/24456#issuecomment-679194117

Comment:
Using `https://api.github.com/` will result in running into API limits very quickly, we don't use that API for a reason.

Re: [RFC] xbps-src: allow JSON in update_check

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 283 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/pull/24456#issuecomment-679195687

Comment:
Makes sense. I missed https://docs.github.com/en/rest/overview/resources-in-the-rest-api#rate-limiting 

Sixty requests per hour is brutal!

Re: [PR PATCH] [Closed]: [RFC] xbps-src: allow JSON in update_check

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 775 bytes --]

There's a closed pull request on the void-packages repository

[RFC] xbps-src: allow JSON in update_check
https://github.com/void-linux/void-packages/pull/24456

Description:
The `update_check` function restricts content types from sites that provide version lists. Adding `application/json` as an accepted type allows querying sites like

https://api.github.com/repos/<org\>/<repo\>/tags

which provide more complete listings of tags than the default locations that return HTML. This can be important for projects that provide a large number of prerelease tarballs, because they can flood the default tag or release pages and prevent proper version detection.

The `python3-bokeh` `update` definition takes advantage of this new capability to properly track versions.