[PR PATCH] xbps-src: allow curl to request compressed responses in update-check

[PR PATCH] xbps-src: allow curl to request compressed responses in update-check

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 783 bytes --]

There is a new pull request by ahesford against master on the void-packages repository

https://github.com/ahesford/void-packages update-check
https://github.com/void-linux/void-packages/pull/49391

xbps-src: allow curl to request compressed responses in update-check
The `python3` and `python3.11` update-check keeps failing because python.org is returning a compressed response that thwarts pattern matching. Adding `--compressed` to the arguments for curl will allow the client to request compressed responses and, more importantly, will attempt to decompress them when received. This fixes the Python update checks.

#### Testing the changes
- I tested the changes in this PR: **YES**

A patch file from https://github.com/void-linux/void-packages/pull/49391.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-update-check-49391.patch --]
[-- Type: text/x-diff, Size: 1969 bytes --]

From b4163da6ebaba0f1d8934dc0db22b742c4177613 Mon Sep 17 00:00:00 2001
From: "Andrew J. Hesford" <ajh@sideband.org>
Date: Wed, 20 Mar 2024 09:34:47 -0400
Subject: [PATCH] xbps-src: allow curl to request compressed responses in
 update-check

---
 common/xbps-src/shutils/update_check.sh | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/common/xbps-src/shutils/update_check.sh b/common/xbps-src/shutils/update_check.sh
index 8bae017bbe4225..4747bd4a1327c3 100644
--- a/common/xbps-src/shutils/update_check.sh
+++ b/common/xbps-src/shutils/update_check.sh
@@ -8,6 +8,11 @@ update_check() {
     local urlpfx urlsfx
     local -A fetchedurls
 
+    local curlargs=(
+        -A "xbps-src-update-check/$XBPS_SRC_VERSION"
+        --max-time 10 --compressed -Lsk
+   )
+
     # XBPS_UPDATE_CHECK_VERBOSE is the old way to show verbose messages
     [ "$XBPS_UPDATE_CHECK_VERBOSE" ] && XBPS_VERBOSE="$XBPS_UPDATE_CHECK_VERBOSE"
 
@@ -95,7 +100,7 @@ update_check() {
             # substitute url if needed
             msg_verbose "(folder) fetching $urlpfx and scanning with $rx\n"
             skipdirs=
-            curl -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$urlpfx" |
+            curl "${curlargs[@]}" "$urlpfx" |
                 grep -Po -i "$rx" |
                 # sort -V places 1.1/ before 1/, but 1A/ before 1.1A/
                 sed -e 's:$:A:' -e 's:/A$:A/:' | sort -Vru | sed -e 's:A/$:/A:' -e 's:A$::' |
@@ -197,7 +202,7 @@ update_check() {
         fi
 
         msg_verbose "fetching $url and scanning with $rx\n"
-        curl -H 'Accept: text/html,application/xhtml+xml,application/xml,text/plain,application/rss+xml' -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$url" |
+        curl "${curlargs[@]}" -H 'Accept: text/html,application/xhtml+xml,application/xml,text/plain,application/rss+xml' "$url" |
             grep -Po -i "$rx"
         fetchedurls[$url]=yes
     done |

Re: [PR PATCH] [Updated] xbps-src: allow curl to request compressed responses in update-check

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 788 bytes --]

There is an updated pull request by ahesford against master on the void-packages repository

https://github.com/ahesford/void-packages update-check
https://github.com/void-linux/void-packages/pull/49391

xbps-src: allow curl to request compressed responses in update-check
The `python3` and `python3.11` update-check keeps failing because python.org is returning a compressed response that thwarts pattern matching. Adding `--compressed` to the arguments for curl will allow the client to request compressed responses and, more importantly, will attempt to decompress them when received. This fixes the Python update checks.

#### Testing the changes
- I tested the changes in this PR: **YES**

A patch file from https://github.com/void-linux/void-packages/pull/49391.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-update-check-49391.patch --]
[-- Type: text/x-diff, Size: 1969 bytes --]

From 68f761dc2c8259a5ae208c5b0713ff9e252b548f Mon Sep 17 00:00:00 2001
From: "Andrew J. Hesford" <ajh@sideband.org>
Date: Wed, 20 Mar 2024 09:34:47 -0400
Subject: [PATCH] xbps-src: allow curl to request compressed responses in
 update-check

---
 common/xbps-src/shutils/update_check.sh | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/common/xbps-src/shutils/update_check.sh b/common/xbps-src/shutils/update_check.sh
index 8bae017bbe4225..4747bd4a1327c3 100644
--- a/common/xbps-src/shutils/update_check.sh
+++ b/common/xbps-src/shutils/update_check.sh
@@ -8,6 +8,11 @@ update_check() {
     local urlpfx urlsfx
     local -A fetchedurls
 
+    local curlargs=(
+        -A "xbps-src-update-check/$XBPS_SRC_VERSION"
+        --max-time 10 --compressed -Lsk
+   )
+
     # XBPS_UPDATE_CHECK_VERBOSE is the old way to show verbose messages
     [ "$XBPS_UPDATE_CHECK_VERBOSE" ] && XBPS_VERBOSE="$XBPS_UPDATE_CHECK_VERBOSE"
 
@@ -95,7 +100,7 @@ update_check() {
             # substitute url if needed
             msg_verbose "(folder) fetching $urlpfx and scanning with $rx\n"
             skipdirs=
-            curl -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$urlpfx" |
+            curl "${curlargs[@]}" "$urlpfx" |
                 grep -Po -i "$rx" |
                 # sort -V places 1.1/ before 1/, but 1A/ before 1.1A/
                 sed -e 's:$:A:' -e 's:/A$:A/:' | sort -Vru | sed -e 's:A/$:/A:' -e 's:A$::' |
@@ -197,7 +202,7 @@ update_check() {
         fi
 
         msg_verbose "fetching $url and scanning with $rx\n"
-        curl -H 'Accept: text/html,application/xhtml+xml,application/xml,text/plain,application/rss+xml' -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$url" |
+        curl "${curlargs[@]}" -H 'Accept: text/html,application/xhtml+xml,application/xml,text/plain,application/rss+xml' "$url" |
             grep -Po -i "$rx"
         fetchedurls[$url]=yes
     done |

Re: [PR PATCH] [Merged]: xbps-src: allow curl to request compressed responses in update-check

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 625 bytes --]

There's a merged pull request on the void-packages repository

xbps-src: allow curl to request compressed responses in update-check
https://github.com/void-linux/void-packages/pull/49391

Description:
The `python3` and `python3.11` update-check keeps failing because python.org is returning a compressed response that thwarts pattern matching. Adding `--compressed` to the arguments for curl will allow the client to request compressed responses and, more importantly, will attempt to decompress them when received. This fixes the Python update checks.

#### Testing the changes
- I tested the changes in this PR: **YES**