Github messages for voidlinux
 help / color / mirror / Atom feed
From: JamiKettunen <JamiKettunen@users.noreply.github.com>
To: ml@inbox.vuxu.org
Subject: [PR PATCH] xbps-src: check for existing packages before syncing chroot repodata
Date: Fri, 17 Jun 2022 12:23:45 +0200	[thread overview]
Message-ID: <gh-mailinglist-notifications-41a7ca26-5023-4802-975b-f1789d68868e-void-packages-37595@inbox.vuxu.org> (raw)

[-- Attachment #1: Type: text/plain, Size: 2731 bytes --]

There is a new pull request by JamiKettunen against master on the void-packages repository

https://github.com/JamiKettunen/void-packages faster-skip-existing-build
https://github.com/void-linux/void-packages/pull/37595

xbps-src: check for existing packages before syncing chroot repodata
#### Testing the changes
- I tested the changes in this PR: **briefly**

This significantly speeds up the package building process for build scripts etc. that run `./xbps-src pkg ...` on a long list of packages that may already have up-to-date local packages built when configured with `XBPS_PRESERVE_PKGS` set.

#### Example with a single package
Before this patch:
```
void-packages $ time ./xbps-src pkg helvum
=> xbps-src: updating repositories for host (x86_64)...
[*] Updating repository `https://repo-default.voidlinux.org/current/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/nonfree/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/debug/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/multilib/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/multilib/nonfree/x86_64-repodata' ...
=> xbps-src: updating software in / masterdir...
=> xbps-src: cleaning up / masterdir...
=> helvum-0.3.4_2: removing autodeps, please wait...
=> WARNING: helvum-0.3.4_2: skipping build due to existing /host/binpkgs//helvum-0.3.4_2.x86_64.xbps

real	0m1.980s
user	0m0.504s
sys	0m0.113s
```
After this patch:
```
void-packages $ time ./xbps-src pkg helvum
=> WARNING: helvum-0.3.4_2: skipping build due to existing /host/binpkgs//helvum-0.3.4_2.x86_64.xbps

real	0m0.062s
user	0m0.041s
sys	0m0.024s
```
Previously this would add up to about a minute of waiting for essentially no reason if 30 packages already have up-to-date local packages, while with this patch it would only take about 2 seconds to come to the same conclusion with the same package list.

I still have one question: what to do with the existing call to `check_existing_pkg` in [`common/xbps-src/libexec/build.sh`](https://github.com/void-linux/void-packages/blob/master/common/xbps-src/libexec/build.sh#L38)? The script appears to be executed from a [few](https://github.com/void-linux/void-packages/tree/master/common/xbps-src/shutils/cross.sh) [other](https://github.com/void-linux/void-packages/tree/master/common/xbps-src/shutils/build_dependencies.sh) [scripts](https://github.com/void-linux/void-packages/tree/master/common/xbps-src/shutils/bulk.sh) as well so I left it alone for now.

A patch file from https://github.com/void-linux/void-packages/pull/37595.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-faster-skip-existing-build-37595.patch --]
[-- Type: text/x-diff, Size: 1006 bytes --]

From f6117695ac78ad078a5c441e329947e1ddd17357 Mon Sep 17 00:00:00 2001
From: Jami Kettunen <jami.kettunen@protonmail.com>
Date: Fri, 17 Jun 2022 13:07:44 +0300
Subject: [PATCH] xbps-src: check for existing packages before syncing chroot
 repodata

This significantly speeds up the package building process for build
scripts etc. that run './xbps-src pkg ...' on a long list of packages
that may already have up-to-date local packages built when configured
with XBPS_PRESERVE_PKGS set.
---
 xbps-src | 1 +
 1 file changed, 1 insertion(+)

diff --git a/xbps-src b/xbps-src
index dfa059927cc2..4e8349091708 100755
--- a/xbps-src
+++ b/xbps-src
@@ -806,6 +806,7 @@ case "$XBPS_TARGET" in
         if [ -n "$CHROOT_READY" -a -z "$IN_CHROOT" ]; then
             chroot_handler $XBPS_TARGET $XBPS_TARGET_PKG
         else
+            check_existing_pkg
             chroot_sync_repodata
             # prevent update_base_chroot from removing the builddir/destdir
             update_base_chroot keep-all-force

             reply	other threads:[~2022-06-17 10:23 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-06-17 10:23 JamiKettunen [this message]
2022-06-17 16:53 ` Chocimier
2022-06-17 17:02 ` JamiKettunen
2022-06-17 23:56 ` [PR PATCH] [Merged]: " the-maldridge

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=gh-mailinglist-notifications-41a7ca26-5023-4802-975b-f1789d68868e-void-packages-37595@inbox.vuxu.org \
    --to=jamikettunen@users.noreply.github.com \
    --cc=ml@inbox.vuxu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).