Github messages for voidlinux
 help / color / mirror / Atom feed
* [PR PATCH] xbps-src: check for existing packages before syncing chroot repodata
@ 2022-06-17 10:23 JamiKettunen
  2022-06-17 16:53 ` Chocimier
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: JamiKettunen @ 2022-06-17 10:23 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 2731 bytes --]

There is a new pull request by JamiKettunen against master on the void-packages repository

https://github.com/JamiKettunen/void-packages faster-skip-existing-build
https://github.com/void-linux/void-packages/pull/37595

xbps-src: check for existing packages before syncing chroot repodata
#### Testing the changes
- I tested the changes in this PR: **briefly**

This significantly speeds up the package building process for build scripts etc. that run `./xbps-src pkg ...` on a long list of packages that may already have up-to-date local packages built when configured with `XBPS_PRESERVE_PKGS` set.

#### Example with a single package
Before this patch:
```
void-packages $ time ./xbps-src pkg helvum
=> xbps-src: updating repositories for host (x86_64)...
[*] Updating repository `https://repo-default.voidlinux.org/current/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/nonfree/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/debug/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/multilib/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/multilib/nonfree/x86_64-repodata' ...
=> xbps-src: updating software in / masterdir...
=> xbps-src: cleaning up / masterdir...
=> helvum-0.3.4_2: removing autodeps, please wait...
=> WARNING: helvum-0.3.4_2: skipping build due to existing /host/binpkgs//helvum-0.3.4_2.x86_64.xbps

real	0m1.980s
user	0m0.504s
sys	0m0.113s
```
After this patch:
```
void-packages $ time ./xbps-src pkg helvum
=> WARNING: helvum-0.3.4_2: skipping build due to existing /host/binpkgs//helvum-0.3.4_2.x86_64.xbps

real	0m0.062s
user	0m0.041s
sys	0m0.024s
```
Previously this would add up to about a minute of waiting for essentially no reason if 30 packages already have up-to-date local packages, while with this patch it would only take about 2 seconds to come to the same conclusion with the same package list.

I still have one question: what to do with the existing call to `check_existing_pkg` in [`common/xbps-src/libexec/build.sh`](https://github.com/void-linux/void-packages/blob/master/common/xbps-src/libexec/build.sh#L38)? The script appears to be executed from a [few](https://github.com/void-linux/void-packages/tree/master/common/xbps-src/shutils/cross.sh) [other](https://github.com/void-linux/void-packages/tree/master/common/xbps-src/shutils/build_dependencies.sh) [scripts](https://github.com/void-linux/void-packages/tree/master/common/xbps-src/shutils/bulk.sh) as well so I left it alone for now.

A patch file from https://github.com/void-linux/void-packages/pull/37595.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-faster-skip-existing-build-37595.patch --]
[-- Type: text/x-diff, Size: 1006 bytes --]

From f6117695ac78ad078a5c441e329947e1ddd17357 Mon Sep 17 00:00:00 2001
From: Jami Kettunen <jami.kettunen@protonmail.com>
Date: Fri, 17 Jun 2022 13:07:44 +0300
Subject: [PATCH] xbps-src: check for existing packages before syncing chroot
 repodata

This significantly speeds up the package building process for build
scripts etc. that run './xbps-src pkg ...' on a long list of packages
that may already have up-to-date local packages built when configured
with XBPS_PRESERVE_PKGS set.
---
 xbps-src | 1 +
 1 file changed, 1 insertion(+)

diff --git a/xbps-src b/xbps-src
index dfa059927cc2..4e8349091708 100755
--- a/xbps-src
+++ b/xbps-src
@@ -806,6 +806,7 @@ case "$XBPS_TARGET" in
         if [ -n "$CHROOT_READY" -a -z "$IN_CHROOT" ]; then
             chroot_handler $XBPS_TARGET $XBPS_TARGET_PKG
         else
+            check_existing_pkg
             chroot_sync_repodata
             # prevent update_base_chroot from removing the builddir/destdir
             update_base_chroot keep-all-force

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: xbps-src: check for existing packages before syncing chroot repodata
  2022-06-17 10:23 [PR PATCH] xbps-src: check for existing packages before syncing chroot repodata JamiKettunen
@ 2022-06-17 16:53 ` Chocimier
  2022-06-17 17:02 ` JamiKettunen
  2022-06-17 23:56 ` [PR PATCH] [Merged]: " the-maldridge
  2 siblings, 0 replies; 4+ messages in thread
From: Chocimier @ 2022-06-17 16:53 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 203 bytes --]

New comment by Chocimier on void-packages repository

https://github.com/void-linux/void-packages/pull/37595#issuecomment-1159063306

Comment:
Looks good to me. Leaving check in other places is correct.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: xbps-src: check for existing packages before syncing chroot repodata
  2022-06-17 10:23 [PR PATCH] xbps-src: check for existing packages before syncing chroot repodata JamiKettunen
  2022-06-17 16:53 ` Chocimier
@ 2022-06-17 17:02 ` JamiKettunen
  2022-06-17 23:56 ` [PR PATCH] [Merged]: " the-maldridge
  2 siblings, 0 replies; 4+ messages in thread
From: JamiKettunen @ 2022-06-17 17:02 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 195 bytes --]

New comment by JamiKettunen on void-packages repository

https://github.com/void-linux/void-packages/pull/37595#issuecomment-1159071897

Comment:
Cool, I'll mark this as ready for review then :)

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PR PATCH] [Merged]: xbps-src: check for existing packages before syncing chroot repodata
  2022-06-17 10:23 [PR PATCH] xbps-src: check for existing packages before syncing chroot repodata JamiKettunen
  2022-06-17 16:53 ` Chocimier
  2022-06-17 17:02 ` JamiKettunen
@ 2022-06-17 23:56 ` the-maldridge
  2 siblings, 0 replies; 4+ messages in thread
From: the-maldridge @ 2022-06-17 23:56 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 2555 bytes --]

There's a merged pull request on the void-packages repository

xbps-src: check for existing packages before syncing chroot repodata
https://github.com/void-linux/void-packages/pull/37595

Description:
#### Testing the changes
- I tested the changes in this PR: **briefly**

This significantly speeds up the package building process for build scripts etc. that run `./xbps-src pkg ...` on a long list of packages that may already have up-to-date local packages built when configured with `XBPS_PRESERVE_PKGS` set.

~~I still have one question: what to do with the existing call to `check_existing_pkg` in [`common/xbps-src/libexec/build.sh`](https://github.com/void-linux/void-packages/blob/master/common/xbps-src/libexec/build.sh#L38)? The script appears to be executed from a [few](https://github.com/void-linux/void-packages/tree/master/common/xbps-src/shutils/cross.sh) [other](https://github.com/void-linux/void-packages/tree/master/common/xbps-src/shutils/build_dependencies.sh) [scripts](https://github.com/void-linux/void-packages/tree/master/common/xbps-src/shutils/bulk.sh) as well so I left it alone for now.~~

#### Example with a single package
Before this patch:
```
void-packages $ time ./xbps-src pkg helvum
=> xbps-src: updating repositories for host (x86_64)...
[*] Updating repository `https://repo-default.voidlinux.org/current/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/nonfree/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/debug/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/multilib/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/multilib/nonfree/x86_64-repodata' ...
=> xbps-src: updating software in / masterdir...
=> xbps-src: cleaning up / masterdir...
=> helvum-0.3.4_2: removing autodeps, please wait...
=> WARNING: helvum-0.3.4_2: skipping build due to existing /host/binpkgs//helvum-0.3.4_2.x86_64.xbps

real	0m1.980s
user	0m0.504s
sys	0m0.113s
```
After this patch:
```
void-packages $ time ./xbps-src pkg helvum
=> WARNING: helvum-0.3.4_2: skipping build due to existing /host/binpkgs//helvum-0.3.4_2.x86_64.xbps

real	0m0.062s
user	0m0.041s
sys	0m0.024s
```
Previously this would add up to about a minute of waiting for essentially no reason if 30 packages already have up-to-date local packages, while with this patch it would only take about 2 seconds to come to the same conclusion with the same package list.

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2022-06-17 23:56 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-06-17 10:23 [PR PATCH] xbps-src: check for existing packages before syncing chroot repodata JamiKettunen
2022-06-17 16:53 ` Chocimier
2022-06-17 17:02 ` JamiKettunen
2022-06-17 23:56 ` [PR PATCH] [Merged]: " the-maldridge

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).