* [PR PATCH] xbps-src: check for existing packages before syncing chroot repodata
@ 2022-06-17 10:23 JamiKettunen
2022-06-17 16:53 ` Chocimier
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: JamiKettunen @ 2022-06-17 10:23 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 2731 bytes --]
There is a new pull request by JamiKettunen against master on the void-packages repository
https://github.com/JamiKettunen/void-packages faster-skip-existing-build
https://github.com/void-linux/void-packages/pull/37595
xbps-src: check for existing packages before syncing chroot repodata
#### Testing the changes
- I tested the changes in this PR: **briefly**
This significantly speeds up the package building process for build scripts etc. that run `./xbps-src pkg ...` on a long list of packages that may already have up-to-date local packages built when configured with `XBPS_PRESERVE_PKGS` set.
#### Example with a single package
Before this patch:
```
void-packages $ time ./xbps-src pkg helvum
=> xbps-src: updating repositories for host (x86_64)...
[*] Updating repository `https://repo-default.voidlinux.org/current/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/nonfree/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/debug/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/multilib/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/multilib/nonfree/x86_64-repodata' ...
=> xbps-src: updating software in / masterdir...
=> xbps-src: cleaning up / masterdir...
=> helvum-0.3.4_2: removing autodeps, please wait...
=> WARNING: helvum-0.3.4_2: skipping build due to existing /host/binpkgs//helvum-0.3.4_2.x86_64.xbps
real 0m1.980s
user 0m0.504s
sys 0m0.113s
```
After this patch:
```
void-packages $ time ./xbps-src pkg helvum
=> WARNING: helvum-0.3.4_2: skipping build due to existing /host/binpkgs//helvum-0.3.4_2.x86_64.xbps
real 0m0.062s
user 0m0.041s
sys 0m0.024s
```
Previously this would add up to about a minute of waiting for essentially no reason if 30 packages already have up-to-date local packages, while with this patch it would only take about 2 seconds to come to the same conclusion with the same package list.
I still have one question: what to do with the existing call to `check_existing_pkg` in [`common/xbps-src/libexec/build.sh`](https://github.com/void-linux/void-packages/blob/master/common/xbps-src/libexec/build.sh#L38)? The script appears to be executed from a [few](https://github.com/void-linux/void-packages/tree/master/common/xbps-src/shutils/cross.sh) [other](https://github.com/void-linux/void-packages/tree/master/common/xbps-src/shutils/build_dependencies.sh) [scripts](https://github.com/void-linux/void-packages/tree/master/common/xbps-src/shutils/bulk.sh) as well so I left it alone for now.
A patch file from https://github.com/void-linux/void-packages/pull/37595.patch is attached
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-faster-skip-existing-build-37595.patch --]
[-- Type: text/x-diff, Size: 1006 bytes --]
From f6117695ac78ad078a5c441e329947e1ddd17357 Mon Sep 17 00:00:00 2001
From: Jami Kettunen <jami.kettunen@protonmail.com>
Date: Fri, 17 Jun 2022 13:07:44 +0300
Subject: [PATCH] xbps-src: check for existing packages before syncing chroot
repodata
This significantly speeds up the package building process for build
scripts etc. that run './xbps-src pkg ...' on a long list of packages
that may already have up-to-date local packages built when configured
with XBPS_PRESERVE_PKGS set.
---
xbps-src | 1 +
1 file changed, 1 insertion(+)
diff --git a/xbps-src b/xbps-src
index dfa059927cc2..4e8349091708 100755
--- a/xbps-src
+++ b/xbps-src
@@ -806,6 +806,7 @@ case "$XBPS_TARGET" in
if [ -n "$CHROOT_READY" -a -z "$IN_CHROOT" ]; then
chroot_handler $XBPS_TARGET $XBPS_TARGET_PKG
else
+ check_existing_pkg
chroot_sync_repodata
# prevent update_base_chroot from removing the builddir/destdir
update_base_chroot keep-all-force
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: xbps-src: check for existing packages before syncing chroot repodata
2022-06-17 10:23 [PR PATCH] xbps-src: check for existing packages before syncing chroot repodata JamiKettunen
@ 2022-06-17 16:53 ` Chocimier
2022-06-17 17:02 ` JamiKettunen
2022-06-17 23:56 ` [PR PATCH] [Merged]: " the-maldridge
2 siblings, 0 replies; 4+ messages in thread
From: Chocimier @ 2022-06-17 16:53 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 203 bytes --]
New comment by Chocimier on void-packages repository
https://github.com/void-linux/void-packages/pull/37595#issuecomment-1159063306
Comment:
Looks good to me. Leaving check in other places is correct.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: xbps-src: check for existing packages before syncing chroot repodata
2022-06-17 10:23 [PR PATCH] xbps-src: check for existing packages before syncing chroot repodata JamiKettunen
2022-06-17 16:53 ` Chocimier
@ 2022-06-17 17:02 ` JamiKettunen
2022-06-17 23:56 ` [PR PATCH] [Merged]: " the-maldridge
2 siblings, 0 replies; 4+ messages in thread
From: JamiKettunen @ 2022-06-17 17:02 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 195 bytes --]
New comment by JamiKettunen on void-packages repository
https://github.com/void-linux/void-packages/pull/37595#issuecomment-1159071897
Comment:
Cool, I'll mark this as ready for review then :)
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PR PATCH] [Merged]: xbps-src: check for existing packages before syncing chroot repodata
2022-06-17 10:23 [PR PATCH] xbps-src: check for existing packages before syncing chroot repodata JamiKettunen
2022-06-17 16:53 ` Chocimier
2022-06-17 17:02 ` JamiKettunen
@ 2022-06-17 23:56 ` the-maldridge
2 siblings, 0 replies; 4+ messages in thread
From: the-maldridge @ 2022-06-17 23:56 UTC (permalink / raw)
To: ml
[-- Attachment #1: Type: text/plain, Size: 2555 bytes --]
There's a merged pull request on the void-packages repository
xbps-src: check for existing packages before syncing chroot repodata
https://github.com/void-linux/void-packages/pull/37595
Description:
#### Testing the changes
- I tested the changes in this PR: **briefly**
This significantly speeds up the package building process for build scripts etc. that run `./xbps-src pkg ...` on a long list of packages that may already have up-to-date local packages built when configured with `XBPS_PRESERVE_PKGS` set.
~~I still have one question: what to do with the existing call to `check_existing_pkg` in [`common/xbps-src/libexec/build.sh`](https://github.com/void-linux/void-packages/blob/master/common/xbps-src/libexec/build.sh#L38)? The script appears to be executed from a [few](https://github.com/void-linux/void-packages/tree/master/common/xbps-src/shutils/cross.sh) [other](https://github.com/void-linux/void-packages/tree/master/common/xbps-src/shutils/build_dependencies.sh) [scripts](https://github.com/void-linux/void-packages/tree/master/common/xbps-src/shutils/bulk.sh) as well so I left it alone for now.~~
#### Example with a single package
Before this patch:
```
void-packages $ time ./xbps-src pkg helvum
=> xbps-src: updating repositories for host (x86_64)...
[*] Updating repository `https://repo-default.voidlinux.org/current/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/nonfree/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/debug/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/multilib/x86_64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/multilib/nonfree/x86_64-repodata' ...
=> xbps-src: updating software in / masterdir...
=> xbps-src: cleaning up / masterdir...
=> helvum-0.3.4_2: removing autodeps, please wait...
=> WARNING: helvum-0.3.4_2: skipping build due to existing /host/binpkgs//helvum-0.3.4_2.x86_64.xbps
real 0m1.980s
user 0m0.504s
sys 0m0.113s
```
After this patch:
```
void-packages $ time ./xbps-src pkg helvum
=> WARNING: helvum-0.3.4_2: skipping build due to existing /host/binpkgs//helvum-0.3.4_2.x86_64.xbps
real 0m0.062s
user 0m0.041s
sys 0m0.024s
```
Previously this would add up to about a minute of waiting for essentially no reason if 30 packages already have up-to-date local packages, while with this patch it would only take about 2 seconds to come to the same conclusion with the same package list.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2022-06-17 23:56 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-06-17 10:23 [PR PATCH] xbps-src: check for existing packages before syncing chroot repodata JamiKettunen
2022-06-17 16:53 ` Chocimier
2022-06-17 17:02 ` JamiKettunen
2022-06-17 23:56 ` [PR PATCH] [Merged]: " the-maldridge
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).