[ISSUE] Incremental compilation support (xbps-src)?

[ISSUE] Incremental compilation support (xbps-src)?

[Animeshz]

[-- Attachment #1: Type: text/plain, Size: 664 bytes --]

New issue by Animeshz on void-packages repository

https://github.com/void-linux/void-packages/issues/41008

Description:
I don't know if this is a valid issue, but compiling big projects like kernel/chromium etc. takes a while, and without cache preservation they take forever to build different versions.

If there could be a incremental compilation support in xbps-src as it already creates a stateful `masterdir`, it could really boost few aspects, such as testing for issues in different versions, and [producing patches](https://www.reddit.com/r/voidlinux/comments/z0mfbh/xbpssrc_creating_patches_for_a_new_package) for them.

Let me know your thoughts!

Re: Incremental compilation support (xbps-src)?

[Animeshz]

[-- Attachment #1: Type: text/plain, Size: 323 bytes --]

New comment by Animeshz on void-packages repository

https://github.com/void-linux/void-packages/issues/41008#issuecomment-1345472042

Comment:
One way could be to link `masterdir/pkgname-pkgver/` to a local git repository via a cli-option, which could let it to use the git-tags for the version information and checkouts.

Re: Incremental compilation support (xbps-src)?

[Animeshz]

[-- Attachment #1: Type: text/plain, Size: 520 bytes --]

New comment by Animeshz on void-packages repository

https://github.com/void-linux/void-packages/issues/41008#issuecomment-1345472042

Comment:
One way could be to link `masterdir/pkgname-pkgver/` to a local git repository via a cli-option, which could let it to use the git-tags for the version information and checkouts.

**Edit:** Also, instead of linking we can also do a clone over the local git-repository (`git clone localpath` works) passed via cli-option and pass what (tag) we want to checkout with depth=1.

Re: Incremental compilation support (xbps-src)?

[Animeshz]

[-- Attachment #1: Type: text/plain, Size: 323 bytes --]

New comment by Animeshz on void-packages repository

https://github.com/void-linux/void-packages/issues/41008#issuecomment-1345472042

Comment:
One way could be to link `masterdir/pkgname-pkgver/` to a local git repository via a cli-option, which could let it to use the git-tags for the version information and checkouts.

Re: Incremental compilation support (xbps-src)?

[Animeshz]

[-- Attachment #1: Type: text/plain, Size: 332 bytes --]

New comment by Animeshz on void-packages repository

https://github.com/void-linux/void-packages/issues/41008#issuecomment-1345472042

Comment:
One way could be to link `masterdir/builddir/pkgname-pkgver/` to a local git repository via a cli-option, which could let it to use the git-tags for the version information and checkouts.

Re: Incremental compilation support (xbps-src)?

[classabbyamp]

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/41008#issuecomment-1345474203

Comment:
xbps-src is not designed for incremental compilation, and doing so might take some large architectural changes.

I would recommend setting the following in `etc/conf`:
```
XBPS_MAKEJOBS=4
XBPS_CCACHE=yes
```
(replace the 4 with the number of cores/threads you have) 

they speed things up a lot.

you can also use the `-t` and `-C` flags to avoid rerunning steps like patch or configure over and over while packaging something

Re: Incremental compilation support (xbps-src)?

[Animeshz]

[-- Attachment #1: Type: text/plain, Size: 404 bytes --]

New comment by Animeshz on void-packages repository

https://github.com/void-linux/void-packages/issues/41008#issuecomment-1345477155

Comment:
I know we can use multithreading, but applying a few line change patch from internet and building kernel again still takes about 30 minutes or so, while that could take a file compilation and merge final output with a small linking process (a few seconds?)...

Re: Incremental compilation support (xbps-src)?

[Animeshz]

[-- Attachment #1: Type: text/plain, Size: 315 bytes --]

New comment by Animeshz on void-packages repository

https://github.com/void-linux/void-packages/issues/41008#issuecomment-1345477645

Comment:
Compilers are self sufficient to handle incrementals, we just gotta preserve the build files for them somehow. That doesn't looks like a big architectural change to me...

Re: Incremental compilation support (xbps-src)?

[Anachron]

[-- Attachment #1: Type: text/plain, Size: 271 bytes --]

New comment by Anachron on void-packages repository

https://github.com/void-linux/void-packages/issues/41008#issuecomment-1345528104

Comment:
Did you try it with the settings @classabbyamp suggested?

Because XBPS_CCACHE=yes is responsible for caching build artifacts.

Re: Incremental compilation support (xbps-src)?

[Animeshz]

[-- Attachment #1: Type: text/plain, Size: 552 bytes --]

New comment by Animeshz on void-packages repository

https://github.com/void-linux/void-packages/issues/41008#issuecomment-1350509816

Comment:
Ah yes, sorry for the late reply, the `-C` actually just keeps the package there at the build directory (I think like `xbps-src install` instead of `xbps-src pkg`), but if you add the patches or do something else it wouldn't rebuild the package. If you really want to build the package introducing a small change you gotta again clean the directory, extract, patch, and finally fully-build (no incremental).

Re: Incremental compilation support (xbps-src)?

[Animeshz]

[-- Attachment #1: Type: text/plain, Size: 566 bytes --]

New comment by Animeshz on void-packages repository

https://github.com/void-linux/void-packages/issues/41008#issuecomment-1350509816

Comment:
Ah yes, sorry for the late reply, the `-C` actually just keeps the package there at the build directory (I think like `xbps-src install` instead of `xbps-src pkg`), but if you add the patches or do something else it wouldn't rebuild the package. If you really want to build the package introducing a small change you gotta again clean the directory, extract, patch, and finally it'll fully-build (that's not incremental).

Re: Incremental compilation support (xbps-src)?

[paper42]

[-- Attachment #1: Type: text/plain, Size: 436 bytes --]

New comment by paper42 on void-packages repository

https://github.com/void-linux/void-packages/issues/41008#issuecomment-1350530620

Comment:
You don't have to do that, any step can be run again with `-f`, but patch and install are not repeatable, so you will have to manually apply the patch/unpatch it (because patching once patched sources obviously won't work) or use -t and remove destdir if you want to run those more than once.

Re: Incremental compilation support (xbps-src)?

[Anachron]

[-- Attachment #1: Type: text/plain, Size: 331 bytes --]

New comment by Anachron on void-packages repository

https://github.com/void-linux/void-packages/issues/41008#issuecomment-1350675700

Comment:
Of course you have to rebuild after patching. But even that should be much faster with ccache because a patch wouldnt touch every file most likely.

How do you suggest this should work? 

Re: Incremental compilation support (xbps-src)?

[paper42]

[-- Attachment #1: Type: text/plain, Size: 386 bytes --]

New comment by paper42 on void-packages repository

https://github.com/void-linux/void-packages/issues/41008#issuecomment-1350703573

Comment:
Running `build` after patching a file should do an incremental build with most regular build systems. It will be faster than ccache, but it won't work if `$version` is updated because then $wrksrc is different, ccache will work in both cases.

Re: Incremental compilation support (xbps-src)?

[github-actions]

[-- Attachment #1: Type: text/plain, Size: 293 bytes --]

New comment by github-actions[bot] on void-packages repository

https://github.com/void-linux/void-packages/issues/41008#issuecomment-1469176112

Comment:
Issues become stale 90 days after last activity and are closed 14 days after that.  If this issue is still relevant bump it or assign it.

Re: [ISSUE] [CLOSED] Incremental compilation support (xbps-src)?

[github-actions]

[-- Attachment #1: Type: text/plain, Size: 667 bytes --]

Closed issue by Animeshz on void-packages repository

https://github.com/void-linux/void-packages/issues/41008

Description:
I don't know if this is a valid issue, but compiling big projects like kernel/chromium etc. takes a while, and without cache preservation they take forever to build different versions.

If there could be a incremental compilation support in xbps-src as it already creates a stateful `masterdir`, it could really boost few aspects, such as testing for issues in different versions, and [producing patches](https://www.reddit.com/r/voidlinux/comments/z0mfbh/xbpssrc_creating_patches_for_a_new_package) for them.

Let me know your thoughts!