Github messages for voidlinux
 help / color / mirror / Atom feed
* [ISSUE] ccache isn't working with xbps-src
@ 2020-11-19 20:30 Logarithmus
  2020-11-19 21:23 ` pullmoll
                   ` (7 more replies)
  0 siblings, 8 replies; 9+ messages in thread
From: Logarithmus @ 2020-11-19 20:30 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 543 bytes --]

New issue by Logarithmus on void-packages repository

https://github.com/void-linux/void-packages/issues/26508

Description:
I've added `XBPS_CCACHE=yes` to `etc/config`. After that, when I build packages, the files are written into `hostdir/ccache`. But for some strange reason they are not used during the recompilation of the same package. E. g. I've built chromium-86, now when I'm building chromium-87, the process isn't sped up at all. Also when I open `htop`, it shows `/usr/bin/clang++` and no `ccache` processes. How can I fix that? 

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: ccache isn't working with xbps-src
  2020-11-19 20:30 [ISSUE] ccache isn't working with xbps-src Logarithmus
@ 2020-11-19 21:23 ` pullmoll
  2020-11-19 21:25 ` pullmoll
                   ` (6 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: pullmoll @ 2020-11-19 21:23 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 1152 bytes --]

New comment by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/26508#issuecomment-730645253

Comment:
1) You probably cannot expect a single hit between chromium-86 and chromium-87 because headers changed and a change in a central header (think `version.h`) prevents a cache hit.
2) You need to make sure to re-bootstrap your environment, i.e. do `./xbps-src zap` and `./xbps-src binary-bootstrap` again to make the `etc/conf` (not `etc/config` AFAICT) change take effect.
3) Any change in the compiler binary invalidates the entire current cache, because a checksum of the compiler binary (`/usr/bin/clang++` in this case) is used to make sure that compilation results will indeed be identical. So the update of llvm10 to llvm11 already invalidated all your existing cache entries. You can clean and zap it (`ccache -C` and `ccache -Z`) in such a case.

I have a separate 1TB SDD for ccache and currently have 653GiB of 768GiB used with a cache hit rate of 8.06%. I'm building literally tens of thousands of packages for all architectures and can confirm that `ccache` indeed works as it is supposed to.


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: ccache isn't working with xbps-src
  2020-11-19 20:30 [ISSUE] ccache isn't working with xbps-src Logarithmus
  2020-11-19 21:23 ` pullmoll
@ 2020-11-19 21:25 ` pullmoll
  2020-11-19 21:27 ` pullmoll
                   ` (5 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: pullmoll @ 2020-11-19 21:25 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 1154 bytes --]

New comment by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/26508#issuecomment-730645253

Comment:
1) You probably cannot expect a single hit between chromium-86 and chromium-87 because headers changed and a change in a central header (think `version.h`) prevents a cache hit.
2) You need to make sure to re-bootstrap your environment, i.e. do `./xbps-src zap` and `./xbps-src binary-bootstrap` again to make the `etc/conf` (not `etc/config` AFAICT) change take effect.
3) Any change in the compiler binary invalidates the entire current cache, because a checksum of the compiler binary (`/usr/bin/clang++` in this case) is used to make sure that compilation results will indeed be identical. So the update of llvm10 to llvm11 already invalidated all your existing cache entries. You can clean and zap it (`ccache -C` and `ccache -Z`) in such a case.

I have a separate 1TB SSD for ccache and currently have 653 GiB of 768 GiB used with a cache hit rate of 8.06%. I'm building literally tens of thousands of packages for all architectures and can confirm that `ccache` indeed works as it is supposed to.


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: ccache isn't working with xbps-src
  2020-11-19 20:30 [ISSUE] ccache isn't working with xbps-src Logarithmus
  2020-11-19 21:23 ` pullmoll
  2020-11-19 21:25 ` pullmoll
@ 2020-11-19 21:27 ` pullmoll
  2020-11-19 21:29 ` pullmoll
                   ` (4 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: pullmoll @ 2020-11-19 21:27 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 1257 bytes --]

New comment by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/26508#issuecomment-730645253

Comment:
1) You probably cannot expect a single hit between chromium-86 and chromium-87 because headers changed and a change in a central header (think `version.h`) prevents a cache hit.
2) You need to make sure to re-bootstrap your environment, i.e. do `./xbps-src zap` and `./xbps-src binary-bootstrap` again to make the `etc/conf` (not `etc/config` AFAICT) change take effect.
3) Any change in the compiler binary invalidates the entire current cache, because a checksum of the compiler binary (`/usr/bin/clang++` in this case) is used to make sure that compilation results will indeed be identical. So the update of llvm10 to llvm11 already invalidated all your existing cache entries for `clang` / `clang++`. You can clean and zap the cache (`ccache -C` and `ccache -Z`) in such a case, except perahaps if you also have lots of `gcc` and `g++` cache entries.

I have a separate 1TB SSD for ccache and currently have 653 GiB of 768 GiB used with a cache hit rate of 8.06%. I'm building literally tens of thousands of packages for all architectures and can confirm that `ccache` indeed works as it is supposed to.


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: ccache isn't working with xbps-src
  2020-11-19 20:30 [ISSUE] ccache isn't working with xbps-src Logarithmus
                   ` (2 preceding siblings ...)
  2020-11-19 21:27 ` pullmoll
@ 2020-11-19 21:29 ` pullmoll
  2020-11-19 21:31 ` pullmoll
                   ` (3 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: pullmoll @ 2020-11-19 21:29 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 1256 bytes --]

New comment by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/26508#issuecomment-730645253

Comment:
1) You probably cannot expect a single hit between chromium-86 and chromium-87 because headers changed and a change in a central header (think `version.h`) prevents a cache hit.
2) You need to make sure to re-bootstrap your environment, i.e. do `./xbps-src zap` and `./xbps-src binary-bootstrap` again to make the `etc/conf` (not `etc/config` AFAICT) change take effect.
3) Any change in the compiler binary invalidates the entire current cache, because a checksum of the compiler binary (`/usr/bin/clang++` in this case) is used to make sure that compilation results will indeed be identical. So the update of llvm10 to llvm11 already invalidated all your existing cache entries for `clang` / `clang++`. You can clean and zap the cache (`ccache -C` and `ccache -Z`) in such a case, except perhaps if you also have lots of `gcc` and `g++` cache entries.

I have a separate 1TB SSD for ccache and currently have 653 GiB of 768 GiB used with a cache hit rate of 8.06%. I'm building literally tens of thousands of packages for all architectures and can confirm that `ccache` indeed works as it is supposed to.


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: ccache isn't working with xbps-src
  2020-11-19 20:30 [ISSUE] ccache isn't working with xbps-src Logarithmus
                   ` (3 preceding siblings ...)
  2020-11-19 21:29 ` pullmoll
@ 2020-11-19 21:31 ` pullmoll
  2020-11-19 21:35 ` pullmoll
                   ` (2 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: pullmoll @ 2020-11-19 21:31 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 2796 bytes --]

New comment by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/26508#issuecomment-730645253

Comment:
1) You probably cannot expect a single hit between chromium-86 and chromium-87 because headers changed and a change in a central header (think `version.h`) prevents a cache hit.
2) You need to make sure to re-bootstrap your environment, i.e. do `./xbps-src zap` and `./xbps-src binary-bootstrap` again to make the `etc/conf` (not `etc/config` AFAICT) change take effect.
3) Any change in the compiler binary invalidates the entire current cache, because a checksum of the compiler binary (`/usr/bin/clang++` in this case) is used to make sure that compilation results will indeed be identical. So the update of llvm10 to llvm11 already invalidated all your existing cache entries for `clang` / `clang++`. You can clean and zap the cache (`ccache -C` and `ccache -Z`) in such a case, except perhaps if you also have lots of `gcc` and `g++` cache entries.

I have a separate 1TB SSD for ccache and currently have 653 GiB of 768 GiB used with a cache hit rate of 8.06%. I'm building literally tens of thousands of packages for all architectures and can confirm that `ccache` indeed works as it is supposed to.

Just to show you the figures this is the output of `ccache -s`. In my user's environment I have `CCACHE_DIR=/work/void/ccache` i.e. the same directory which is used from inside the chroot:
```
cache directory                     /work/void/ccache
primary config                      /work/void/ccache/ccache.conf
secondary config (readonly)         /etc/ccache.conf
stats updated                       Thu Nov 19 22:29:31 2020
cache hit (direct)                819345
cache hit (preprocessed)          178710
cache miss                      11381469
cache hit rate                      8.06 %
called for link                   994196
called for preprocessing          994011
multiple source files               1762
compiler produced stdout            9438
compiler produced no output           77
compiler produced empty output     10314
compile failed                    200101
ccache internal error                613
preprocessor error                145901
can't use precompiled header       89380
bad compiler arguments            162657
unsupported source language        62376
autoconf compile/link             867964
unsupported compiler option        93317
unsupported code directive          1051
output to stdout                       4
could not write to output file         6
no input file                     345577
cleanups performed                  2489
files in cache                   8008769
cache size                         654.3 GB
max cache size                     768.0 GB
```


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: ccache isn't working with xbps-src
  2020-11-19 20:30 [ISSUE] ccache isn't working with xbps-src Logarithmus
                   ` (4 preceding siblings ...)
  2020-11-19 21:31 ` pullmoll
@ 2020-11-19 21:35 ` pullmoll
  2020-11-20 19:18 ` Logarithmus
  2020-11-21 19:56 ` [ISSUE] [CLOSED] " Logarithmus
  7 siblings, 0 replies; 9+ messages in thread
From: pullmoll @ 2020-11-19 21:35 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 3028 bytes --]

New comment by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/26508#issuecomment-730645253

Comment:
1) You probably cannot expect a single hit between chromium-86 and chromium-87 because headers changed and a change in a central header (think `version.h`) prevents a cache hit.
2) You need to make sure to re-bootstrap your environment, i.e. do `./xbps-src zap` and `./xbps-src binary-bootstrap` again to make the `etc/conf` (not `etc/config` AFAICT) change take effect.
3) Any change in the compiler binary invalidates the entire current cache, because a checksum of the compiler binary (`/usr/bin/clang++` in this case) is used to make sure that compilation results will indeed be identical. So the update of llvm10 to llvm11 already invalidated all your existing cache entries for `clang` / `clang++`. You can clean and zap the cache (`ccache -C` and `ccache -Z`) in such a case, except perhaps if you also have lots of `gcc` and `g++` cache entries.

I have a separate 1TB SSD for ccache and currently have 653 GiB of 768 GiB used with a cache hit rate of 8.06%. I'm building literally tens of thousands of packages for all architectures and can confirm that `ccache` indeed works as it is supposed to.

Just to show you the figures this is the output of `ccache -s`. In my user's environment I have `CCACHE_DIR=/work/void/ccache` i.e. the same directory which is used from inside the chroot:
```
cache directory                     /work/void/ccache
primary config                      /work/void/ccache/ccache.conf
secondary config (readonly)         /etc/ccache.conf
stats updated                       Thu Nov 19 22:29:31 2020
cache hit (direct)                819345
cache hit (preprocessed)          178710
cache miss                      11381469
cache hit rate                      8.06 %
called for link                   994196
called for preprocessing          994011
multiple source files               1762
compiler produced stdout            9438
compiler produced no output           77
compiler produced empty output     10314
compile failed                    200101
ccache internal error                613
preprocessor error                145901
can't use precompiled header       89380
bad compiler arguments            162657
unsupported source language        62376
autoconf compile/link             867964
unsupported compiler option        93317
unsupported code directive          1051
output to stdout                       4
could not write to output file         6
no input file                     345577
cleanups performed                  2489
files in cache                   8008769
cache size                         654.3 GB
max cache size                     768.0 GB
```

Edit2: This script is my `$HOME/bin/ccwatch` and I use to watch ccache working...
```bash
#!/bin/bash
CCACHE_DIR=/work/void/ccache
if [ -n "$1" ]; then
        CCACHE_DIR="$1"
fi
export CCACHE_DIR
watch -n 1 ccache -s
```

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: ccache isn't working with xbps-src
  2020-11-19 20:30 [ISSUE] ccache isn't working with xbps-src Logarithmus
                   ` (5 preceding siblings ...)
  2020-11-19 21:35 ` pullmoll
@ 2020-11-20 19:18 ` Logarithmus
  2020-11-21 19:56 ` [ISSUE] [CLOSED] " Logarithmus
  7 siblings, 0 replies; 9+ messages in thread
From: Logarithmus @ 2020-11-20 19:18 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 263 bytes --]

New comment by Logarithmus on void-packages repository

https://github.com/void-linux/void-packages/issues/26508#issuecomment-731360215

Comment:
Thank you for the explanation! Unfortunately, my `ccache` hit rate is only 1.40% 😃
Anyway, that's better than 0.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [ISSUE] [CLOSED] ccache isn't working with xbps-src
  2020-11-19 20:30 [ISSUE] ccache isn't working with xbps-src Logarithmus
                   ` (6 preceding siblings ...)
  2020-11-20 19:18 ` Logarithmus
@ 2020-11-21 19:56 ` Logarithmus
  7 siblings, 0 replies; 9+ messages in thread
From: Logarithmus @ 2020-11-21 19:56 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 546 bytes --]

Closed issue by Logarithmus on void-packages repository

https://github.com/void-linux/void-packages/issues/26508

Description:
I've added `XBPS_CCACHE=yes` to `etc/config`. After that, when I build packages, the files are written into `hostdir/ccache`. But for some strange reason they are not used during the recompilation of the same package. E. g. I've built chromium-86, now when I'm building chromium-87, the process isn't sped up at all. Also when I open `htop`, it shows `/usr/bin/clang++` and no `ccache` processes. How can I fix that? 

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2020-11-21 19:56 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-19 20:30 [ISSUE] ccache isn't working with xbps-src Logarithmus
2020-11-19 21:23 ` pullmoll
2020-11-19 21:25 ` pullmoll
2020-11-19 21:27 ` pullmoll
2020-11-19 21:29 ` pullmoll
2020-11-19 21:31 ` pullmoll
2020-11-19 21:35 ` pullmoll
2020-11-20 19:18 ` Logarithmus
2020-11-21 19:56 ` [ISSUE] [CLOSED] " Logarithmus

Github messages for voidlinux

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://inbox.vuxu.org/voidlinux-github

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 voidlinux-github voidlinux-github/ http://inbox.vuxu.org/voidlinux-github \
		voidlinux-github@inbox.vuxu.org
	public-inbox-index voidlinux-github

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.vuxu.org/vuxu.github.voidlinux


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git