[ISSUE] Expand using Profile-Guided Optimization (PGO) and Post Link Optimization (PLO) in the Void Linux packages

[ISSUE] Expand using Profile-Guided Optimization (PGO) and Post Link Optimization (PLO) in the Void Linux packages

[zamazan4ik]

[-- Attachment #1: Type: text/plain, Size: 5162 bytes --]

New issue by zamazan4ik on void-packages repository

https://github.com/void-linux/void-packages/issues/47215

Description:
Hello.

Now I am investigating PGO and PLO effects on different kinds of software - all my current results are available at https://github.com/zamazan4ik/awesome-pgo. According to these results, enabling PGO and PLO can help with achieving better overall performance in many cases. I think expanding PGO and PLO usage for Void Linux packages would be a good idea.

PGO is already a well-known technique. All currently known PGO effects on performance can be found at https://github.com/zamazan4ik/awesome-pgo#pgo-showcases . Several OS distros already enabled PGO for some packages like GCC, Rustc, Chromium, Firefox, and others (it depends on each OS distro, of course).

I think we can try to expand PGO usage across Void Linux packages. I think we can start by enabling PGO at least for the following projects:

* Clang
* GCC
* Rust (`rustc`)
* Firefox (however, I am not sure is it PGO-optimized right now or not in Void Linux)

I am sure it's possible to find more packages with upstream PGO support since some packages Void Linux packages like CPython already [support](https://github.com/void-linux/void-packages/pull/43791) building with PGO.

As an example, PGO is already used in other Linux distributions for these packages:

* GCC in Solus: [link](https://github.com/getsolus/packages/blob/main/packages/g/gcc/package.yml#L164)
* Clang in Solus: [link](https://github.com/getsolus/packages/blob/main/packages/l/llvm/package.yml#L116)
* Firefox in Fedora: [link](https://src.fedoraproject.org/rpms/firefox/blob/rawhide/f/firefox.spec)
* Rust in Fedora: [link](https://src.fedoraproject.org/rpms/rust/blob/rawhide/f/rust.spec#_772)

Regarding Post Link Optimization (PLO), right now there are two main tools - LLVM BOLT and Google Propeller.

According to the Facebook Research Paper (https://research.facebook.com/publications/bolt-a-practical-binary-optimizer-for-data-centers-and-beyond/), LLVM BOLT (https://github.com/llvm/llvm-project/blob/main/bolt/README.md) helps with achieving better performance for various packages like compilers and interpreters. I think it would be a good idea to enable LLVM BOLT for some packages to deliver faster binaries for users (since Propeller is less stable right now).

Here I got some examples of how LLVM BOLT is already integrated into other projects:

* Rustc: https://github.com/rust-lang/rust/pull/116352
* CPython: https://github.com/python/cpython/pull/95908
* Pyston:
  - https://github.com/pyston/pyston#building
  - https://github.com/pyston/pyston/blob/pyston_main/Makefile#L200
* Clang: https://github.com/llvm/llvm-project/blob/main/clang/cmake/caches/BOLT.cmake

So at least for the projects above LLVM BOLT effects are tested and some preparations are already done in the upstream projects. In this case, it should be easier to enable BOLT for these packages.

For some projects right now there is ongoing work on integrating LLVM BOLT into the build scripts:

* Chromium: https://bugs.chromium.org/p/chromium/issues/detail?id=1163978
* Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1789087
  - The same for Propeller (a LLVM BOLT alternative): https://bugzilla.mozilla.org/show_bug.cgi?id=1509314
* NodeJS: https://github.com/nodejs/node/issues/50379
* LDC: https://github.com/ldc-developers/ldc/issues/4228
* GCC: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112492

More about LLVM BOLT performance results for other projects can be found in:

* Rustc:
  - https://github.com/rust-lang/rust/pull/116352
  - https://www.reddit.com/r/rust/comments/y4w2kr/llvm_used_by_rustc_is_now_optimized_with_bolt_on/
* CPython: https://github.com/python/cpython/pull/95908
* YDB: https://github.com/ydb-platform/ydb/issues/140
* Clang:
  - [Slides](https://llvm.org/devmtg/2022-11/slides/Lightning15-OptimizingClangWithBOLTUsingCMake.pdf)
  - [Results on building Clang](https://github.com/ptr1337/llvm-bolt-scripts/blob/master/results.md)
  - [Linaro results](https://android-review.linaro.org/plugins/gitiles/toolchain/llvm_android/+/f36c64eeddf531b7b1a144c40f61d6c9a78eee7a)
  - [on AMD 7950X3D](https://github.com/llvm/llvm-project/issues/65010#issuecomment-1701255347)
* LDC: https://github.com/ldc-developers/ldc/issues/4228#issuecomment-1334499428
* NodeJS: https://aaupov.github.io/blog/2020/10/08/bolt-nodejs
* Chromium: https://aaupov.github.io/blog/2022/11/12/bolt-chromium
* MySQL, MongoDB, memcached, Verilator: https://people.ucsc.edu/~hlitz/papers/ocolos.pdf

Some OS already using LLVM BOLT in their build scripts - [check](https://github.com/getsolus/packages/blob/main/packages/l/llvm/package.yml#L116) Clang recipe in Solus.

I don't create an issue per project (like "Enable BOLT for Clang", "Enable PGO for GCC", etc.) since I think first we need to discuss the approach. If we agree with enabling BOLT or expanding PGO usage for some packages, then we can create an additional issue (and use this issue as a meta issue). In this issue, we can discuss approaches regarding PGO and PLO.

Re: Expand using Profile-Guided Optimization (PGO) and Post Link Optimization (PLO) in the Void Linux packages

[dmarto]

[-- Attachment #1: Type: text/plain, Size: 323 bytes --]

New comment by dmarto on void-packages repository

https://github.com/void-linux/void-packages/issues/47215#issuecomment-1812575454

Comment:
On PGO for Firefox - https://github.com/void-linux/void-packages/issues/39652#issuecomment-1265308710

I would guess that extends to other large/heavy-to-build packages as well. 

Re: Expand using Profile-Guided Optimization (PGO) and Post Link Optimization (PLO) in the Void Linux packages

[zamazan4ik]

[-- Attachment #1: Type: text/plain, Size: 475 bytes --]

New comment by zamazan4ik on void-packages repository

https://github.com/void-linux/void-packages/issues/47215#issuecomment-1813224103

Comment:
> On PGO for Firefox - [#39652 (comment)](https://github.com/void-linux/void-packages/issues/39652#issuecomment-1265308710)
> 
> I would guess that extends to other large/heavy-to-build packages as well.

I saw this comment but I wanted to recheck that the situation with build resources hasn't changed during the last year.

Re: Expand using Profile-Guided Optimization (PGO) and Post Link Optimization (PLO) in the Void Linux packages

[classabbyamp]

[-- Attachment #1: Type: text/plain, Size: 204 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/47215#issuecomment-1813227065

Comment:
the build servers have not changed in that time, afaik.

Re: Expand using Profile-Guided Optimization (PGO) and Post Link Optimization (PLO) in the Void Linux packages

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 382 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/issues/47215#issuecomment-1837298023

Comment:
I suspect the benefits of PGO to be dubious for precompiled packages, because timing results that select between alternatives will be determined by the microarchitecture of the build servers, which will not generally match that of users.

Re: Expand using Profile-Guided Optimization (PGO) and Post Link Optimization (PLO) in the Void Linux packages

[zamazan4ik]

[-- Attachment #1: Type: text/plain, Size: 1337 bytes --]

New comment by zamazan4ik on void-packages repository

https://github.com/void-linux/void-packages/issues/47215#issuecomment-1837299561

Comment:
> I suspect the benefits of PGO to be dubious for precompiled packages, because timing results that select between alternatives will be determined by the microarchitecture of the build servers, which will not generally match that of users.

PGO does not depend on the timings. Instrumentation PGO "just" collects counters from the instrumented program, so there is no actual difference if a program finishes the same workload in minutes or hours - the counter values would be the same for almost all applications. The only thing that can differ - is if a program has internal time-dependent logic (like "run this subroutine every N seconds"). In this case, the counters will differ. But from my experience, such software is really rare in real life and such differences do not introduce measurable differences in the optimization results.

Regarding (micro)architecture - PGO does not depend on the architecture features. However, generally [is not recommended](https://discourse.llvm.org/t/profile-guided-optimization-pgo-related-questions-and-suggestions/75232/2) (check the answer for the fifth question) to re-use profiles between different architectures (e.g  from x86-64 to ARM). 

Re: Expand using Profile-Guided Optimization (PGO) and Post Link Optimization (PLO) in the Void Linux packages

[github-actions]

[-- Attachment #1: Type: text/plain, Size: 293 bytes --]

New comment by github-actions[bot] on void-packages repository

https://github.com/void-linux/void-packages/issues/47215#issuecomment-1974177510

Comment:
Issues become stale 90 days after last activity and are closed 14 days after that.  If this issue is still relevant bump it or assign it.

Re: [ISSUE] [CLOSED] Expand using Profile-Guided Optimization (PGO) and Post Link Optimization (PLO) in the Void Linux packages

[github-actions]

[-- Attachment #1: Type: text/plain, Size: 5165 bytes --]

Closed issue by zamazan4ik on void-packages repository

https://github.com/void-linux/void-packages/issues/47215

Description:
Hello.

Now I am investigating PGO and PLO effects on different kinds of software - all my current results are available at https://github.com/zamazan4ik/awesome-pgo. According to these results, enabling PGO and PLO can help with achieving better overall performance in many cases. I think expanding PGO and PLO usage for Void Linux packages would be a good idea.

PGO is already a well-known technique. All currently known PGO effects on performance can be found at https://github.com/zamazan4ik/awesome-pgo#pgo-showcases . Several OS distros already enabled PGO for some packages like GCC, Rustc, Chromium, Firefox, and others (it depends on each OS distro, of course).

I think we can try to expand PGO usage across Void Linux packages. I think we can start by enabling PGO at least for the following projects:

* Clang
* GCC
* Rust (`rustc`)
* Firefox (however, I am not sure is it PGO-optimized right now or not in Void Linux)

I am sure it's possible to find more packages with upstream PGO support since some packages Void Linux packages like CPython already [support](https://github.com/void-linux/void-packages/pull/43791) building with PGO.

As an example, PGO is already used in other Linux distributions for these packages:

* GCC in Solus: [link](https://github.com/getsolus/packages/blob/main/packages/g/gcc/package.yml#L164)
* Clang in Solus: [link](https://github.com/getsolus/packages/blob/main/packages/l/llvm/package.yml#L116)
* Firefox in Fedora: [link](https://src.fedoraproject.org/rpms/firefox/blob/rawhide/f/firefox.spec)
* Rust in Fedora: [link](https://src.fedoraproject.org/rpms/rust/blob/rawhide/f/rust.spec#_772)

Regarding Post Link Optimization (PLO), right now there are two main tools - LLVM BOLT and Google Propeller.

According to the Facebook Research Paper (https://research.facebook.com/publications/bolt-a-practical-binary-optimizer-for-data-centers-and-beyond/), LLVM BOLT (https://github.com/llvm/llvm-project/blob/main/bolt/README.md) helps with achieving better performance for various packages like compilers and interpreters. I think it would be a good idea to enable LLVM BOLT for some packages to deliver faster binaries for users (since Propeller is less stable right now).

Here I got some examples of how LLVM BOLT is already integrated into other projects:

* Rustc: https://github.com/rust-lang/rust/pull/116352
* CPython: https://github.com/python/cpython/pull/95908
* Pyston:
  - https://github.com/pyston/pyston#building
  - https://github.com/pyston/pyston/blob/pyston_main/Makefile#L200
* Clang: https://github.com/llvm/llvm-project/blob/main/clang/cmake/caches/BOLT.cmake

So at least for the projects above LLVM BOLT effects are tested and some preparations are already done in the upstream projects. In this case, it should be easier to enable BOLT for these packages.

For some projects right now there is ongoing work on integrating LLVM BOLT into the build scripts:

* Chromium: https://bugs.chromium.org/p/chromium/issues/detail?id=1163978
* Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1789087
  - The same for Propeller (a LLVM BOLT alternative): https://bugzilla.mozilla.org/show_bug.cgi?id=1509314
* NodeJS: https://github.com/nodejs/node/issues/50379
* LDC: https://github.com/ldc-developers/ldc/issues/4228
* GCC: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112492

More about LLVM BOLT performance results for other projects can be found in:

* Rustc:
  - https://github.com/rust-lang/rust/pull/116352
  - https://www.reddit.com/r/rust/comments/y4w2kr/llvm_used_by_rustc_is_now_optimized_with_bolt_on/
* CPython: https://github.com/python/cpython/pull/95908
* YDB: https://github.com/ydb-platform/ydb/issues/140
* Clang:
  - [Slides](https://llvm.org/devmtg/2022-11/slides/Lightning15-OptimizingClangWithBOLTUsingCMake.pdf)
  - [Results on building Clang](https://github.com/ptr1337/llvm-bolt-scripts/blob/master/results.md)
  - [Linaro results](https://android-review.linaro.org/plugins/gitiles/toolchain/llvm_android/+/f36c64eeddf531b7b1a144c40f61d6c9a78eee7a)
  - [on AMD 7950X3D](https://github.com/llvm/llvm-project/issues/65010#issuecomment-1701255347)
* LDC: https://github.com/ldc-developers/ldc/issues/4228#issuecomment-1334499428
* NodeJS: https://aaupov.github.io/blog/2020/10/08/bolt-nodejs
* Chromium: https://aaupov.github.io/blog/2022/11/12/bolt-chromium
* MySQL, MongoDB, memcached, Verilator: https://people.ucsc.edu/~hlitz/papers/ocolos.pdf

Some OS already using LLVM BOLT in their build scripts - [check](https://github.com/getsolus/packages/blob/main/packages/l/llvm/package.yml#L116) Clang recipe in Solus.

I don't create an issue per project (like "Enable BOLT for Clang", "Enable PGO for GCC", etc.) since I think first we need to discuss the approach. If we agree with enabling BOLT or expanding PGO usage for some packages, then we can create an additional issue (and use this issue as a meta issue). In this issue, we can discuss approaches regarding PGO and PLO.