[PR PATCH] python3: enable optimizations.

[PR PATCH] python3: enable optimizations.

[mhmdanas]

[-- Attachment #1: Type: text/plain, Size: 2186 bytes --]

There is a new pull request by mhmdanas against master on the void-packages repository

https://github.com/mhmdanas/void-packages python3-enable-optimizations-flag
https://github.com/void-linux/void-packages/pull/43791

python3: enable optimizations.
This `--enable-optimizations` flag enables profile-guided optimizations (PGO) and link-time optimization (LTO, but apparently only sometimes, depending on the platform). My rudimentary benchmarking indicates that compared to python3 from the repos, python3 built from this PR yielded performance improvements of around 10%-12% (I tried building python3 from master locally, but that one had horrendous performance for some reason). I'm not a benchmarking expert though, so I wanted others to try out their own benchmarks and post their results here.

On my potato laptop, this change tripled build time from 5 minutes to 15 minutes. I don't think this is an issue however because changes to the python3 template are not done often (averaging once per month in the last 12 months), and the builders are much beefier than my laptop, so their build times will be even shorter anyway.

TODO:
- [ ] Is this also needed for `python3-tkinter` (I don't think so, but better check)

#### Testing the changes
- I tested the changes in this PR: **YES** (only python3, not python3-tkinter)

@ahesford

<!--
#### New package
- This new package conforms to the [package requirements](https://github.com/void-linux/void-packages/blob/master/CONTRIBUTING.md#package-requirements): **YES**|**NO**
-->

<!-- Note: If the build is likely to take more than 2 hours, please add ci skip tag as described in
https://github.com/void-linux/void-packages/blob/master/CONTRIBUTING.md#continuous-integration
and test at least one native build and, if supported, at least one cross build.
Ignore this section if this PR is not skipping CI.
-->
<!--
#### Local build testing
- I built this PR locally for my native architecture, (ARCH-LIBC)
- I built this PR locally for these architectures (if supported. mark crossbuilds):
  - aarch64-musl
  - armv7l
  - armv6l-musl
-->


A patch file from https://github.com/void-linux/void-packages/pull/43791.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-python3-enable-optimizations-flag-43791.patch --]
[-- Type: text/x-diff, Size: 1834 bytes --]

From a17071235e403805efed992992adf9e0f6363724 Mon Sep 17 00:00:00 2001
From: triallax <triallax@tutanota.com>
Date: Sun, 7 May 2023 19:39:08 +0300
Subject: [PATCH 1/2] python3: enable optimizations.

---
 srcpkgs/python3/template | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/srcpkgs/python3/template b/srcpkgs/python3/template
index aebf41852764..3cac7ce90aa1 100644
--- a/srcpkgs/python3/template
+++ b/srcpkgs/python3/template
@@ -4,9 +4,9 @@
 #
 pkgname=python3
 version=3.11.3
-revision=1
-build_style="gnu-configure"
-configure_args="--enable-shared --enable-ipv6
+revision=2
+build_style=gnu-configure
+configure_args="--enable-shared --enable-ipv6 --enable-optimizations
  --enable-loadable-sqlite-extensions --with-computed-gotos
  --with-dbmliborder=gdbm:ndbm --with-system-expat --with-system-ffi
  --without-ensurepip ac_cv_working_tzset=yes"

From ee7cb61bf54b602ea98e7361267becb1651f3eb6 Mon Sep 17 00:00:00 2001
From: triallax <triallax@tutanota.com>
Date: Sun, 7 May 2023 19:40:32 +0300
Subject: [PATCH 2/2] python3-tkinter: enable optimizations.

---
 srcpkgs/python3-tkinter/template | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/srcpkgs/python3-tkinter/template b/srcpkgs/python3-tkinter/template
index ec080a818e69..2004df4c28f8 100644
--- a/srcpkgs/python3-tkinter/template
+++ b/srcpkgs/python3-tkinter/template
@@ -9,9 +9,9 @@ _desc="Python programming language"
 
 pkgname=python3-tkinter
 version=3.11.3
-revision=1
-build_style="gnu-configure"
-configure_args="--enable-shared --enable-ipv6
+revision=2
+build_style=gnu-configure
+configure_args="--enable-shared --enable-ipv6 --enable-optimizations
  --enable-loadable-sqlite-extensions --with-computed-gotos
  --with-dbmliborder=gdbm:ndbm --with-system-expat --with-system-ffi
  --without-ensurepip"

Re: python3: enable optimizations.

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 535 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/pull/43791#issuecomment-1544084463

Comment:
Sorry for the delay. I don't really have a problem with enabling optimizations; if they improve runtime performance, we can pay the price in compile time. The question is how much PGO makes improves things for the average user if the optimization is tailored to the build system. I'd like to test a few things, including building on different CPUs to see how well the optimizations translate.

Re: python3: enable optimizations.

[tornaria]

[-- Attachment #1: Type: text/plain, Size: 779 bytes --]

New comment by tornaria on void-packages repository

https://github.com/void-linux/void-packages/pull/43791#issuecomment-1545645951

Comment:
> Sorry for the delay. I don't really have a problem with enabling optimizations; if they improve runtime performance, we can pay the price in compile time. The question is how much PGO makes improves things for the average user if the optimization is tailored to the build system. I'd like to test a few things, including building on different CPUs to see how well the optimizations translate.

As a data point, I built this on my box (i7-9700). Running the sagemath testsuite with this (in the same box) takes ~5% less time to complete. As in 12:00 vs 12:30 for the normal test, and 23:17 vs 24:40 for the long test using 8 threads.

Re: [PR PATCH] [Updated] python3: enable optimizations.

[mhmdanas]

[-- Attachment #1: Type: text/plain, Size: 2404 bytes --]

There is an updated pull request by mhmdanas against master on the void-packages repository

https://github.com/mhmdanas/void-packages python3-enable-optimizations-flag
https://github.com/void-linux/void-packages/pull/43791

python3: enable optimizations.
This `--enable-optimizations` flag enables profile-guided optimizations (PGO) and link-time optimization (LTO, but apparently only sometimes, depending on the platform). My rudimentary benchmarking indicates that compared to python3 from the repos, python3 built from this PR yielded performance improvements of around 10%-12% (I tried building python3 from master locally, but that one had horrendous performance for some reason). I'm not a benchmarking expert though, so I wanted others to try out their own benchmarks and post their results here.

On my potato laptop, this change tripled build time from 5 minutes to 15 minutes. I don't think this is an issue however because changes to the python3 template are not done often (averaging once per month in the last 12 months), and the builders are much beefier than my laptop, so their build times will be even shorter anyway.

TODO:
- [ ] Is this also needed for `python3-tkinter`? (I don't think so, but better check)
- [ ] Are there similar performance improvements with cross builds? (PGO runs and profiles a Python that is compiled for the host architecture, which is why I want to confirm this)

#### Testing the changes
- I tested the changes in this PR: **YES** (only python3, not python3-tkinter)

@ahesford

<!--
#### New package
- This new package conforms to the [package requirements](https://github.com/void-linux/void-packages/blob/master/CONTRIBUTING.md#package-requirements): **YES**|**NO**
-->

<!-- Note: If the build is likely to take more than 2 hours, please add ci skip tag as described in
https://github.com/void-linux/void-packages/blob/master/CONTRIBUTING.md#continuous-integration
and test at least one native build and, if supported, at least one cross build.
Ignore this section if this PR is not skipping CI.
-->
<!--
#### Local build testing
- I built this PR locally for my native architecture, (ARCH-LIBC)
- I built this PR locally for these architectures (if supported. mark crossbuilds):
  - aarch64-musl
  - armv7l
  - armv6l-musl
-->


A patch file from https://github.com/void-linux/void-packages/pull/43791.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-python3-enable-optimizations-flag-43791.patch --]
[-- Type: text/x-diff, Size: 1780 bytes --]

From 9a34543ec4ecfaee1aaf232d385a48edb39e6566 Mon Sep 17 00:00:00 2001
From: triallax <triallax@tutanota.com>
Date: Tue, 8 Aug 2023 17:27:43 +0100
Subject: [PATCH 1/2] python3: enable optimizations.

---
 srcpkgs/python3/template | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/srcpkgs/python3/template b/srcpkgs/python3/template
index ef07cb7d2f396..ca23bbfc54895 100644
--- a/srcpkgs/python3/template
+++ b/srcpkgs/python3/template
@@ -4,9 +4,9 @@
 #
 pkgname=python3
 version=3.11.4
-revision=1
+revision=2
 build_style="gnu-configure"
-configure_args="--enable-shared --enable-ipv6
+configure_args="--enable-shared --enable-ipv6 --enable-optimizations
  --enable-loadable-sqlite-extensions --with-computed-gotos
  --with-dbmliborder=gdbm:ndbm --with-system-expat --with-system-ffi
  --without-ensurepip ac_cv_working_tzset=yes"

From eb0b7292e101be6ce21682de87d07cec72a29bb8 Mon Sep 17 00:00:00 2001
From: triallax <triallax@tutanota.com>
Date: Sun, 7 May 2023 19:40:32 +0300
Subject: [PATCH 2/2] python3-tkinter: enable optimizations.

---
 srcpkgs/python3-tkinter/template | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/srcpkgs/python3-tkinter/template b/srcpkgs/python3-tkinter/template
index fc16bfdf9dce4..10e0fe38104c1 100644
--- a/srcpkgs/python3-tkinter/template
+++ b/srcpkgs/python3-tkinter/template
@@ -9,9 +9,9 @@ _desc="Python programming language"
 
 pkgname=python3-tkinter
 version=3.11.4
-revision=1
+revision=2
 build_style="gnu-configure"
-configure_args="--enable-shared --enable-ipv6
+configure_args="--enable-shared --enable-ipv6 --enable-optimizations
  --enable-loadable-sqlite-extensions --with-computed-gotos
  --with-dbmliborder=gdbm:ndbm --with-system-expat --with-system-ffi
  --without-ensurepip"

Re: python3: enable optimizations.

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 742 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/pull/43791#issuecomment-1676356184

Comment:
I built this PR on a Zen 3 CPU and ran a test that built a sparse finite-element matrix and then used SuperLU to performan LU factorization. Although the SuperLU performance won't change, the setup does some heavy looping in Python and the overall time for the routine dropped from 2.78 sec to 2.3 sec, each trial averaged over several runs. I installed the Zen-built package on another system with a 9th-generation Intel CPU and re-ran the same test, with the times dropping from 3.7 sec to 2.75 sec.

It seems these optimizations offer tangible benefits, and they translate to different systems.

Re: [PR PATCH] [Closed]: python3: enable optimizations.

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 2220 bytes --]

There's a closed pull request on the void-packages repository

python3: enable optimizations.
https://github.com/void-linux/void-packages/pull/43791

Description:
This `--enable-optimizations` flag enables profile-guided optimizations (PGO) and link-time optimization (LTO, but apparently only sometimes, depending on the platform). My rudimentary benchmarking indicates that compared to python3 from the repos, python3 built from this PR yielded performance improvements of around 10%-12% (I tried building python3 from master locally, but that one had horrendous performance for some reason). I'm not a benchmarking expert though, so I wanted others to try out their own benchmarks and post their results here.

On my potato laptop, this change tripled build time from 5 minutes to 15 minutes. I don't think this is an issue however because changes to the python3 template are not done often (averaging once per month in the last 12 months), and the builders are much beefier than my laptop, so their build times will be even shorter anyway.

TODO:
- [ ] Is this also needed for `python3-tkinter`? (I don't think so, but better check)
- [ ] Are there similar performance improvements with cross builds? (PGO runs and profiles a Python that is compiled for the host architecture, which is why I want to confirm this)

#### Testing the changes
- I tested the changes in this PR: **YES** (only python3, not python3-tkinter)

@ahesford

<!--
#### New package
- This new package conforms to the [package requirements](https://github.com/void-linux/void-packages/blob/master/CONTRIBUTING.md#package-requirements): **YES**|**NO**
-->

<!-- Note: If the build is likely to take more than 2 hours, please add ci skip tag as described in
https://github.com/void-linux/void-packages/blob/master/CONTRIBUTING.md#continuous-integration
and test at least one native build and, if supported, at least one cross build.
Ignore this section if this PR is not skipping CI.
-->
<!--
#### Local build testing
- I built this PR locally for my native architecture, (ARCH-LIBC)
- I built this PR locally for these architectures (if supported. mark crossbuilds):
  - aarch64-musl
  - armv7l
  - armv6l-musl
-->

Re: python3: enable optimizations.

[tornaria]

[-- Attachment #1: Type: text/plain, Size: 571 bytes --]

New comment by tornaria on void-packages repository

https://github.com/void-linux/void-packages/pull/43791#issuecomment-1677263396

Comment:
Just for the record, the sagemath testsuite with `python3-3.11.4_1`:
```
Total time for all tests: 843.5 seconds
    cpu time: 5039.4 seconds
    cumulative wall time: 5700.8 seconds
```
After updating to `python3-3.11.4_2` (no other change, no recompilation):
```
Total time for all tests: 790.8 seconds
    cpu time: 4601.8 seconds
    cumulative wall time: 5175.3 seconds
```
That's a neat 5-10 % speedup! Thanks!

Re: python3: enable optimizations.

[tornaria]

[-- Attachment #1: Type: text/plain, Size: 582 bytes --]

New comment by tornaria on void-packages repository

https://github.com/void-linux/void-packages/pull/43791#issuecomment-1677263396

Comment:
Just for the record, the sagemath (10.1.rc0) testsuite with `python3-3.11.4_1`:
```
Total time for all tests: 843.5 seconds
    cpu time: 5039.4 seconds
    cumulative wall time: 5700.8 seconds
```
After updating to `python3-3.11.4_2` (no other change, no recompilation):
```
Total time for all tests: 790.8 seconds
    cpu time: 4601.8 seconds
    cumulative wall time: 5175.3 seconds
```
That's a neat 5-10 % speedup! Thanks!