[PR PATCH] python3-Cython: fix memory leak + debug output for generators

[PR PATCH] python3-Cython: fix memory leak + debug output for generators

[tornaria]

[-- Attachment #1: Type: text/plain, Size: 1738 bytes --]

There is a new pull request by tornaria against master on the void-packages repository

https://github.com/tornaria/void-packages cython
https://github.com/void-linux/void-packages/pull/46553

python3-Cython: fix memory leak + debug output for generators
https://github.com/cython/cython/pull/5690
https://github.com/cython/cython/pull/5725

Both are merged in cython 3.0.3, however that release breaks scipy.

<!-- Uncomment relevant sections and delete options which are not applicable -->

#### Testing the changes
- I tested the changes in this PR: **briefly**

The memory leak was detected due to 3 doctest failures in sagemath (when built with cython 3) and this PR fixes it.

The debug output issue symptom is that sagemath can't be built unless one sets `SAGE_DEBUG=no` (however, this only triggers when `python3-lxml` is installed).

cc: @ahesford 

<!--
#### New package
- This new package conforms to the [package requirements](https://github.com/void-linux/void-packages/blob/master/CONTRIBUTING.md#package-requirements): **YES**|**NO**
-->

<!-- Note: If the build is likely to take more than 2 hours, please add ci skip tag as described in
https://github.com/void-linux/void-packages/blob/master/CONTRIBUTING.md#continuous-integration
and test at least one native build and, if supported, at least one cross build.
Ignore this section if this PR is not skipping CI.
-->
<!--
#### Local build testing
- I built this PR locally for my native architecture, (ARCH-LIBC)
- I built this PR locally for these architectures (if supported. mark crossbuilds):
  - aarch64-musl
  - armv7l
  - armv6l-musl
-->


A patch file from https://github.com/void-linux/void-packages/pull/46553.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-cython-46553.patch --]
[-- Type: text/x-diff, Size: 8011 bytes --]

From 685eabf68060217b379a5f3d07d6e09f34e5c909 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Gonzalo=20Tornar=C3=ADa?= <tornaria@cmat.edu.uy>
Date: Tue, 10 Oct 2023 08:51:25 -0300
Subject: [PATCH] python3-Cython: fix memory leak + debug output for generators

https://github.com/cython/cython/pull/5690
https://github.com/cython/cython/pull/5725

Both are merged in cython 3.0.3, however that release breaks scipy.
---
 srcpkgs/python3-Cython/patches/5690.patch | 138 ++++++++++++++++++++++
 srcpkgs/python3-Cython/patches/5725.patch |  49 ++++++++
 srcpkgs/python3-Cython/template           |   2 +-
 3 files changed, 188 insertions(+), 1 deletion(-)
 create mode 100644 srcpkgs/python3-Cython/patches/5690.patch
 create mode 100644 srcpkgs/python3-Cython/patches/5725.patch

diff --git a/srcpkgs/python3-Cython/patches/5690.patch b/srcpkgs/python3-Cython/patches/5690.patch
new file mode 100644
index 0000000000000..b3a515c3f40f4
--- /dev/null
+++ b/srcpkgs/python3-Cython/patches/5690.patch
@@ -0,0 +1,138 @@
+Taken from https://github.com/cython/cython/pull/5690
+
+From d0139394794c207956c972e90ce59f33ef1fd709 Mon Sep 17 00:00:00 2001
+From: Oleksandr Pavlyk <oleksandr.pavlyk@intel.com>
+Date: Fri, 8 Sep 2023 14:30:06 -0500
+Subject: [PATCH 1/4] Skip building trees for nodes with invalid tag name
+
+This change attempts to exclude arguments with invalid tag names
+from being inserted into the TreeBuilder
+---
+ Cython/Debugger/DebugWriter.py | 18 ++++++++++++++----
+ 1 file changed, 14 insertions(+), 4 deletions(-)
+
+diff --git a/Cython/Debugger/DebugWriter.py b/Cython/Debugger/DebugWriter.py
+index 8b1fb75b027..dbe7cf55671 100644
+--- a/Cython/Debugger/DebugWriter.py
++++ b/Cython/Debugger/DebugWriter.py
+@@ -20,6 +20,13 @@
+ from ..Compiler import Errors
+ 
+ 
++def is_valid_tag(name):
++    if hasattr(name, "startswith"):
++        if name.startswith(".") and name[1:].isnumeric():
++            return False
++    return True
++
++
+ class CythonDebugWriter(object):
+     """
+     Class to output debugging information for cygdb
+@@ -39,14 +46,17 @@ def __init__(self, output_dir):
+         self.start('cython_debug', attrs=dict(version='1.0'))
+ 
+     def start(self, name, attrs=None):
+-        self.tb.start(name, attrs or {})
++        if is_valid_tag(name):
++            self.tb.start(name, attrs or {})
+ 
+     def end(self, name):
+-        self.tb.end(name)
++        if is_valid_tag(name):
++            self.tb.end(name)
+ 
+     def add_entry(self, name, **attrs):
+-        self.tb.start(name, attrs)
+-        self.tb.end(name)
++        if is_valid_tag(name):
++            self.tb.start(name, attrs)
++            self.tb.end(name)
+ 
+     def serialize(self):
+         self.tb.end('Module')
+
+From 1ea76c1adc3f5ab003a6ad9513ce25e5e7620388 Mon Sep 17 00:00:00 2001
+From: Oleksandr Pavlyk <oleksandr.pavlyk@intel.com>
+Date: Fri, 8 Sep 2023 15:18:21 -0500
+Subject: [PATCH 2/4] Use type checking is is_valid_tag for efficiency
+
+Documented the purpose of is_valid_tag function in its docstring.
+---
+ Cython/Debugger/DebugWriter.py | 10 +++++++++-
+ 1 file changed, 9 insertions(+), 1 deletion(-)
+
+diff --git a/Cython/Debugger/DebugWriter.py b/Cython/Debugger/DebugWriter.py
+index dbe7cf55671..57ada84ac2e 100644
+--- a/Cython/Debugger/DebugWriter.py
++++ b/Cython/Debugger/DebugWriter.py
+@@ -18,10 +18,18 @@
+             etree = None
+ 
+ from ..Compiler import Errors
++from ..Compiler.StringEncoding import EncodedString
+ 
+ 
+ def is_valid_tag(name):
+-    if hasattr(name, "startswith"):
++    """
++    Names like '.0' are used internally for arguments
++    to functions creating generator expressions,
++    however they are not identifiers. 
++
++    See gh-5552
++    """
++    if isinstance(name, EncodedString):
+         if name.startswith(".") and name[1:].isnumeric():
+             return False
+     return True
+
+From 4f0cf58ef1176af6c66e492428b78a4da37c41dc Mon Sep 17 00:00:00 2001
+From: scoder <stefan_ml@behnel.de>
+Date: Sat, 9 Sep 2023 11:28:47 +0200
+Subject: [PATCH 3/4] Fix minor issues.
+
+---
+ Cython/Debugger/DebugWriter.py | 4 ++--
+ 1 file changed, 2 insertions(+), 2 deletions(-)
+
+diff --git a/Cython/Debugger/DebugWriter.py b/Cython/Debugger/DebugWriter.py
+index 57ada84ac2e..46db48a6bd8 100644
+--- a/Cython/Debugger/DebugWriter.py
++++ b/Cython/Debugger/DebugWriter.py
+@@ -27,10 +27,10 @@ def is_valid_tag(name):
+     to functions creating generator expressions,
+     however they are not identifiers. 
+ 
+-    See gh-5552
++    See https://github.com/cython/cython/issues/5552
+     """
+     if isinstance(name, EncodedString):
+-        if name.startswith(".") and name[1:].isnumeric():
++        if name.startswith(".") and name[1:].isdecimal():
+             return False
+     return True
+ 
+
+From 0d13568946f5865a55340db09310accd2323f6bf Mon Sep 17 00:00:00 2001
+From: scoder <stefan_ml@behnel.de>
+Date: Sat, 9 Sep 2023 12:22:21 +0200
+Subject: [PATCH 4/4] Remove trailing whitespace.
+
+---
+ Cython/Debugger/DebugWriter.py | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/Cython/Debugger/DebugWriter.py b/Cython/Debugger/DebugWriter.py
+index 46db48a6bd8..2c3c310fc64 100644
+--- a/Cython/Debugger/DebugWriter.py
++++ b/Cython/Debugger/DebugWriter.py
+@@ -25,7 +25,7 @@ def is_valid_tag(name):
+     """
+     Names like '.0' are used internally for arguments
+     to functions creating generator expressions,
+-    however they are not identifiers. 
++    however they are not identifiers.
+ 
+     See https://github.com/cython/cython/issues/5552
+     """
diff --git a/srcpkgs/python3-Cython/patches/5725.patch b/srcpkgs/python3-Cython/patches/5725.patch
new file mode 100644
index 0000000000000..e2f811fe0e020
--- /dev/null
+++ b/srcpkgs/python3-Cython/patches/5725.patch
@@ -0,0 +1,49 @@
+Taken from https://github.com/cython/cython/pull/5725
+
+From c38b320764bd2a7c089b332e0a3d27ec87b725d1 Mon Sep 17 00:00:00 2001
+From: yut23 <yut23@gvljohnsons.com>
+Date: Mon, 25 Sep 2023 16:34:50 -0400
+Subject: [PATCH 1/2] Release references to traceback objects in Python 3.12+
+
+Fixes: #5724
+---
+ Cython/Utility/Exceptions.c | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Cython/Utility/Exceptions.c b/Cython/Utility/Exceptions.c
+index 20dc7faf3a8..a4416462500 100644
+--- a/Cython/Utility/Exceptions.c
++++ b/Cython/Utility/Exceptions.c
+@@ -190,6 +190,8 @@ static CYTHON_INLINE void __Pyx_ErrRestoreInState(PyThreadState *tstate, PyObjec
+     tmp_value = tstate->current_exception;
+     tstate->current_exception = value;
+     Py_XDECREF(tmp_value);
++    Py_XDECREF(tb);
++    Py_XDECREF(type);
+ #else
+     PyObject *tmp_type, *tmp_value, *tmp_tb;
+     tmp_type = tstate->curexc_type;
+
+From d8b71ef8ccb5ef114a77ce499f192f121060ca02 Mon Sep 17 00:00:00 2001
+From: scoder <stefan_ml@behnel.de>
+Date: Tue, 26 Sep 2023 11:08:24 +0200
+Subject: [PATCH 2/2] Remove now-unused "unused" marker.
+
+---
+ Cython/Utility/Exceptions.c | 3 +--
+ 1 file changed, 1 insertion(+), 2 deletions(-)
+
+diff --git a/Cython/Utility/Exceptions.c b/Cython/Utility/Exceptions.c
+index a4416462500..2144594ba53 100644
+--- a/Cython/Utility/Exceptions.c
++++ b/Cython/Utility/Exceptions.c
+@@ -190,8 +189,8 @@ static CYTHON_INLINE void __Pyx_ErrRestoreInState(PyThreadState *tstate, PyObjec
+     tmp_value = tstate->current_exception;
+     tstate->current_exception = value;
+     Py_XDECREF(tmp_value);
+-    Py_XDECREF(tb);
+     Py_XDECREF(type);
++    Py_XDECREF(tb);
+ #else
+     PyObject *tmp_type, *tmp_value, *tmp_tb;
+     tmp_type = tstate->curexc_type;
diff --git a/srcpkgs/python3-Cython/template b/srcpkgs/python3-Cython/template
index f297933592545..782de6ad66468 100644
--- a/srcpkgs/python3-Cython/template
+++ b/srcpkgs/python3-Cython/template
@@ -1,7 +1,7 @@
 # Template file for 'python3-Cython'
 pkgname=python3-Cython
 version=3.0.2
-revision=3
+revision=4
 build_style=python3-module
 hostmakedepends="python3-setuptools"
 makedepends="python3-devel"

Re: python3-Cython: fix memory leak + debug output for generators

[tornaria]

[-- Attachment #1: Type: text/plain, Size: 315 bytes --]

New comment by tornaria on void-packages repository

https://github.com/void-linux/void-packages/pull/46553#issuecomment-1755231377

Comment:
Alternatively, we could try updating to 3.0.3 + https://github.com/cython/cython/pull/5752 which claims to fix the scipy build regression. I haven't tested that one though.

Re: python3-Cython: fix memory leak + debug output for generators

[tornaria]

[-- Attachment #1: Type: text/plain, Size: 564 bytes --]

New comment by tornaria on void-packages repository

https://github.com/void-linux/void-packages/pull/46553#issuecomment-1755231377

Comment:
Alternatively, we could try updating to 3.0.3 + https://github.com/cython/cython/pull/5752 which claims to fix the scipy build regression. I haven't tested that one though.

Edit: or just wait for 3.0.4. This only affects sagemath with cython 3 which is not what we ship. I already have patched cython in my development box. As long as this is fixed by the time sagemath 10.2 is released (cython 3 only), we'll be fine.

Re: python3-Cython: fix memory leak + debug output for generators

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 307 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/pull/46553#issuecomment-1755425786

Comment:
Thanks for digging into this. Because you know this fixes issues with sagemath, let's take these patches for now and wait to see if Cython 3.0.4 fixes SciPy builds.

Re: [PR PATCH] [Merged]: python3-Cython: fix memory leak + debug output for generators

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 1586 bytes --]

There's a merged pull request on the void-packages repository

python3-Cython: fix memory leak + debug output for generators
https://github.com/void-linux/void-packages/pull/46553

Description:
https://github.com/cython/cython/pull/5690
https://github.com/cython/cython/pull/5725

Both are merged in cython 3.0.3, however that release breaks scipy.

<!-- Uncomment relevant sections and delete options which are not applicable -->

#### Testing the changes
- I tested the changes in this PR: **briefly**

The memory leak was detected due to 3 doctest failures in sagemath (when built with cython 3) and this PR fixes it.

The debug output issue symptom is that sagemath can't be built unless one sets `SAGE_DEBUG=no` (however, this only triggers when `python3-lxml` is installed).

cc: @ahesford 

<!--
#### New package
- This new package conforms to the [package requirements](https://github.com/void-linux/void-packages/blob/master/CONTRIBUTING.md#package-requirements): **YES**|**NO**
-->

<!-- Note: If the build is likely to take more than 2 hours, please add ci skip tag as described in
https://github.com/void-linux/void-packages/blob/master/CONTRIBUTING.md#continuous-integration
and test at least one native build and, if supported, at least one cross build.
Ignore this section if this PR is not skipping CI.
-->
<!--
#### Local build testing
- I built this PR locally for my native architecture, (ARCH-LIBC)
- I built this PR locally for these architectures (if supported. mark crossbuilds):
  - aarch64-musl
  - armv7l
  - armv6l-musl
-->