Python 3.12: "Python memory allocator called without holding the GIL" on shutdown

[Florian Bruhin]

> 
> On 09/06/2023 15:30, Florian Bruhin wrote:
> > Hey,
> > 
> > After avoiding multiple inheritance, I was able to run almost the 
> > entire
> > qutebrowser testsuite successfully on Python 3.12.
> > 
> > There seems to be one remaining issue, where something like:
> > 
> >     python3.12 -c "from PyQt6.QtWidgets import QApplication"
> > 
> > results in:
> > 
> >     Fatal Python error: _PyMem_DebugFree: Python memory allocator
> > called without holding the GIL
> >     Python runtime state: finalizing (tstate=0x0000555555c75118)
> > 
> > With the following stacktrace:
> > 
> > 	[...]
> > 	#4  0x000055555584790a in fatal_error_exit (status=<optimized out>)
> > at Python/pylifecycle.c:2693
> > 	#5  0x00005555558491c4 in fatal_error (fd=2, header=header at entry=1,
> > prefix=prefix at entry=0x5555559206d0 <__func__.3> "_PyMem_DebugFree",
> > msg=msg at entry=0x55555591ff70 "Python memory allocator called without
> > holding the GIL",
> > 		status=status at entry=-1) at Python/pylifecycle.c:2874
> > 	#6  0x000055555584922e in _Py_FatalErrorFunc
> > (func=func at entry=0x5555559206d0 <__func__.3> "_PyMem_DebugFree",
> > msg=msg at entry=0x55555591ff70 "Python memory allocator called without
> > holding the GIL") at Python/pylifecycle.c:2890
> > 	#7  0x000055555573f0ba in _PyMem_DebugCheckGIL
> > (func=func at entry=0x5555559206d0 <__func__.3> "_PyMem_DebugFree") at
> > Objects/obmalloc.c:2146
> > 	#8  0x000055555573f137 in _PyMem_DebugFree (ctx=0x555555c01168
> > <_PyRuntime+200>, ptr=0x555555df1cc0) at Objects/obmalloc.c:2170
> > 	#9  0x00005555557406a5 in PyMem_Free (ptr=<optimized out>) at
> > Objects/obmalloc.c:616
> > 	#10 0x00007ffff74e5296 in sip_api_free (mem=<optimized out>) at 
> > sip_core.c:1905
> > 	#11 0x00007ffff74f529a in sipOMFinalise (om=om at entry=0x7ffff74ff820
> > <cppPyMap>) at sip_object_map.c:69
> > 	#12 0x00007ffff74e5351 in finalise () at sip_core.c:1807
> > 	#13 0x0000555555847634 in call_ll_exitfuncs
> > (runtime=runtime at entry=0x555555c010a0 <_PyRuntime>) at
> > Python/pylifecycle.c:2993
> > 	#14 0x0000555555848d9b in Py_FinalizeEx () at 
> > Python/pylifecycle.c:1966
> > 	#15 0x00005555558755d9 in Py_RunMain () at Modules/main.c:691
> > 	[...]
> > 
> > I was able to bisect this to the following Python change:
> > 
> > 	https://github.com/python/cpython/commit/6036c3e856f033bf13e929536e7bf127fdd921c9
> > 	"Clarify GILState-related Code"
> >     https://github.com/python/cpython/pull/101161
> > 
> > As before: Is this a CPython bug, or just some problem in sip/PyQt 
> > uncovered by
> > that change?
> 
> finalise() is registered with Py_AtExit() so that it is called by 
> Py_FinalizeEx()...
> 
> https://docs.python.org/3/c-api/sys.html#c.Py_AtExit
> 
> The docs say nothing about the state of the GIL when a cleanup function 
> is called. The docs also state that the cleanup handler should not make 
> calls to the Python API, so I suppose the state of the GIL shouldn't 
> matter.
> 
> However finalise() does make calls to the Python API - but only to free 
> memory (one of the stated purposes of Py_FinalizeEx()). It seems a 
> little bit contradictory to me to disallow cleanup functions calling 
> Py_MemFree().
> 
> If this new behaviour were to remain then sip would have to stop using 
> the Python memory allocator and just use malloc() and free().

Makes sense. Given that the commit message claims:

    The objective of this change is to help make the GILState-related code
    easier to understand.  This mostly involves moving code around and some
    semantically equivalent refactors.  However, there are a also a small
    number of slight changes in structure and behavior:

    * tstate_current is moved out of _PyRuntimeState.gilstate
    * autoTSSkey is moved out of _PyRuntimeState.gilstate
    * autoTSSkey is initialized earlier
    * autoTSSkey is re-initialized (after fork) earlier

This might be an oversight?

Do you want to open a CPython issue (given that you're more familiar
with how the details work), or should I do so?

Florian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://www.riverbankcomputing.com/pipermail/pyqt/attachments/20230609/67376bf8/attachment.sig>