Python 3.12: "Python memory allocator called without holding the GIL" on shutdown

[Phil Thompson]

On 09/06/2023 15:30, Florian Bruhin wrote:
> Hey,
> 
> After avoiding multiple inheritance, I was able to run almost the 
> entire
> qutebrowser testsuite successfully on Python 3.12.
> 
> There seems to be one remaining issue, where something like:
> 
>     python3.12 -c "from PyQt6.QtWidgets import QApplication"
> 
> results in:
> 
>     Fatal Python error: _PyMem_DebugFree: Python memory allocator
> called without holding the GIL
>     Python runtime state: finalizing (tstate=0x0000555555c75118)
> 
> With the following stacktrace:
> 
> 	[...]
> 	#4  0x000055555584790a in fatal_error_exit (status=<optimized out>)
> at Python/pylifecycle.c:2693
> 	#5  0x00005555558491c4 in fatal_error (fd=2, header=header at entry=1,
> prefix=prefix at entry=0x5555559206d0 <__func__.3> "_PyMem_DebugFree",
> msg=msg at entry=0x55555591ff70 "Python memory allocator called without
> holding the GIL",
> 		status=status at entry=-1) at Python/pylifecycle.c:2874
> 	#6  0x000055555584922e in _Py_FatalErrorFunc
> (func=func at entry=0x5555559206d0 <__func__.3> "_PyMem_DebugFree",
> msg=msg at entry=0x55555591ff70 "Python memory allocator called without
> holding the GIL") at Python/pylifecycle.c:2890
> 	#7  0x000055555573f0ba in _PyMem_DebugCheckGIL
> (func=func at entry=0x5555559206d0 <__func__.3> "_PyMem_DebugFree") at
> Objects/obmalloc.c:2146
> 	#8  0x000055555573f137 in _PyMem_DebugFree (ctx=0x555555c01168
> <_PyRuntime+200>, ptr=0x555555df1cc0) at Objects/obmalloc.c:2170
> 	#9  0x00005555557406a5 in PyMem_Free (ptr=<optimized out>) at
> Objects/obmalloc.c:616
> 	#10 0x00007ffff74e5296 in sip_api_free (mem=<optimized out>) at 
> sip_core.c:1905
> 	#11 0x00007ffff74f529a in sipOMFinalise (om=om at entry=0x7ffff74ff820
> <cppPyMap>) at sip_object_map.c:69
> 	#12 0x00007ffff74e5351 in finalise () at sip_core.c:1807
> 	#13 0x0000555555847634 in call_ll_exitfuncs
> (runtime=runtime at entry=0x555555c010a0 <_PyRuntime>) at
> Python/pylifecycle.c:2993
> 	#14 0x0000555555848d9b in Py_FinalizeEx () at 
> Python/pylifecycle.c:1966
> 	#15 0x00005555558755d9 in Py_RunMain () at Modules/main.c:691
> 	[...]
> 
> I was able to bisect this to the following Python change:
> 
> 	https://github.com/python/cpython/commit/6036c3e856f033bf13e929536e7bf127fdd921c9
> 	"Clarify GILState-related Code"
>     https://github.com/python/cpython/pull/101161
> 
> As before: Is this a CPython bug, or just some problem in sip/PyQt 
> uncovered by
> that change?

finalise() is registered with Py_AtExit() so that it is called by 
Py_FinalizeEx()...

https://docs.python.org/3/c-api/sys.html#c.Py_AtExit

The docs say nothing about the state of the GIL when a cleanup function 
is called. The docs also state that the cleanup handler should not make 
calls to the Python API, so I suppose the state of the GIL shouldn't 
matter.

However finalise() does make calls to the Python API - but only to free 
memory (one of the stated purposes of Py_FinalizeEx()). It seems a 
little bit contradictory to me to disallow cleanup functions calling 
Py_MemFree().

If this new behaviour were to remain then sip would have to stop using 
the Python memory allocator and just use malloc() and free().

Phil