Python 3.12: "Python memory allocator called without holding the GIL" on shutdown

[Florian Bruhin]

Hey,

>> On 09/06/2023 15:30, Florian Bruhin wrote:
>> > Hey,
>> >
>> > After avoiding multiple inheritance, I was able to run almost the
>> > entire qutebrowser testsuite successfully on Python 3.12.
>> >
>> > There seems to be one remaining issue, where something like:
>> >
>> >     python3.12 -c "from PyQt6.QtWidgets import QApplication"
>> >
>> > results in:
>> >
>> >     Fatal Python error: _PyMem_DebugFree: Python memory allocator
>> > called without holding the GIL
>> >     Python runtime state: finalizing (tstate=0x0000555555c75118)
>> >
>> > [...]
>> >
>> > As before: Is this a CPython bug, or just some problem in sip/PyQt
>> > uncovered by
>> > that change?
>> 
>> finalise() is registered with Py_AtExit() so that it is called by
>> Py_FinalizeEx()...
>> 
>> https://docs.python.org/3/c-api/sys.html#c.Py_AtExit
>> 
>> The docs say nothing about the state of the GIL when a cleanup
>> function is called. The docs also state that the cleanup handler
>> should not make calls to the Python API, so I suppose the state of
>> the GIL shouldn't matter.
>> 
>> However finalise() does make calls to the Python API - but only to
>> free memory (one of the stated purposes of Py_FinalizeEx()). It
>> seems a little bit contradictory to me to disallow cleanup
>> functions calling Py_MemFree().
>> 
>> If this new behaviour were to remain then sip would have to stop using
>> the Python memory allocator and just use malloc() and free().
> 
> Makes sense. Given that the commit message claims:
> 
>     The objective of this change is to help make the
>     GILState-related code easier to understand.  This mostly
>     involves moving code around and some semantically equivalent
>     refactors.  However, there are a also a small number of slight
>     changes in structure and behavior:
> 
>     * tstate_current is moved out of _PyRuntimeState.gilstate
>     * autoTSSkey is moved out of _PyRuntimeState.gilstate
>     * autoTSSkey is initialized earlier
>     * autoTSSkey is re-initialized (after fork) earlier
> 
> This might be an oversight?
> 
> Do you want to open a CPython issue (given that you're more familiar
> with how the details work), or should I do so?

I opened a CPython issue:
https://github.com/python/cpython/issues/105690

Florian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://www.riverbankcomputing.com/pipermail/pyqt/attachments/20230612/355a1bb2/attachment.sig>