Python 3.13 refcounting related memory corruption

[Phil Thompson]

On 28/05/2024 13:34, Florian Bruhin wrote:
> Hey,
> 
> When running the qutebrowser testsuite with Python 3.13, I get a
> segfault pointing to a memory corruption inside Python - including 
> funny
> exceptions such as:
> 
>     TypeError: replace() argument 2 must be str, not +
> 
> when doing
> 
>     value = value.replace('default_family', self.default_family)
> 
> and/or assertions about PyUnicode_KIND returning an invalid value.
> 
> I've reported it here:
> https://github.com/python/cpython/issues/119462
> 
> and bisected it to a change in CPython, "Limit the number of versions
> that a single class can use.":
> https://github.com/python/cpython/pull/114900
> 
> Unfortunately I've not been able to extract a minimal example by any
> means. The best I could get is getting it down to running 11 test files
> in qutebrowser, with almost 2000 test cases. If I remove any of those
> files, the bug doesn't trigger anymore.
> 
> I've now run it through valgrind/ASan, which report that a value
> allocated by PyQt:
> 
>     #0 0x5f1017de70a9 in malloc
> (/home/florian/proj/cpython/python+0x32f0a9) (BuildId:
> 472839294c0413d2f8beb5aa8c1f9ae1822dc88b)
>     #1 0x5f10184cd6c8 in tracemalloc_alloc
> /home/florian/proj/cpython/Python/tracemalloc.c:536:15
>     #2 0x5f10184cdc4d in tracemalloc_alloc_gil
> /home/florian/proj/cpython/Python/tracemalloc.c:640:11
>     #3 0x5f1018192780 in PyUnicode_New
> /home/florian/proj/cpython/Objects/unicodeobject.c:1238:24
>     #4 0x744327ffacf2 in sip_api_unicode_new
> /tmp/pip-install-y87d3n5v/pyqt6-sip_626af1e47840447c9f7c098c3c07b101/sip_core.c:11388:16
>     #5 0x744308e15146 in qpycore_PyObject_FromQString(QString const&)
> (/home/florian/proj/qutebrowser/git/.venv-py313dev/lib/python3.13/site-packages/PyQt6/QtCore.abi3.so+0x215146)
> (BuildId: 74342f967c05e7e19c5fd152ab11397679a17e61)
> 
> gets freed by Python:
> 
>     #0 0x5f1017de6072 in free.part.0 asan_malloc_linux.cpp.o
>     #1 0x5f10184c84d7 in tracemalloc_free
> /home/florian/proj/cpython/Python/tracemalloc.c:614:5
>     #2 0x5f10180e82c7 in _Py_Dealloc
> /home/florian/proj/cpython/Objects/object.c:2875:5
>     #3 0x5f1018098ac3 in Py_DECREF
> /home/florian/proj/cpython/./Include/object.h:922:9
>     #4 0x5f1018098ac3 in Py_XDECREF
> /home/florian/proj/cpython/./Include/object.h:1030:9
>     #5 0x5f1018098ac3 in insertdict
> /home/florian/proj/cpython/Objects/dictobject.c:1311:5
>     #6 0x5f10180b135a in _PyObjectDict_SetItem
> /home/florian/proj/cpython/Objects/dictobject.c
>     #7 0x5f10180e5490 in _PyObject_GenericSetAttrWithDict
> /home/florian/proj/cpython/Objects/object.c:1730:19
>     #8 0x5f101814f877 in type_setattro
> /home/florian/proj/cpython/Objects/typeobject.c:4981:11
>     #9 0x5f10180e20ac in PyObject_SetAttr
> /home/florian/proj/cpython/Objects/object.c:1306:15
> 
> but then accessed by Python again:
> 
>     #0 0x5f101832396c in Py_INCREF
> /home/florian/proj/cpython/./Include/object.h:812:30
>     #1 0x5f101832396c in _Py_NewRef
> /home/florian/proj/cpython/./Include/object.h:1046:5
>     #2 0x5f101832396c in _PyEval_EvalFrameDefault
> /home/florian/proj/cpython/Python/generated_cases.c.h:3795:24
>     #3 0x5f1017fe5a62 in _PyObject_VectorcallTstate
> /home/florian/proj/cpython/./Include/internal/pycore_call.h:168:11
>     #4 0x5f1018159a41 in call_attribute
> /home/florian/proj/cpython/Objects/typeobject.c:8874:15
>     #5 0x5f1018159a41 in _Py_slot_tp_getattr_hook
> /home/florian/proj/cpython/Objects/typeobject.c:8924:19
>     #6 0x5f10180e16a7 in PyObject_GetAttr
> /home/florian/proj/cpython/Objects/object.c:1153:18
>     #7 0x5f10182cdb21 in builtin_getattr
> /home/florian/proj/cpython/Python/bltinmodule.c:1179:18
>     #8 0x5f1018303e87 in _PyEval_EvalFrameDefault
> /home/florian/proj/cpython/Python/generated_cases.c.h:1072:19
>     #9 0x5f101803861c in _PyEval_EvalFrame
> /home/florian/proj/cpython/./Include/internal/pycore_ceval.h:115:16
>     #10 0x5f101803861c in gen_send_ex2
> /home/florian/proj/cpython/Objects/genobject.c:228:24
>     #11 0x5f101803419a in gen_iternext
> /home/florian/proj/cpython/Objects/genobject.c:586:9
>     #12 0x5f1018061d35 in list_extend_iter
> /home/florian/proj/cpython/Objects/listobject.c:985:26
>     #13 0x5f1018061d35 in list_extend
> /home/florian/proj/cpython/Objects/listobject.c:1042:16
>     #14 0x5f1018061628 in _PyList_Extend
> /home/florian/proj/cpython/Objects/listobject.c:1050:9
>     #15 0x5f1017f9a856 in PySequence_List
> /home/florian/proj/cpython/Objects/abstract.c:2135:10
>     #16 0x5f1017f9aaca in PySequence_Fast
> /home/florian/proj/cpython/Objects/abstract.c:2166:9
>     #17 0x5f10181cb7ce in PyUnicode_Join
> /home/florian/proj/cpython/Objects/unicodeobject.c:9569:12
> 
> I'm assuming this is a CPython bug, given that I have it bisected to
> that commit - but it's unclear to me why/how PyQt is involved here.
> 
> Is this anything where PyQt could possibly be to blame for some sort of
> refcounting issue? Is there a better way to debug those rather than
> ASan?
> 
> Thanks,
> Florian

What was the PyQt API call that created the unicode object? When would 
that be deallocated?

Phil