Python 3.13 refcounting related memory corruption

[Florian Bruhin]

Hey,

When running the qutebrowser testsuite with Python 3.13, I get a
segfault pointing to a memory corruption inside Python - including funny
exceptions such as:

    TypeError: replace() argument 2 must be str, not +

when doing

    value = value.replace('default_family', self.default_family)

and/or assertions about PyUnicode_KIND returning an invalid value.

I've reported it here:
https://github.com/python/cpython/issues/119462

and bisected it to a change in CPython, "Limit the number of versions
that a single class can use.":
https://github.com/python/cpython/pull/114900

Unfortunately I've not been able to extract a minimal example by any
means. The best I could get is getting it down to running 11 test files
in qutebrowser, with almost 2000 test cases. If I remove any of those
files, the bug doesn't trigger anymore.

I've now run it through valgrind/ASan, which report that a value
allocated by PyQt:

    #0 0x5f1017de70a9 in malloc (/home/florian/proj/cpython/python+0x32f0a9) (BuildId: 472839294c0413d2f8beb5aa8c1f9ae1822dc88b)
    #1 0x5f10184cd6c8 in tracemalloc_alloc /home/florian/proj/cpython/Python/tracemalloc.c:536:15
    #2 0x5f10184cdc4d in tracemalloc_alloc_gil /home/florian/proj/cpython/Python/tracemalloc.c:640:11
    #3 0x5f1018192780 in PyUnicode_New /home/florian/proj/cpython/Objects/unicodeobject.c:1238:24
    #4 0x744327ffacf2 in sip_api_unicode_new /tmp/pip-install-y87d3n5v/pyqt6-sip_626af1e47840447c9f7c098c3c07b101/sip_core.c:11388:16
    #5 0x744308e15146 in qpycore_PyObject_FromQString(QString const&) (/home/florian/proj/qutebrowser/git/.venv-py313dev/lib/python3.13/site-packages/PyQt6/QtCore.abi3.so+0x215146) (BuildId: 74342f967c05e7e19c5fd152ab11397679a17e61)

gets freed by Python:

    #0 0x5f1017de6072 in free.part.0 asan_malloc_linux.cpp.o
    #1 0x5f10184c84d7 in tracemalloc_free /home/florian/proj/cpython/Python/tracemalloc.c:614:5
    #2 0x5f10180e82c7 in _Py_Dealloc /home/florian/proj/cpython/Objects/object.c:2875:5
    #3 0x5f1018098ac3 in Py_DECREF /home/florian/proj/cpython/./Include/object.h:922:9
    #4 0x5f1018098ac3 in Py_XDECREF /home/florian/proj/cpython/./Include/object.h:1030:9
    #5 0x5f1018098ac3 in insertdict /home/florian/proj/cpython/Objects/dictobject.c:1311:5
    #6 0x5f10180b135a in _PyObjectDict_SetItem /home/florian/proj/cpython/Objects/dictobject.c
    #7 0x5f10180e5490 in _PyObject_GenericSetAttrWithDict /home/florian/proj/cpython/Objects/object.c:1730:19
    #8 0x5f101814f877 in type_setattro /home/florian/proj/cpython/Objects/typeobject.c:4981:11
    #9 0x5f10180e20ac in PyObject_SetAttr /home/florian/proj/cpython/Objects/object.c:1306:15

but then accessed by Python again:

    #0 0x5f101832396c in Py_INCREF /home/florian/proj/cpython/./Include/object.h:812:30
    #1 0x5f101832396c in _Py_NewRef /home/florian/proj/cpython/./Include/object.h:1046:5
    #2 0x5f101832396c in _PyEval_EvalFrameDefault /home/florian/proj/cpython/Python/generated_cases.c.h:3795:24
    #3 0x5f1017fe5a62 in _PyObject_VectorcallTstate /home/florian/proj/cpython/./Include/internal/pycore_call.h:168:11
    #4 0x5f1018159a41 in call_attribute /home/florian/proj/cpython/Objects/typeobject.c:8874:15
    #5 0x5f1018159a41 in _Py_slot_tp_getattr_hook /home/florian/proj/cpython/Objects/typeobject.c:8924:19
    #6 0x5f10180e16a7 in PyObject_GetAttr /home/florian/proj/cpython/Objects/object.c:1153:18
    #7 0x5f10182cdb21 in builtin_getattr /home/florian/proj/cpython/Python/bltinmodule.c:1179:18
    #8 0x5f1018303e87 in _PyEval_EvalFrameDefault /home/florian/proj/cpython/Python/generated_cases.c.h:1072:19
    #9 0x5f101803861c in _PyEval_EvalFrame /home/florian/proj/cpython/./Include/internal/pycore_ceval.h:115:16
    #10 0x5f101803861c in gen_send_ex2 /home/florian/proj/cpython/Objects/genobject.c:228:24
    #11 0x5f101803419a in gen_iternext /home/florian/proj/cpython/Objects/genobject.c:586:9
    #12 0x5f1018061d35 in list_extend_iter /home/florian/proj/cpython/Objects/listobject.c:985:26
    #13 0x5f1018061d35 in list_extend /home/florian/proj/cpython/Objects/listobject.c:1042:16
    #14 0x5f1018061628 in _PyList_Extend /home/florian/proj/cpython/Objects/listobject.c:1050:9
    #15 0x5f1017f9a856 in PySequence_List /home/florian/proj/cpython/Objects/abstract.c:2135:10
    #16 0x5f1017f9aaca in PySequence_Fast /home/florian/proj/cpython/Objects/abstract.c:2166:9
    #17 0x5f10181cb7ce in PyUnicode_Join /home/florian/proj/cpython/Objects/unicodeobject.c:9569:12

I'm assuming this is a CPython bug, given that I have it bisected to
that commit - but it's unclear to me why/how PyQt is involved here.

Is this anything where PyQt could possibly be to blame for some sort of
refcounting issue? Is there a better way to debug those rather than
ASan?

Thanks,
Florian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://www.riverbankcomputing.com/pipermail/pyqt/attachments/20240528/f07c3c07/attachment.sig>