[PyQt] Multithreading, signals, reference counting and crash

[Phil Thompson]

On 12 Feb 2016, at 10:39 pm, Jones, Bryan <bjones at ece.msstate.edu> wrote:
> 
> Phil,
> 
> Thanks for your response. I'm specifically concerned about the context of passing data between threads using the signal/slot mechanism. As you say, in the direct or blocking case, there's no concern with object lifetime.
> 
> To test this, I modified Ilya's code to make it easier to test the signal/slot mechanism on different types. A QObject fails, but standard Python types (dict, list, object, etc.) pass, showing that their reference count increases after an emit. I can even pass a list containing a QObject safely. This suggests to me that your code increases the reference count when emitting pure Python objects, but not when emitting Qt objects. Based on some digging (see below), I would expect Qt objects to arrive safely to the slot because they're copied, but this doesn't work in practice. What's safe and what's not safe when using the signal/slot mechanism when crossing thread boundaries?

PyQt tries to convert a Python object to something that the Qt meta-type system understands. If it can't then it wraps it in a PyQt_PyObject. This is registered with the meta-type system and part of its job is to manage the reference count of the object as it gets copied around the meta-type system.

If PyQt can convert the object then it's up to the meta-type system to manage the lifecycle of the converted C++ instance as you describe below.

It would be wrong for PyQt to use PyQt_PyObject for every type of Python object because that would mean you couldn't connect to slots implemented in C++ that do not understand PyQt_PyObject. However you can explicitly use PyQt_PyObject yourself by declaring the signal as...

    mysig = pyqtSignal('PyQt_PyObject')

...and this will "protect" the object even if it was a QObject. I was about to include the link to the relevant bit of the documentation and realised that it's only in the current snapshot and I must have added it (the documentation) since the last release.

> Looking at the Qt source, from what I understand:
> 
> 1. In qobject.cpp, the queued_activate function is used to post an signal to another thread's event queue. (See qobject.cpp::activate for the generic mechanism used to emit a signal, which calls queued_activate for queued connections). Here's a helpful blog post on the topic.
> 2. To do this, queued_activate makes a copy of each argument by invoking QMetaType::create (which invokes that type's copy constructor) in the signal being emitted.
> 3. When all signals have been delivered, QMetaCallEvent::destroy frees memory used by this copy.
> 
> Based on this, I conclude that Qt allows the emission of signals with any type registered with the Qt meta-type system by copying the type, delivering it to all slots, then destroying the copy.

Your analysis is correct, but with one key omission. You cannot copy QObject instances. It is QObject* that is supported by the meta-type system and not QObject. You still have to manage the lifecycle of what the pointer points to (or use PyQt_PyObject).

Phil