[PyQt] Multithreading, signals, reference counting and crash
Jones, Bryan
bjones at ece.msstate.edu
Fri Feb 12 22:39:29 GMT 2016
Phil,
Thanks for your response. I'm specifically concerned about the context of
passing data between threads using the signal/slot mechanism. As you say,
in the direct or blocking case, there's no concern with object lifetime.
To test this, I modified Ilya's code to make it easier to test the
signal/slot mechanism on different types. A QObject fails, but standard
Python types (dict, list, object, etc.) pass, showing that their reference
count increases after an emit. I can even pass a list containing a QObject
safely. This suggests to me that your code increases the reference count
when emitting pure Python objects, but not when emitting Qt objects. Based
on some digging (see below), I would expect Qt objects to arrive safely to
the slot because they're copied, but this doesn't work in practice. *What's
safe and what's not safe when using the signal/slot mechanism when crossing
thread boundaries?*
Looking at the Qt source, from what I understand:
1. In qobject.cpp, the queued_activate
<https://code.woboq.org/qt5/qtbase/src/corelib/kernel/qobject.cpp.html#_ZL15queued_activateP7QObjectiPN14QObjectPrivate10ConnectionEPPvR12QMutexLocker>
function is used to post an signal to another thread's event queue. (See
qobject.cpp::activate
<https://code.woboq.org/qt5/qtbase/src/corelib/kernel/qobject.cpp.html#_ZN11QMetaObject8activateEP7QObjectiiPPv>
for the generic mechanism used to emit a signal, which calls
queued_activate for queued connections). Here's a helpful blog post on the
topic
<https://woboq.com/blog/how-qt-signals-slots-work-part3-queuedconnection.html>
.
2. To do this, queued_activate makes a copy of each argument by invoking
QMetaType::create
<https://code.woboq.org/qt5/qtbase/src/corelib/kernel/qobject.cpp.html#3575>
(which
invokes that type's copy constructor) in the signal being emitted.
3. When all signals have been delivered, QMetaCallEvent::destroy
<https://code.woboq.org/qt5/qtbase/src/corelib/kernel/qobject.cpp.html#_ZN14QMetaCallEventD1Ev>
frees memory used by this copy.
Based on this, I conclude that Qt allows the emission of signals with any
type registered with the Qt meta-type system by copying the type,
delivering it to all slots, then destroying the copy.
Thanks!
Bryan
import time
import sys
import gc
import os
import platform
import sip
from PyQt5.QtCore import QT_VERSION_STR
from PyQt5.Qt import PYQT_VERSION_STR
from PyQt5.QtCore import QObject, QThread, pyqtSignal, pyqtSlot
from PyQt5.QtCore import qDebug, Qt
from PyQt5.QtWidgets import QApplication, QWidget
# Change to set the type of object passed between threads.
objType = QObject
def printObject(obj):
qDebug('{} {:#X} {}'.format(type(obj), id(obj), obj))
class SomeThread(QObject):
signal = pyqtSignal(objType)
def __init__(self):
QObject.__init__(self)
self.thread = QThread()
self.wait = self.thread.wait
self.moveToThread(self.thread)
self.signal.connect(self.signalHandler, type = Qt.QueuedConnection)
self.thread.started.connect(self.run)
self.thread.start()
@pyqtSlot(objType)
def signalHandler(self, obj):
qDebug('In signalHandler.')
printObject(obj)
qDebug("signalHandler obj ref counts: " + str(sys.getrefcount(obj)))
@pyqtSlot()
def run(self):
qDebug("Thread sleeps for 0.1 sec")
time.sleep(0.1)
qDebug("Process events now")
QApplication.processEvents()
self.thread.exit()
def someFunc(objType, th):
obj = objType()
printObject(obj)
qDebug("obj ref counts before emit: " + str(sys.getrefcount(obj)))
th.signal.emit(obj)
qDebug("obj ref counts after emit: " + str(sys.getrefcount(obj)))
app = None
def main():
print('Qt {}, SIP {}, PtQt {}, Python {}, OS {}, platform {}
{}\n'.format(
QT_VERSION_STR, sip.SIP_VERSION_STR, PYQT_VERSION_STR, sys.version,
os.name, platform.system(), platform.release()))
global app
app = QApplication([])
for x in range(100):
print('\n\nTest {}'.format(x + 1))
testType(objType)
def testType(objType):
th = SomeThread()
someFunc(objType, th)
gc.collect()
th.thread.wait()
if __name__ == '__main__':
main()
On Fri, Feb 12, 2016 at 1:57 PM, Jones, Bryan <bjones at ece.msstate.edu>
wrote:
> I'd like to thank Ilya for bringing this to light. I thought I'd worked
> around a similar problem, but I now see that I mistakenly assumed Python
> semantics where C++ semantics apply. To make sure I understand correctly:
>
> 1. If a Python type can converted to a pass-by-value C++ type (int, long,
> etc.) then it will be passed as the C++ value through the signal/slot
> mechanism, meaning I don't need to retain a Python reference to it to keep
> it valid.
> 2. If a Python type cannot be converted to a pass-by-value C++ type
> (string, big int, dict, object, etc.) then a pointer to it will be passed
> through the signal/slot mechanism, meaning I must retain a Python reference
> until the signal has been delivered to all slots.
>
> However, I do see Qt employing pointers (QFileSystemWatcher's signals emit
> const QString&, QLabel::setMovie(QMovie*). Does this imply that the
> emitters of these signals can somehow "know" when it's free to destroy the
> QString they emitted? Or that developers which use QFileSystemWatcher
> should not destroy it until all of its signals have been delivered? This
> seems hard to do. How do C++ developers using Qt do this? Perhaps I can
> learn from their approach.
>
> Bryan
>
> On Fri, Feb 12, 2016 at 3:43 AM, Phil Thompson <
> phil at riverbankcomputing.com> wrote:
>
>> On 11 Feb 2016, at 7:15 pm, Ilya Kulakov <kulakov.ilya at gmail.com> wrote:
>> >
>> > Phil,
>> >
>> > How does Qt's automatic unsubscribing in QObject's destructor works
>> then?
>> >
>> > Best Regards
>> > Ilya Kulakov
>>
>> Unsubscribing to what?
>>
>> Qt maintains lots of internal data structures about connections, but
>> these aren't exposed. As far as I know Qt does not track pointers to
>> QObjects when they are sitting in a thread's event queue.
>>
>> Phil
>>
>> >> On 12 февр. 2016 г., at 0:30, Phil Thompson <
>> phil at riverbankcomputing.com> wrote:
>> >>
>> >>
>> >>> On 11 Feb 2016, at 6:16 pm, Ilya Kulakov <kulakov.ilya at gmail.com>
>> wrote:
>> >>>
>> >>> Phil,
>> >>>
>> >>> I said I don't know all the details :)
>> >>>
>> >>> PyQt "controls" both signals and slots. Doesn't it know how many
>> receivers are there before it sends?
>> >>
>> >> The short answer to the question is no. Also, with queued connections,
>> a signal may be sitting in an unprocessed event queue for an indeterminate
>> amount of time. In network terms, signals are UDP, not TCP.
>> >>
>> >> Phil
>> >
>>
>> _______________________________________________
>> PyQt mailing list PyQt at riverbankcomputing.com
>> https://www.riverbankcomputing.com/mailman/listinfo/pyqt
>>
>
>
>
> --
> Bryan A. Jones, Ph.D.
> Associate Professor
> Department of Electrical and Computer Engineering
> 231 Simrall / PO Box 9571
> Mississippi State University
> Mississippi state, MS 39762
> http://www.ece.msstate.edu/~bjones
> bjones AT ece DOT msstate DOT edu
> voice 662-325-3149
> fax 662-325-2298
>
> Our Master, Jesus Christ, is on his way. He'll show up right on
> time, his arrival guaranteed by the Blessed and Undisputed Ruler,
> High King, High God.
> - 1 Tim. 6:14b-15 (The Message)
>
--
Bryan A. Jones, Ph.D.
Associate Professor
Department of Electrical and Computer Engineering
231 Simrall / PO Box 9571
Mississippi State University
Mississippi state, MS 39762
http://www.ece.msstate.edu/~bjones
bjones AT ece DOT msstate DOT edu
voice 662-325-3149
fax 662-325-2298
Our Master, Jesus Christ, is on his way. He'll show up right on
time, his arrival guaranteed by the Blessed and Undisputed Ruler,
High King, High God.
- 1 Tim. 6:14b-15 (The Message)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.riverbankcomputing.com/pipermail/pyqt/attachments/20160212/ba0733a6/attachment-0001.html>
More information about the PyQt
mailing list