[PyQt] QProcess hangup using PyQt + threads
denis.riviere at cea.fr
Fri Feb 26 10:05:44 GMT 2010
We have run into what we think to be more or less a bug in PyQt4, in QProcess,
QProcess sometimes hangsup in threaded PyQt applications, at least on Linux
The situation is the following:
- A PyQt program is using two or more threads (say, python threads)
- one of the threads is using a QProcess (whatever thread), while the other is
using python (of course).
We think we have understood what is going on:
Internally, QProcess forks then probably uses a kind of exec().
But the child forked process sometimes gets locked on the python GIL lock.
Then the calling process waits for the child, and also gets hung on a select()
Actually, when forking a multi-threaded process, only the forking thread is
duplicated: so the child process has only 1 thread. But the locks, mutexes
and other threading items have kept the state they had in their parent
process: some are locked, but the other threads which would normally release
them do not exist in the child.
Here, the python GIL happens to be locked by another thread in the parent
process, and the child process seems to try to lock the GIL too. We guess
QProcess calls a virtual method in the forked process, which is overloaded by
SIP/PyQt, which in turn calls the Python API, and finally gets hung trying to
lock the GIL.
We have never encountered this problem using PyQt3, maybe because QProcess or
its PyQt binding may be implemented differently.
This threading + fork problem is a known difficulty in fork(), explained in
the linux manpages for instance.
We have 2 possible solutions for this problem:
1. Using pthreads, it is possible to define handlers functions that can take
care of locking/unlocking the needed locks whenever fork() is called, both in
the parent and child processes: this is done using the pthread_atfork()
function. Typically PyQt may define such handlers to take care of the GIL
locking when fork() is used. However we don't know if such a mechanism exists
on non-pthread systems (Windows?). We also don't really know if the problem
exists on Windows; possibly not because Windows doesn't use fork() but
probably rather a spawn equivalent (?).
2. The problem actually happens because the python API gets called from the
forked process. Normally, in C++ Qt, the QProcess doesn't trigger such a
locking problem. However the SIP binding defines a subclass which traps all
virtual methods calls and redirects them to python. So if we are not using a
sip-inherited QProcess class, the problem should not happen. We have made the
test, making a sip-bound factory function which instantiates a QProcess from
C++, then returns it to the pyton layer. This way no inherited class is used,
and the problem is avoided. And it actually works.
Would it be difficult to have solution 1 included in the standard PyQt ?
More information about the PyQt