[PyQt] QProcess hangup using PyQt + threads

[Phil Thompson]

On Fri, 26 Feb 2010 11:05:44 +0100, Denis RIVIERE <denis.riviere at cea.fr>
wrote:
> We have run into what we think to be more or less a bug in PyQt4, in
> QProcess, 
> somewhat indirectly.
> QProcess sometimes hangsup in threaded PyQt applications, at least on
Linux
> 
> systems.
> The situation is the following:
> - A PyQt program is using two or more threads (say, python threads)
> - one of the threads is using a QProcess (whatever thread), while the
other
> is 
> using python (of course).
> 
> We think we have understood what is going on:
> 
> Internally, QProcess forks then probably uses a kind of exec().
> But the child forked process sometimes gets locked on the python GIL
lock.
> Then the calling process waits for the child, and also gets hung on a
> select() 
> call.
> 
> Actually, when forking a multi-threaded process, only the forking thread
is
> 
> duplicated: so the child process has only 1 thread. But the locks,
mutexes 
> and other threading items have kept the state they had in their parent 
> process: some are locked, but the other threads which would normally
> release 
> them do not exist in the child.
> Here, the python GIL happens to be locked by another thread in the parent

> process, and the child process seems to try to lock the GIL too. We guess

> QProcess calls a virtual method in the forked process, which is
overloaded
> by 
> SIP/PyQt, which in turn calls the Python API, and finally gets hung
trying
> to 
> lock the GIL.
> We have never encountered this problem using PyQt3, maybe because
QProcess
> or 
> its PyQt binding may be implemented differently.
> 
> This threading + fork problem is a known difficulty in fork(), explained
in
> 
> the linux manpages for instance.
> 
> We have 2 possible solutions for this problem:
> 1. Using pthreads, it is possible to define handlers functions that can
> take 
> care of locking/unlocking the needed locks whenever fork() is called,
both
> in 
> the parent and child processes: this is done using the pthread_atfork() 
> function. Typically PyQt may define such handlers to take care of the GIL

> locking when fork() is used. However we don't know if such a mechanism
> exists 
> on non-pthread systems (Windows?). We also don't really know if the
problem
> 
> exists on Windows; possibly not because Windows doesn't use fork() but 
> probably rather a spawn equivalent (?).
> 2. The problem actually happens because the python API gets called from
the
> 
> forked process. Normally, in C++ Qt, the QProcess doesn't trigger such a 
> locking problem. However the SIP binding defines a subclass which traps
all
> 
> virtual methods calls and redirects them to python. So if we are not
using
> a 
> sip-inherited QProcess class, the problem should not happen. We have made
> the 
> test, making a sip-bound factory function which instantiates a QProcess
> from 
> C++, then returns it to the pyton layer. This way no inherited class is
> used, 
> and the problem is avoided. And it actually works.
> 
> Would it be difficult to have solution 1 included in the standard PyQt ?

Unless you can provide me with a test case you'll have to experiment...

Try adding...

    pthread_atfork(NULL, NULL, PyOS_AfterFork);

...after the call to PyEval_InitThreads() in siplib.c.

Phil