[PyQt] QProcess hangup using PyQt + threads

[Phil Thompson]

On Fri, 26 Feb 2010 12:08:58 +0000, Phil Thompson
<phil at riverbankcomputing.com> wrote:
> On Fri, 26 Feb 2010 11:05:44 +0100, Denis RIVIERE <denis.riviere at cea.fr>
> wrote:
>> We have run into what we think to be more or less a bug in PyQt4, in
>> QProcess, 
>> somewhat indirectly.
>> QProcess sometimes hangsup in threaded PyQt applications, at least on
> Linux
>> 
>> systems.
>> The situation is the following:
>> - A PyQt program is using two or more threads (say, python threads)
>> - one of the threads is using a QProcess (whatever thread), while the
> other
>> is 
>> using python (of course).
>> 
>> We think we have understood what is going on:
>> 
>> Internally, QProcess forks then probably uses a kind of exec().
>> But the child forked process sometimes gets locked on the python GIL
> lock.
>> Then the calling process waits for the child, and also gets hung on a
>> select() 
>> call.
>> 
>> Actually, when forking a multi-threaded process, only the forking thread
> is
>> 
>> duplicated: so the child process has only 1 thread. But the locks,
> mutexes 
>> and other threading items have kept the state they had in their parent 
>> process: some are locked, but the other threads which would normally
>> release 
>> them do not exist in the child.
>> Here, the python GIL happens to be locked by another thread in the
parent
> 
>> process, and the child process seems to try to lock the GIL too. We
guess
> 
>> QProcess calls a virtual method in the forked process, which is
> overloaded
>> by 
>> SIP/PyQt, which in turn calls the Python API, and finally gets hung
> trying
>> to 
>> lock the GIL.
>> We have never encountered this problem using PyQt3, maybe because
> QProcess
>> or 
>> its PyQt binding may be implemented differently.
>> 
>> This threading + fork problem is a known difficulty in fork(), explained
> in
>> 
>> the linux manpages for instance.
>> 
>> We have 2 possible solutions for this problem:
>> 1. Using pthreads, it is possible to define handlers functions that can
>> take 
>> care of locking/unlocking the needed locks whenever fork() is called,
> both
>> in 
>> the parent and child processes: this is done using the pthread_atfork() 
>> function. Typically PyQt may define such handlers to take care of the
GIL
> 
>> locking when fork() is used. However we don't know if such a mechanism
>> exists 
>> on non-pthread systems (Windows?). We also don't really know if the
> problem
>> 
>> exists on Windows; possibly not because Windows doesn't use fork() but 
>> probably rather a spawn equivalent (?).
>> 2. The problem actually happens because the python API gets called from
> the
>> 
>> forked process. Normally, in C++ Qt, the QProcess doesn't trigger such a

>> locking problem. However the SIP binding defines a subclass which traps
> all
>> 
>> virtual methods calls and redirects them to python. So if we are not
> using
>> a 
>> sip-inherited QProcess class, the problem should not happen. We have
made
>> the 
>> test, making a sip-bound factory function which instantiates a QProcess
>> from 
>> C++, then returns it to the pyton layer. This way no inherited class is
>> used, 
>> and the problem is avoided. And it actually works.
>> 
>> Would it be difficult to have solution 1 included in the standard PyQt ?
> 
> Unless you can provide me with a test case you'll have to experiment...
> 
> Try adding...
> 
>     pthread_atfork(NULL, NULL, PyOS_AfterFork);
> 
> ...after the call to PyEval_InitThreads() in siplib.c.

An alternative (which I may be more comfortable with) is to add the
/HoldGIL/ annotation to all QProcess.start() overloads. However the
asynchronous nature of process starts may mean it has no effect.

Phil