[PyQt] QProcess hangup using PyQt + threads

[Denis RIVIERE]

We have run into what we think to be more or less a bug in PyQt4, in QProcess, 
somewhat indirectly.
QProcess sometimes hangsup in threaded PyQt applications, at least on Linux 
systems.
The situation is the following:
- A PyQt program is using two or more threads (say, python threads)
- one of the threads is using a QProcess (whatever thread), while the other is 
using python (of course).

We think we have understood what is going on:

Internally, QProcess forks then probably uses a kind of exec().
But the child forked process sometimes gets locked on the python GIL lock.
Then the calling process waits for the child, and also gets hung on a select() 
call.

Actually, when forking a multi-threaded process, only the forking thread is 
duplicated: so the child process has only 1 thread. But the locks, mutexes 
and other threading items have kept the state they had in their parent 
process: some are locked, but the other threads which would normally release 
them do not exist in the child.
Here, the python GIL happens to be locked by another thread in the parent 
process, and the child process seems to try to lock the GIL too. We guess 
QProcess calls a virtual method in the forked process, which is overloaded by 
SIP/PyQt, which in turn calls the Python API, and finally gets hung trying to 
lock the GIL.
We have never encountered this problem using PyQt3, maybe because QProcess or 
its PyQt binding may be implemented differently.

This threading + fork problem is a known difficulty in fork(), explained in 
the linux manpages for instance.

We have 2 possible solutions for this problem:
1. Using pthreads, it is possible to define handlers functions that can take 
care of locking/unlocking the needed locks whenever fork() is called, both in 
the parent and child processes: this is done using the pthread_atfork() 
function. Typically PyQt may define such handlers to take care of the GIL 
locking when fork() is used. However we don't know if such a mechanism exists 
on non-pthread systems (Windows?). We also don't really know if the problem 
exists on Windows; possibly not because Windows doesn't use fork() but 
probably rather a spawn equivalent (?).
2. The problem actually happens because the python API gets called from the 
forked process. Normally, in C++ Qt, the QProcess doesn't trigger such a 
locking problem. However the SIP binding defines a subclass which traps all 
virtual methods calls and redirects them to python. So if we are not using a 
sip-inherited QProcess class, the problem should not happen. We have made the 
test, making a sip-bound factory function which instantiates a QProcess from 
C++, then returns it to the pyton layer. This way no inherited class is used, 
and the problem is avoided. And it actually works.

Would it be difficult to have solution 1 included in the standard PyQt ?

Denis