[PyQt] multiprocessing with QApplication

[Kovid Goyal]

Don't use multiprocessing. multiprocessing is not thread safe, on unix
it uses fork() without exec() which means that it inherits *everything*
from the parent process including locks (which are in an invalid state
in the child process), file handles, global objects like QApplication
and so on. Just as an illustration of the problems multiprocessing can
cause, if you use it with the standard library logging module you can
have your worker processes crash, since the logging module uses locks.

After all, what do you expect to happen to QApplication on fork()?
There's no way for fork() to have the QApplication object magically
re-initialize itself.

Using multiprocessing *will* bite you in the rear on any project of
moderate complexity. Heck it bit me on the rear even while implementing
a muti-core replacment for grep. Instead use subprocess to launch a
worker process, feed it a module name, functions and arguments on stdin
using cPickle or json and then have it run the task. About the only use
cases that I can think of that multiprocessing is suitable for are:

1) Toy examples to demonstrate how "easy" using multiple processes in
python is
2) Performance critical applications where the overhead of launching and
initializing a new python interpreter should be avoided. (And this only on
Unix, since on windows the interpreter has to be restarted anyway).
3) Tasks where the processes need to share memory, which is somewhat
easier to do with multiprocessing than other schemes, though you should
only do this if you *really* know what you are doing and are very
careful about the side-effects of fork.

The one really useful thing in the multiprocessing module is the
Listener and Client classes that allow you to establish easy
communication between your parent and child process. Though even there,
the implementation in multiprocessing is not EINTR safe so you have to
wrap calls to accept(), read(), write() in a wrapper that deals with
EINTR.

Kovid.

On Mon, Jun 11, 2012 at 11:44:27AM -0400, Luke Campagnola wrote:
> Howdy,
> 
> I am trying to use multiprocessing in a PyQt application to spawn an extra
> process with its own GUI. The problem I am running into is that the child
> process seems to be instantiated with a broken QApplication instance that
> causes the child to crash. If I understand multiprocessing correctly, child
> processes will inherit some resources from the parent process (file
> handles, for example) and I suspect this is leading the child process to
> believe it already has a QApplication.
> 
> I have attached a simple script that demonstrates this problem. If the
> child process is spawned before the parent process instantiates its
> QApplication, then everything works as expected. If the parent QApplication
> is created before spawning the child process, then we see that  1) the
> child thinks it already has a QApplication and 2) the program crashes,
> presumably because the QApplication was never properly instantiated by the
> child. (If I try creating a new QApplication in the child then the process
> simply hangs or has other erratic behavior, which is usually the result of
> creating more than one QApplication).
> 
> Any experience / suggestions?
> 
> Thanks,
> Luke
> 
> 
> !DSPAM:3,4fd6128d18074722032819!


> _______________________________________________
> PyQt mailing list    PyQt at riverbankcomputing.com
> http://www.riverbankcomputing.com/mailman/listinfo/pyqt
> 
> !DSPAM:3,4fd6128d18074722032819!


-- 
_____________________________________

Dr. Kovid Goyal 
http://www.kovidgoyal.net
http://calibre-ebook.com
_____________________________________
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://www.riverbankcomputing.com/pipermail/pyqt/attachments/20120611/323f5bd9/attachment.pgp>