[PyQt] Mixture of C++/Python and threading

[Giovanni Bajo]

On Wed, 2011-02-09 at 10:57 +0100, Martin Dobias wrote:
> Hi list,
> 
> for Quantum GIS project [1] we have been adding support for rendering
> of maps in threads by the means of QtConcurrent framework. Rendering
> is started asynchronously from the main thread, spawning several
> worker threads where all the painting is done. This works very well.
> 
> We have also PyQt4-based bindings which are used by 3rd party plugins.
> The rendering stack allows custom implementation of some classes. The
> problem starts at the point if any part of the rendering stack (now
> running in worker threads) is implemented in Python. Please note that
> this is not so typical situation since the core implementation is in
> C++, this is just the case when a Python plugin implements custom
> rendering for one of the map layers.
> 
> With the multithreaded rendering and Python code in the rendering
> stack, the rendering gets stuck: GIL that has been acquired by the
> main thread has not been released, so the Python code in a worker
> thread keeps waiting for GIL. My question is if there is any simple
> and elegant solution for this problem.
> 
> First thing I have tried was to run some Python code in the main
> thread while multithreaded rendering is waiting for GIL. This worked
> as expected: Python interpreter periodically released and acquired
> GIL, so the worker thread was eventually able to finish its work. So a
> silly solution would be to use a timer to run a small amount of Python
> code in intervals in order to allow worker thread to acquire GIL. I
> know that is stupid and ineffective.
> 
> I have also tried to explicitly release GIL when rendering starts and
> acquire it again when finished. This worked, but only until I have
> tried to run some Python code in main thread while still rendering. I
> understand that trying to run Python code without acquiring GIL will
> inevitably lead to a segfault, though I have failed to find a way
> around this with Python API (the docs are clear as mud for this topic,
> as someone already noted). Ideally I would like to tell SIP that when
> it is going to run some Python code (e.g. when a slot has been
> invoked), it has to acquire GIL, run it, and finally release GIL again
> - so that rendering in threads can continue.
> 
> My last idea was that if I were able to identify whether any class of
> the rendering stack (for a map layer) has Python implementation, I
> could run the rendering of such map layer in main thread, effectively
> avoiding the multithreading issues. But I would like to solve it more
> generically with correct GIL handling.
> 
> Please do you have any ideas how to solve this problem?

Which version of PyQt4/sip are you using? In recent PyQt versions, GIL
is always released when entering Qt.

In fact, sip does not release the GIL by default when calling a C++
libraries, but it supports a "-g" option that does just that. You can
check that PyQt4's configure.py, in sip_set_flags(), passes "-g" to sip
while compiling.

You might want to make sure that all your bindings/plugins/whatever that
are sip-based are compiled with the "-g" flag.
-- 
Giovanni Bajo   ::  rasky at develer.com
Develer S.r.l.  ::  http://www.develer.com

My Blog: http://giovanni.bajo.it
Last post: Compile-time Function Execution in D