[PyQt] Mixture of C++/Python and threading

[Martin Dobias]

Hi list,

for Quantum GIS project [1] we have been adding support for rendering
of maps in threads by the means of QtConcurrent framework. Rendering
is started asynchronously from the main thread, spawning several
worker threads where all the painting is done. This works very well.

We have also PyQt4-based bindings which are used by 3rd party plugins.
The rendering stack allows custom implementation of some classes. The
problem starts at the point if any part of the rendering stack (now
running in worker threads) is implemented in Python. Please note that
this is not so typical situation since the core implementation is in
C++, this is just the case when a Python plugin implements custom
rendering for one of the map layers.

With the multithreaded rendering and Python code in the rendering
stack, the rendering gets stuck: GIL that has been acquired by the
main thread has not been released, so the Python code in a worker
thread keeps waiting for GIL. My question is if there is any simple
and elegant solution for this problem.

First thing I have tried was to run some Python code in the main
thread while multithreaded rendering is waiting for GIL. This worked
as expected: Python interpreter periodically released and acquired
GIL, so the worker thread was eventually able to finish its work. So a
silly solution would be to use a timer to run a small amount of Python
code in intervals in order to allow worker thread to acquire GIL. I
know that is stupid and ineffective.

I have also tried to explicitly release GIL when rendering starts and
acquire it again when finished. This worked, but only until I have
tried to run some Python code in main thread while still rendering. I
understand that trying to run Python code without acquiring GIL will
inevitably lead to a segfault, though I have failed to find a way
around this with Python API (the docs are clear as mud for this topic,
as someone already noted). Ideally I would like to tell SIP that when
it is going to run some Python code (e.g. when a slot has been
invoked), it has to acquire GIL, run it, and finally release GIL again
- so that rendering in threads can continue.

My last idea was that if I were able to identify whether any class of
the rendering stack (for a map layer) has Python implementation, I
could run the rendering of such map layer in main thread, effectively
avoiding the multithreading issues. But I would like to solve it more
generically with correct GIL handling.

Please do you have any ideas how to solve this problem?

[1] http://qgis.org/

Thanks
Martin