[PyQt] Running Python scripts from a SIP-wrapped library

[Matt Newell]

On Saturday 30 July 2011 13:33:55 Jens Thoms Toerring wrote:
> Hi,
> 
>    I guess this isn't the best place to ask but I haven't
> found anything really fitting yet. So, please, if this is
> too off-topic don't hesitate to redirect me to a better
> place (a "RTFM" is also going to be appreciated as long
> as it comes with a pointer to the relevant "M";-).
> 
> In short, I have a C++ library that has a SIP wrapper, so
> it can be used from both Python and C++. Within the library
> it should be possible to execute "modules", i.e. Python
> scripts, using the Python C-API. Those "modules" also need
> to 'import' the SIP-wrapper for the library.
> 
> I got this working using the following approach: In the
> code that runs a Python "module" I check first if Python
> is already running (that would be the case when the li-
> brary is used from Python via the SIP-wrapper) by testing
> with Py_IsInitialized(). If that isn't the case I call
> PyInitialize(). On the other hand, when it's true I start
> a new Python interpreter, using Py_NewInterpreter()
> (creating a new interpreter seems to be necessary since
> otherwise Python crashes somewhere in its innards with
> a segmentation fault once I call PyImport_Import() on
> the Python module to be executed). Once the "module" is
> done I stop the new interpreter with Py_EndInterpreter()
> (if one was started) or just call PyFinalize().
> 
>   All this seems to work fine when the Python script
> using my library is started from the command line. But
> now someone wanted to use spyder
> 
>   http://packages.python.org/spyder/
> 
> And if the Python script is run from spyder all kinds of
> weired things happen. In some cases the script simply seems
> to hang (one place seems to be the call of Py_NewInterpre-
> ter() as far as I can tell at the moment - I haven't found
> a good way to run spyder under a debugger and then getting
> at the place where things happen) or it aborts with one
> of several error messages about 'tstate mixup' and similar
> when the module has been run and the interpreter is removed.
> 
> I have been googling and trying to understand the docu-
> mentation I found for several days now and am still com-
> pletely confused. I haven't found some really coherent
> description what all this stuff what the GIL and Thread-
> States are all about and how far that could be at the
> heart of my problems. The C-API reference documentation
> is nice as far as it goes, but it's mainly a description
> of a lot of functions with not enough background informa-
> tion for me to help me figure out what actually is going
> wrong.
> 
> Perhaps someone with more knowledge about the details of
> Python and interfacing with C/C++ can tell me either that
> my whole approach is completely idiotic or, if it's not
> that bad, can give me some hints where I'm going wrong
> - or at least has some tips for things I should read and
> try to understand...
>                        Thanks and best regards, Jens


My quick guess is that you should not be calling Py_NewInterpreter, but 
instead wrap your calls into python code with

SIP_BLOCK_THREADS
// Call python functions
SIP_UNBLOCK_THREADS

These are just macros for calling PyGILState_Ensure/PyGILState_Release, so you 
can call them directly if you aren't doing this inside a sip module.

#define SIP_BLOCK_THREADS   {PyGILState_STATE sipGIL = PyGILState_Ensure();
#define SIP_UNBLOCK_THREADS PyGILState_Release(sipGIL);}


Matt