[PyKDE] PyThreadState_Get, PyEval_*Thread, NULL tstate.

[Patrick Stinson]

On Saturday 14 February 2004 01:59 am, Phil Thompson wrote:
> On Saturday 14 February 2004 10:44, Patrick Stinson wrote:
> > On Friday 13 February 2004 22:51, Phil Thompson wrote:
> > > On Saturday 14 February 2004 07:27, Patrick Stinson wrote:
> > > > On Friday 13 February 2004 10:53, Phil Thompson wrote:
> > > > > On Thursday 12 February 2004 20:48, Patrick Stinson wrote:
> > > > > > I posted a while ago about the 'PyEval_RestoreThread: NULL
> > > > > > tstate:' error, and now I'm getting the same problem with
> > > > > > PyThreadState_Get (if I remember correctly, its the call that
> > > > > > happens before PyEval_RestoreThread).
> > > > > >
> > > > > > urls:
> > > > > > http://www.mail-archive.com/pykde%40mats.imk.fraunhofer.de/msg014
> > > > > >37 .h tm l
> > > > > > http://mats.imk.fraunhofer.de/pipermail/pykde/2003-December/00679
> > > > > >8. ht ml and even earlier:
> > > > > > http://www.mail-archive.com/pykde%40mats.imk.fraunhofer.de/msg014
> > > > > >37 .h tm l
> > > > > >
> > > > > > As Before, I'm writing an audio application that uses background
> > > > > > threads for buffering and audio streaming. The threads are
> > > > > > entirely contained in my c++ (sipped) lib and execution never
> > > > > > enters python code. What could cause a NULL thread state in the
> > > > > > interpreter? This occurs at different times, and with what seem
> > > > > > slike varying levels of complexity in pyqt Gui code. Does anyone
> > > > > > know how I should be interpreting this? What does this generally
> > > > > > mean?
> > > > >
> > > > > I can't remember which version of SIP you are using.
> > > > >
> > > > > If you are using SIP v3 then you have to be *extremely* careful
> > > > > about managing the thread state.  Are you sure that "execution
> > > > > never enters python code"?
> > > > >
> > > > > Things are much easier with SIP v4 because it uses the new thread
> > > > > API in Python 2.3.
> > > > >
> > > > > Phil
> > > >
> > > > I will check harder for spots where it might. Currently I am using
> > > > sip and PyQt 3.8, because of the odd problem I was having with
> > > > threads appearing to 'lock up'. How is it that I can 'manage' the
> > > > thread state at the python level? Do you mean I should be careful how
> > > > my py-objects are used by different python threads, as the
> > > > interpreter is not considered thread-safe?
> > >
> > > You have to manage the GIL and the current thread state carefully to
> > > make sure that they are correct before you make any calls to Python
> > > from C/C++. The problem with the old API is that there is no way to
> > > safely determine the current state.
> > >
> > > SIP v3 deals with this by having a very clear sense of "Python-land"
> > > and "C++-land". The GIL is always released when going from Python to
> > > C++ and always acquired when going from C++ to Python. If you don't
> > > follow similar conventions then you will have problems.
> > >
> > > With virtual functions it can sometimes be difficult to see exactly
> > > when execution crosses the Python/C++ border. Sometimes I found calls
> > > to Python where I didn't expect them because deep in Qt a call to a
> > > virtual of a wrapped instance was made. This is why I added the -r flag
> > > to SIP to generate code to trace the execution.
> > >
> > > The new thread API used by SIP v4 is much easier to use because you can
> > > safely acquire the GIL without knowing if you already have it. The GIL
> > > is only released when going from Python to C++ if the call might take
> > > significant time to execute.
> > >
> > > Phil
> > >
> > > _______________________________________________
> > > PyKDE mailing list    PyKDE at mats.imk.fraunhofer.de
> > > http://mats.imk.fraunhofer.de/mailman/listinfo/pykde
> >
> > wow, thank you very much for the in-depth response. Because my
> > sip-wrapped lib only provides class-types, virtuals (which do exist
> > there) are the only way 'out' of c++ and into python. In fact,
> > subclassing those types in python is a large part of my design. I'm
> > getting the hint that if you are wrapping a c/c++ library that has no
> > knowledge of python (like qt) and therefore cannot control attempts on
> > the GIL, you really shouldn't subclass using virtuals in that fashion
> > with sip3.
>
> There is no problem with doing this as SIP generates the code to handle it.
> Problems can arise if...
>
> - your library does things with threads
>
> - your wrappers have handwritten code that doesn't follow the border
> conventions
>
> - you are interacting with wrappers created by another wrapper generator
> that doesn't handle the GIL (eg. Boost).
>
> > But sip3-wrapped qt widgets are
> > meant to be subclassed, hence my confusion.
> > I'll understand if I've carried this on too long.
>
> No problem.
>
> Phil
>
> _______________________________________________
> PyKDE mailing list    PyKDE at mats.imk.fraunhofer.de
> http://mats.imk.fraunhofer.de/mailman/listinfo/pykde


Just to prod this a bit further, my qpp workds quite well with different 
threads when I use qt classes. After thinking a bit more about this 
particular stack trace,

pkaudio: Deregistering module: Main Mixer 0
pkaudio: Done.
Fatal Python error: PyEval_RestoreThread: NULL tstate
WAD: Collecting debugging information...
WAD: Abort.
#19  0x08054da1 in ?()
#18  0x4017ac6c in __libc_start_main()
#17  0x401907c0 in exit()
#16  0x41dc65cc in __tcf_1()
#15  0x41dbaf15 in _ZN2PK6EngineD1Ev()
#14  0x41dc16e4 in _ZN2PK5MixerD0Ev()
#13  0x41dc2c48 in _ZN2PK6ModuleD2Ev()
#12  0x41dc6f8e in _ZN2PK11PassivePortD0Ev()
#11  0x41dc680b in _ZN2PK4PortD2Ev()
#10  0x41dc6c71 in _ZN2PK4Port10disconnectEv()
#9   0x41dc6d79 in _ZN2PK4Port10disconnectEv()
#8   0x41ddfcca in _ZN15sipPK_PanEffect12disconnectedEPN2PK4PortE()
#7   0x4001f28a in sipIsPyMethod()
#6   0x400225ea in sipBlockThreads()
#5   0x080b9f0a in PyEval_RestoreThread()
#4   0x080e8611 in Py_AtExit()
#3   0x4018f708 in abort()
#2   0x4018e1d4 in gsignal()
#1   0x40034f1b in raise()
#0   0x4018e421 in kill()
ajole at ajole pksampler $

I've come to the conclusion that the fact that these two lines

#8   0x41ddfcca in _ZN15sipPK_PanEffect12disconnectedEPN2PK4PortE()
#7   0x4001f28a in sipIsPyMethod()

generate the 'NULL tstate error' is perfectly valid, considering your 
information above about C++ code not aquiring the GIL (assuming that the 
PK::PanEffect is actually instantiated from python). 

What is a bit confusing about this is that while PK::PanEffect::disconnected() 
*is* virtual, and the sip implementation *is* being called, the PanEffect 
object in question was not a python subclass, but was simply an instantiation 
of a sip-wrapped class from python just like sip-wrapped qt objects. 

Why does this work with qt objects that rely on their virtual methods  
(especially in multi-threaded apps), and not in mine? Interesting topic.

Cheers!
-P