[PyQt] Virtual methods and an extra reference

[Sundance]

Kevin Watters wrote:

> I'm tracking down a memory leak in my app--and I think it's boiling
> down to a virtual method on one of my classes that has an extra
> reference, one not coming from any Python object.

Hi Kevin, hi Phil, hi all,

Okay, Kevin, you MUST be reading my mind because, first thing I did this 
morning after yesterday evening's work was fire up my email client and 
start a message entitled "Serious memory leak in PyQt?".

I can confirm the issue and, perhaps, provide a little more data.

The core issue is the way Python handles references around bound 
methods.

Let obj.meth() be a method on an object. By default, the reference count 
to the bound method is zero, and the reference count to the object is, 
exactly, the number of external references you have to that object.

I am not sure how it is that the refcount to the bound method is zero by 
default. I suspect it may have to do with the way methods are 
implemented in Python; i.e. the method is not on the object but on 
the /class/, and the method is only bound to the /object/ at call time.

Let us now extract the bound method under an external label:
m = obj.meth

The method object returned by the implicit getattr is /bound/: it comes 
with a closure that contains the object the method is bound to, and 
that object's refcount thus increases by one.

And here's the issue: what PyQt (well, SIP, really) seems to do with 
virtual methods that are reimplemented in Python, is grab a reference 
to the bound method, and /never let it go/.

Hence, your bound method's refcount never returns to zero. Hence, 
your /object's/ refcount never returns to zero. And you get a memory 
leak.

This is an issue because all the event management in Qt is done with 
virtual methods.


Example code:

---[ Code ]------------------------------------------------------------
import sys
from PyQt4 import QtGui

app = QtGui.QApplication( sys.argv )

class MyWidget( QtGui.QWidget ):

  def resizeEvent( s, e ):
    print "In resizeEvent."
    return QtGui.QWidget.resizeEvent( s, e )

  def __del__( s ):
    print "%s deleted." % s

class MyOtherWidget( QtGui.QWidget ):

  def nonVirtualMethod( s ):
    print "All's good."

  def __del__( s ):
    print "%s deleted." % s

## Let us now test all the possible cases.

obj1 = MyOtherWidget()
obj1.nonVirtualMethod()
del obj1  ## No virtual method, no problem.

obj2 = MyWidget()
del obj2  ## Virtual method never used, no problem.

obj3 = MyWidget()
obj3.show()  ## resizeEvent is bound by SIP.
app.processEvents()
obj3.close()
del obj3     ## ... But never released. Memory leak.
-----------------------------------------------------------------------


Usually this wouldn't be a huge issue, but when you are creating and 
destroying lots of widgets, suddenly it's a thorny problem.


Phil, may we please discuss possible ways to fix this? It might be as 
simple as using weak references to the bound methods instead of direct 
references.

Arguably, 'simple' is perhaps something of a misnommer, because Python's 
weakrefs don't handle bound methods well at all (the weak reference 
dies the moment the bound method goes out of scope).

Fortunately, that can be worked around reasonably simply:


---[ Code ]------------------------------------------------------------
import weakref

class WeakCallableRef:

  def __init__( s, fn, callback=None ):

    assert callable( fn )

    s._methname = None
    s._objref   = None
    s._fnref    = None
    s._dead     = False
    s._callback = callback

    obj = getattr( fn, "im_self", None )

    if obj:  ## fn is a bound method
      s._objref   = weakref.ref( obj, s.markDead )
      s._methname = fn.im_func.func_name

    else:  ## fn is a static method or a plain function
      s._fnref = weakref.ref( fn, s.markDead )


  def markDead( s, ref ):

    s._dead = True
    if s._callback: s._callback( s )


  def __call__( s ):

    if s._dead: return None

    if s._objref:  ## bound method
      return getattr( s._objref(), s._methname, None )

    else:
      return s._fnref()
-----------------------------------------------------------------------

This class behaves like weakref.ref, only for callables.

Might it be possible to have it used by PyQt for virtual methods?

Thanks,

-- S.