[PyKDE] Issues using SIP for small C++ classes

[Giovanni Bajo]

Hello,

I'm trying to use SIP to wrap a simple C++ vector library. The simplest
example I'll use is a 2d vector:

struct P
{
   double x, y;

   operator+(...);
   operator-(...);
   // etc.
};

The whole wrapper works correctly, but I have issues with scalability
concerns. Let me explain.

It looks like that SIP only works with *pointers* to C++ objects: this is
crucial to its design. Basically, there is no way I can make it generate a
PyObject with the "struct P" embedded in it. sipWrapper always holds a
*pointer* to P. This contradicts the usual semantic of P in C++, which is a
lightweight aggregate of two doubles, meant to be used on the stack (and
even decomposed to its members by the C++ compiler), without ever going
through a heap allocation.

For reference, if I reimplement P in Pyrex, I obtain this:

struct __pyx_obj_6_geo2d_P {
  PyObject_HEAD
  double x;
  double y;
};

while the sipWrapper looks like this:

typedef struct _sipWrapper {
        PyObject_HEAD
        PyObject *user;   /* For the user to use. */
        union {
                void *cppPtr;  /* C/C++ object pointer. */
                void *(*afPtr)(); /* Access function. */
        } u;
        int flags;   /* Object flags. */
        PyObject *dict;   /* The instance dictionary. */
        struct _sipPySig *pySigList; /* Python signal list (complex). */
        struct _sipWrapper *next; /* Next object at this address. */
        struct _sipWrapper *first_child; /* First child object. */
        struct _sipWrapper *sibling_next; /* Next sibling. */
        struct _sipWrapper *sibling_prev; /* Previous sibling. */
        struct _sipWrapper *parent; /* Owning object. */
} sipWrapper;


This produces several bad pessimizations:

- SIP will always go through a heap allocation for every function that
returns a P ("new P"). It will also allocate the PyObject of course, but
that's necessary (and will go through the python obmalloc which I kind of
trust). It will *also* allocate a dict (see below). The overhead of these
heap allocations is dramatic.

- sizeof(sipWrapper) is large. Too large in this case: I create literally
millions of these objects... if there was infrastructure in-place to
generate sipWrapper-like types specific for each wrapped C++ type, I guess
that it could be modularized better, so not to waste memory for features
that are not wanted/needed/used (like the "user" pointer, or the signal
list).

- Member accessors are properties in SIP, and they go through a function
call, even if I don't want to add any specific %MethodCode. The Pyrex
equivalent is able to go to the actual member like builtin types do (without
any custom code):

static struct PyMemberDef __pyx_members_6_geo2d_P[] = {
  {"x", T_DOUBLE, offsetof(struct __pyx_obj_6_geo2d_P, x), READONLY, 0},
  {"y", T_DOUBLE, offsetof(struct __pyx_obj_6_geo2d_P, y), READONLY, 0},
  {0, 0, 0, 0, 0}
};

- Every instance of sipWrapper has an instance dictionary, created in
sipWrapper_init(). There does not seem to be a way to tell SIP not to create
this dictionary (which is totally wasted memory in my case): P is an
immutable type, and I don't need/want other attributes added to it.

In the end, real-world benchmarks showed almost a 100% increase in runtime
by using SIP to wrap the C++ library (compared to an API-identical Pyrex
implementation of the library). I assume that memory occupation (and
fragmentation) also grew a lot, but I haven't produced numbers yet.

So, is there something that can be done? Is SIP the wrong tool for wrapping
this kind of lightweight C++ objects? Or would there be something that could
be planned to improve this situation? I'm happy to contribute something, if
there is something that can be done about it.

Thanks!
-- 
Giovanni Bajo