[PyKDE] Re: Issues using SIP for small C++ classes

[Phil Thompson]

On Thursday 07 December 2006 5:11 pm, Giovanni Bajo wrote:
> Hello,
>
> I'm trying to use SIP to wrap a simple C++ vector library. The simplest
> example I'll use is a 2d vector:
>
> struct P
> {
>    double x, y;
>
>    operator+(...);
>    operator-(...);
>    // etc.
> };
>
> The whole wrapper works correctly, but I have issues with scalability
> concerns. Let me explain.
>
> It looks like that SIP only works with *pointers* to C++ objects: this is
> crucial to its design. Basically, there is no way I can make it generate a
> PyObject with the "struct P" embedded in it. sipWrapper always holds a
> *pointer* to P. This contradicts the usual semantic of P in C++, which is a
> lightweight aggregate of two doubles, meant to be used on the stack (and
> even decomposed to its members by the C++ compiler), without ever going
> through a heap allocation.
>
> For reference, if I reimplement P in Pyrex, I obtain this:
>
> struct __pyx_obj_6_geo2d_P {
>   PyObject_HEAD
>   double x;
>   double y;
> };
>
> while the sipWrapper looks like this:
>
> typedef struct _sipWrapper {
>         PyObject_HEAD
>         PyObject *user;   /* For the user to use. */
>         union {
>                 void *cppPtr;  /* C/C++ object pointer. */
>                 void *(*afPtr)(); /* Access function. */
>         } u;
>         int flags;   /* Object flags. */
>         PyObject *dict;   /* The instance dictionary. */
>         struct _sipPySig *pySigList; /* Python signal list (complex). */
>         struct _sipWrapper *next; /* Next object at this address. */
>         struct _sipWrapper *first_child; /* First child object. */
>         struct _sipWrapper *sibling_next; /* Next sibling. */
>         struct _sipWrapper *sibling_prev; /* Previous sibling. */
>         struct _sipWrapper *parent; /* Owning object. */
> } sipWrapper;
>
>
> This produces several bad pessimizations:
>
> - SIP will always go through a heap allocation for every function that
> returns a P ("new P"). It will also allocate the PyObject of course, but
> that's necessary (and will go through the python obmalloc which I kind of
> trust). It will *also* allocate a dict (see below). The overhead of these
> heap allocations is dramatic.
>
> - sizeof(sipWrapper) is large. Too large in this case: I create literally
> millions of these objects... if there was infrastructure in-place to
> generate sipWrapper-like types specific for each wrapped C++ type, I guess
> that it could be modularized better, so not to waste memory for features
> that are not wanted/needed/used (like the "user" pointer, or the signal
> list).
>
> - Member accessors are properties in SIP, and they go through a function
> call, even if I don't want to add any specific %MethodCode. The Pyrex
> equivalent is able to go to the actual member like builtin types do
> (without any custom code):
>
> static struct PyMemberDef __pyx_members_6_geo2d_P[] = {
>   {"x", T_DOUBLE, offsetof(struct __pyx_obj_6_geo2d_P, x), READONLY, 0},
>   {"y", T_DOUBLE, offsetof(struct __pyx_obj_6_geo2d_P, y), READONLY, 0},
>   {0, 0, 0, 0, 0}
> };
>
> - Every instance of sipWrapper has an instance dictionary, created in
> sipWrapper_init(). There does not seem to be a way to tell SIP not to
> create this dictionary (which is totally wasted memory in my case): P is an
> immutable type, and I don't need/want other attributes added to it.
>
> In the end, real-world benchmarks showed almost a 100% increase in runtime
> by using SIP to wrap the C++ library (compared to an API-identical Pyrex
> implementation of the library). I assume that memory occupation (and
> fragmentation) also grew a lot, but I haven't produced numbers yet.
>
> So, is there something that can be done? Is SIP the wrong tool for wrapping
> this kind of lightweight C++ objects? Or would there be something that
> could be planned to improve this situation? I'm happy to contribute
> something, if there is something that can be done about it.

The short answer is that there is nothing that can be done in the short term.

Note that Pyrex, as you describe, is not wrapping C++ classes - it is creating 
Python equivalents of C++ classes (or do I mean C structs?). I'm not sure how 
you would represent any class methods - and I don't see how they could work 
anyway.

I think that the objects you could emulate would be so lightweight that I 
wouldn't even bother with C++ in the first place - unless space is the 
showstopper.

Phil