[PyQt] operator new/delete and PyMem_New/Del

[Giovanni Bajo]

On 6/4/2008 7:42 PM, Kevin Watters wrote:
> I was profiling my SIP extension, seeing that I allocate lots and lots
> of a certain class of small objects, whose ownership is never 
> transferred to C++. I remembered reading a section from the Python C API
> manual on memory management, here:
> 
> http://docs.python.org/api/memoryOverview.html -- the relevant bit is:
> 
> "In most situations, however, it is recommended to allocate memory 
> from the Python heap specifically because the latter is under control
> of the Python memory manager. For example, this is required when the
> interpreter is extended with new object types written in C. Another
> reason for using the Python heap is the desire to inform the Python
> memory manager about the memory needs of the extension module. Even
> when the requested memory is used exclusively for internal, 
> highly-specific purposes, delegating all memory requests to the 
> Python memory manager causes the interpreter to have a more accurate
> image of its memory footprint as a whole. Consequently, under certain
> circumstances, the Python memory manager may or may not trigger
> appropriate actions, like garbage collection, memory compaction or
> other preventive procedures. Note that by using the C library
> allocator as shown in the previous example, the allocated memory for
> the I/O buffer escapes completely the Python memory manager."
> 
> Overriding operator new and operator delete in SIP's generated code for
> this class with the PyMem_New and PyMem_Delete functions would
> effectively accomplish what that passage is talking about.
> 
> This might, of course, be pointless, without more detailed profiling.
> But I was wondering if anyone had an experience trying anything like
> this this in extensions of their own.

Yes, we do. We use this code:

#include <Python.h>

struct PyMallocObject
{
   static void* operator new (size_t n)
   {
     return PyObject_Malloc(n);
   }

   static void* operator new (size_t n, void *p)
   {
     return ::operator new(n, p);
   }

   static void operator delete(void *p, size_t)
   {
     PyObject_Free(p);
   }

   static void operator delete(void *p, void *a)
   {
     return ::operator delete(p, a);
   }
};

You just inherit the small objects from this class and you're done with. 
If you do many allocations, this should be a win over the standard 
new/delete.
-- 
Giovanni Bajo
Develer S.r.l.
http://www.develer.com