[PyKDE] How to contribute to sip ?

[Gerard Vermeulen]

On Tue, 28 Feb 2006 10:18:31 -0600
Nigel Stewart <ns at fluent.com> wrote:

> 
> >> It could be argued that SIP is part of a broader toolchain 
> > 
> > What does this have to do with SIP again? I don't remember GCC not
> > generating the object files if the source code is not changed.
> 
>      A compiler takes one input file and creates one output file.
> 
>      The granularity of SIP is different, it's taking multiple
>      inputs (via %Import) and creating multiple outputs.  This
>      design poses a "choke point".  I think being time-stamp
>      friendly is a way for SIP to compensate for this.
> 
> > So, why can't you configure your build system so that it doesn't call SIP in
> > the first place if the header file has not changed?
> 
>      Here is the scenario - the header foo.h is changed, triggering
>      SIP to regenerate the bindings.  In most cases, the change to
>      foo.h doesn't affect the code that SIP outputs.  Of the 250
>      .cpp files generated by SIP, 10 depend on foo.h.  However,
>      all 250 are rebuilt due to SIP changing the time stamp.
> 
>      That's a 25x slowdown that provides me more time to surf the
>      PyKDE mailing list...  :-)
> 
>      Perhaps our situation is not typical, it doesn't affect our
>      developers that _depend_ on our SIP bindings.  But it certainly
>      affects development _upstream_ of the SIP bindings.
> 
>      Imagine someone hacking away at Qt and depending on PyQt
>      based regression tests to know if they've broken anything...
>      That's my day-to-day situation in a nutshell...
> 

My configure.py scripts put the the output of sip in a temporary
directory and they use the following function to copy the new files
to the build directory:

def lazy_copy_file(source, target):
    '''Lazy copy a file to another file:
    - check for a SIP time stamp to skip,
    - check if source and target do really differ,
    - copy the source file to the target if they do,
    - return True on copy and False on no copy.
    '''
    if not os.path.exists(target):
        shutil.copy2(source, target)
        return True

    sourcelines = open(source).readlines()
    targetlines = open(target).readlines()

    # global length check
    if len(sourcelines) != len(targetlines):
        shutil.copy2(source, target)
        return True

    # skip a SIP time stamp
    if (len(sourcelines) > 3
        and sourcelines[3].startswith(' * Generated by SIP')
        ):
        line = 4
    else:
        line = 0

    # line by line check
    while line < len(sourcelines):
        if sourcelines[line] != targetlines[line]:
            shutil.copy2(source, target)
            return True
        line = line + 1

    return False

# lazy_copy_file()

The function is invoked by code like this:

    lazy_copies = 0
    for pattern in ('*.c', '*.cpp', '*.h', '*.sbf'):
        for source in glob.glob(os.path.join(tmp_dir, pattern)):
            target = os.path.join(build_dir, os.path.basename(source))
            if lazy_copy_file(source, target):
                print "Copy %s -> %s." % (source, target)
                lazy_copies += 1
    print "%s file(s) lazily copied." % lazy_copies


If you are using g++, you should look into ccache.samba.org.
This gives additional speed-ups of about a factor 10.

Gerard