[Scons-dev] Optimizing SCons...

[Dirk Bächle]

Hi,

over the last days I created a patch that aims at reducing SCons'
memory footprint, especially for large (in terms of number of
files) C/C++ projects.
It can be applied to the current latest revision (b496d47c4efb)
and is attached as archive to this email.

The amount of changes is split into the basic steps:

1) Make Node classes use slots.
2) Make SigInfo classes use slots.
3) Make Executor and Batch use slots and
    stop caching full paths in File nodes.


I had to rewrite the memoizer count framework to use
decorators instead of the original metaclass approach.
Additionally, I had to correct a lot of tests and Tools
to ensure that they still pass without any fails
(at least for those tests that I can run locally under Linux,
some further adaptions might be necessary).

Here are some numbers for the testsuite that I compiled
(it consists of several real-life SCons projects):

Project  |  before   |   after  |
=================================

ascend        96MB        82MB
bombono      120MB       104MB
lumiera      114MB       101MB
mapnik       148MB       129MB
sconsbld     540MB       377MB

=================================

They list the maximum of allocated memory for a clean build, before and
after the optimizations.
For the first four projects the results don't show that much of an
improvement. But this is because they are still relatively small,
compared to the basic overhead that is needed while parsing the
SConstruct/SConscript files for example.
The last (sconsbld) is a benchmark, created by a Perl script,
with a number of 12000 source files, and 12000 includes...resulting in
12000 object files and 600 executables.
Increasing the number of files seems to drive up the percentage
of memory you can save. Doubling them up for "sconsbld" (24000 source
and include files) gives 1579MB vs 1052MB.

I also used cProfile and simple timing of the single runs with the
"time" command, to ensure that no speed penalties are involved with
the changes. Good news is that all runs, clean builds as well as
updates, tend to get a little faster...probably due to the usage
of slots.

So, that's what I have right now. My questions are:

- Is this of any interest (or is it still "too early" for
   optimizations ;) )?
- If yes, what could be the next steps?

I didn't simply upload this as a pull request, because the changes
would definitely break existing projects and custom Tools.
So, either some form of deprecation cycle or a separate branch/repository
would be needed (default vs optimized).

Comments anyone?

Best regards,

Dirk

-------------- next part --------------
A non-text attachment was scrubbed...
Name: changes.tgz
Type: application/x-compressed-tar
Size: 43006 bytes
Desc: not available
Url : <http://two.pairlist.net/pipermail/scons-dev/attachments/20121202/11b909c2/attachment-0001.bin>