[Scons-dev] Optimizing SCons...

[Dirk Bächle]

Hi Anatoly,

On 04.12.2012 06:57, anatoly techtonik wrote:

> [...]

>

> It will be easier to review this on rietveld.


I don't think we have reached full review state yet. As the comments and 
suggestions (regarding properties, for example) show, the patch still 
might need some work.
After getting more input from other users, I can upload the single 
commits to a branch/repository somewhere and then also a review can 
happen in the form of a normal pull request.


>     [...]

>

>

> Do you have stats what objects contribute to the most amount of memory 

> being taken and what are the dependencies of these objects from the 

> inputs? It is interesting to see what is the major reason to SCons 

> memory growth. Maybe we should bzip strings on the fly. =)

>

> Does this slot optimization makes sense under pypy? I think it is a 

> way to go anyway if you want to optimize and speed up anything.


The main portions of SCons are spent on:

   1.) memoizing values/results  (=caching), and
   2.) Nodes.

By reducing the caching, you slow SCons down...I didn't want this. So I 
took care of the nodes instead.

I found this out by:

   ad 1.) removing all the _memo dicts (= no caching), and
   ad 2.) adding my own Node class with the same memory layout. For 
every Node that got created, I added
             a copy...and then gradually removed attributes from this 
"shadow" node class.

One of the basic problems is, that we always have to keep the whole tree 
with all its nodes until the very end. It would help a lot if we could 
throw away visited nodes, but this is not possible as long as we allow 
the DAG to change during the build, while still correctly tracking 
implicit dependencies.


>

>     [...]

>

>     sconsbld     540MB       377MB

>

>     =================================

>

>     They list the maximum of allocated memory for a clean build,

>     before and

>     after the optimizations.

>

>

> Impressive. But still 540MB of memory is not much for the most complex 

> build.

>


As I wrote in my email, I also compared a project that used about 1.5GB 
with the current default SCons version. I can't test anything larger 
than that, because I only have a 2GB machine. If you have a bigger 
project available just take the patch and run it through, with and 
without slots. That's what this thread is all about... ;)


>     So, that's what I have right now. My questions are:

>

>     - Is this of any interest (or is it still "too early" for

>       optimizations ;) )?

>     - If yes, what could be the next steps?

>

>

> I'd say - yes. It is too early for me. I don't feel that SCons code 

> layout is clean enough - there is still magic that could be less 

> magic. Not all tags are present in repository to compare. No pictures 

> in documentation, and no working bug tracker.

>


My very personal impression is: the SCons design is very good already. I 
had to delve deeper into it while compiling my patch, and I'm liking it 
in its current state. It will prove to be very hard to come up with a 
different design, which saves significantly more memory while keeping 
all the functionality and level of flexibility that we have now.
SCons will always be sort of a memory hog, unless you cut out major 
features like the dependency tracking or the interactive mode. We can 
only try to trim down the inherently humongous memory consumption and 
that's what I wanted to accomplish.

Regards,

Dirk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://two.pairlist.net/pipermail/scons-dev/attachments/20121204/6c5e37bb/attachment.html>