[Scons-dev] Optimizing SCons...

[William Deegan]

On Dec 3, 2012, at 9:57 PM, anatoly techtonik <techtonik at gmail.com> wrote:


> On Sun, Dec 2, 2012 at 5:01 PM, Dirk Bächle <tshortik at gmx.de> wrote:

> Hi,

> 

> over the last days I created a patch that aims at reducing SCons'

> memory footprint, especially for large (in terms of number of

> files) C/C++ projects.

> It can be applied to the current latest revision (b496d47c4efb)

> and is attached as archive to this email.

> 

> It will be easier to review this on rietveld.


Are you unable to review it on bitbucket?


>  

> The amount of changes is split into the basic steps:

> 

> 1) Make Node classes use slots.

> 2) Make SigInfo classes use slots.

> 3) Make Executor and Batch use slots and

>    stop caching full paths in File nodes.

> 

> Do you have stats what objects contribute to the most amount of memory being taken and what are the dependencies of these objects from the inputs? It is interesting to see what is the major reason to SCons memory growth. Maybe we should bzip strings on the fly. =)

> 

> Does this slot optimization makes sense under pypy? I think it is a way to go anyway if you want to optimize and speed up anything.

> 

> I had to rewrite the memoizer count framework to use

> decorators instead of the original metaclass approach.

> Additionally, I had to correct a lot of tests and Tools

> to ensure that they still pass without any fails

> (at least for those tests that I can run locally under Linux,

> some further adaptions might be necessary).

> 

> Here are some numbers for the testsuite that I compiled

> (it consists of several real-life SCons projects):

> 

> Project  |  before   |   after  |

> =================================

> 

> ascend        96MB        82MB

> bombono      120MB       104MB

> lumiera      114MB       101MB

> mapnik       148MB       129MB

> sconsbld     540MB       377MB

> 

> =================================

> 

> They list the maximum of allocated memory for a clean build, before and

> after the optimizations.

> 

> Impressive. But still 540MB of memory is not much for the most complex build.


It's still quite a bit of memory, so reducing it and getting the speedup is a worthy effort.


> 

> So, that's what I have right now. My questions are:

> 

> - Is this of any interest (or is it still "too early" for

>   optimizations ;) )?

> - If yes, what could be the next steps?


Next steps would be running these code changes on a number of different system configurations.
Taking a look at Mark's suggestion about turning File Node attributes into properties perhaps?


> 

> I'd say - yes. It is too early for me. I don't feel that SCons code layout is clean enough - there is still magic that could be less magic. Not all tags are present in repository to compare. No pictures in documentation, and no working bug tracker.

> 

> The approximate wishlist:

> [ ] make "import * from SCons" possible - it could be convenient to use SCons like Fabric

So you want to encapsulate SCons' running?
This is a very infrequently requested feature, so I'd put this after most of the other items on your list.


> [ ] create/find docbook template for Sphinx to keep docs old-style, but easier to update (esp. on Windows)


I think there was a user who was working on revamping the doc toolchain for SCons (Who that is is escaping me at the moment).


> [ ] think about Python 3

I floated the question of the importance of this on the users mailing list a few minutes ago.


> [ ] see what components are there, how are they decoupled and interchangeable

> [ ] issues to a new tracker


Not really a priority.  We've discussed this a number of times in the past.
What tracker would you suggest moving to?


> [ ] restore tags


What tags are missing?


> 

> I didn't simply upload this as a pull request, because the changes

> would definitely break existing projects and custom Tools.

> So, either some form of deprecation cycle or a separate branch/repository

> would be needed (default vs optimized).

> 

> Comments anyone?

> 


-Bill 
SCons Project Co-Manager.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://two.pairlist.net/pipermail/scons-dev/attachments/20121204/0af66e64/attachment.htm>