[Scons-dev] SCons-like build system

[Kenny, Jason L]

>>Aqualid uses __slots__ since initial version, but it used a lot of memory on large projects in the past. Fortunately, I resolved this issue with quite simple fix. I added a function Node.shrink() (see https://github.com/aqualid/aqualid/blob/master/aql/nodes/aql_node.py)
It simply resets all already unneeded attributes to None.
>>

I agree with what you are saying here. There are some things to consider. In Parts we looked at messing with the tree SCons uses by removing nodes. This helps in that as you build, you reduce memory, but you still have the issue with you do a fresh build or are building everything as you will need all the nodes loaded. This also messes up a nice feature in scons ( ie the –interactive ) as the tree is messed up, ie gone as we removed nodes. I believe an idea that I like a lot to help improve this, is to create a node tree of builders ( based on Gregs TNG that he never finished). As we find that an input nodes depend on other items, we make the link in the builder tree. This makes a much smaller tree than the current SCons node tree ( in which we build more off of a given node and see if we called the builder yet for it). This I think would save some memory and speed up builds. This information could also be tweak to store in a file cache or something to allow stuff like interactive build quietly reload and go when it is needed so overall memory can be reduced to what is needed at a given moment.

Jason

From: Scons-dev [mailto:scons-dev-bounces at scons.org] On Behalf Of Constantine
Sent: Tuesday, January 13, 2015 4:28 PM
To: dl9obn at darc.de; SCons developer list
Subject: Re: [Scons-dev] SCons-like build system


Thank you Dirk for the very interesting response.
I will review closely all the links.

I definitely see improvements in SCons performance during the last years.

I think that moving to __slots__ is good way, it will improve memory consumption, but for the large projects it still could consume a lot of memory.
Aqualid uses __slots__ since initial version, but it used a lot of memory on large projects in the past. Fortunately, I resolved this issue with quite simple fix. I added a function Node.shrink() (see https://github.com/aqualid/aqualid/blob/master/aql/nodes/aql_node.py)
It simply resets all already unneeded attributes to None.

>> We're also currently redesigning the ways we handle Tools and Toolchains, trying to make things more flexible for the user. Maybe you want to join this discussion?
Yes. As example I can explain how it works in Aqualid. I don't have a link to documentation yet. I'm still writing user guide.

>> We could try to provide a "stack" of build systems, where users can migrate from "aqualid" -> "scons" -> "parts" whenever their requirements grow.
I fully support this idea. I remember there was a discussion of similar ideas about seven years ago. Email thread "SCons Future Directions and Thought"

Thanks!
Best regards,
Konstantin,

On Tue, Jan 13, 2015 at 9:55 PM, Dirk Bächle <tshortik at gmx.de<mailto:tshortik at gmx.de>> wrote:
Hi Constantine,

thank you very much for the note about your project "Aqualid". I already had a short look at your benchmark, and plan to also check the actual sources soon. Allow me to give you a few pointers as well:

In the recent past we have done some extensive performance analysis and profilings for SCons, you can find the details and results at http://www.scons.org/wiki/WhySconsIsNotSlow . I also gave a talk about this ( http://www.pycon.fr/2014/schedule/presentation/25/ , slides at https://bitbucket.org/dirkbaechle/scons_talks ), unfortunately the video hasn't been edited/published yet. As a follow-up of these investigations I have prepared a patch that switches SCons (especially the Node class) to using slots, in order to reduce the overall memory consumption. If you're interested in trying it out with your benchmark, you can get a copy with

  hg clone http://bitbucket.org/dirkbaechle/scons_experimental -r switch_node_to_slots

. See also https://pairlist2.pair.net/pipermail/scons-dev/2014-December/002107.html for more infos about this.

Once these changes are in, we'll integrate the stubprocess.py wrapper for speeding up clean builds...as described on the WhySconsIsNotSlow page.

We're also currently redesigning the ways we handle Tools and Toolchains, trying to make things more flexible for the user. Maybe you want to join this discussion?

Given the upcoming improvements in our codebase it might also make sense that you, Jason Kenny from the Parts project (parts.tigris.org<http://parts.tigris.org>) and our project managers Gary and Bill stick their heads together. I have a vision of using the SCons framework as building block for: the classic "scons" script, "parts" and "aqualid" as well. We could try to provide a "stack" of build systems, where users can migrate from "aqualid" -> "scons" -> "parts" whenever their requirements grow.
Just as an idea...and we already have some plans to move stuff from parts into SCons. If we want to sort of combine our forces, now's the time to talk about it. ;)

Best regards,

Dirk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20150113/a3b740e5/attachment-0001.html>