[Scons-dev] SCons-like build system

[Constantine]

*>> but you still have the issue with you do a fresh build or are building
everything as you will need all the nodes loaded.*

Aqualid checks and builds only leaf nodes. Signatures are not calculated and
dependencies are not checked for non-leaf nodes. When a leaf node is
built Aqualid
saves node's state into internal DB. Then we clear all node's attributes
except 'targets'. 'targets' are kept to be used as sources for other nodes.
When all nodes which use this node are built we may completely forget about
this node.


On Wed, Jan 14, 2015 at 1:58 AM, Kenny, Jason L <jason.l.kenny at intel.com>
wrote:

>  >>Aqualid uses __slots__ since initial version, but it used a lot of
> memory on large projects in the past. Fortunately, I resolved this issue
> with quite simple fix. I added a function Node.shrink() (see
> https://github.com/aqualid/aqualid/blob/master/aql/nodes/aql_node.py)
>
> It simply resets all already unneeded attributes to None.
>
> >>
>
>
>
> I agree with what you are saying here. There are some things to consider.
> In Parts we looked at messing with the tree SCons uses by removing nodes.
> This helps in that as you build, you reduce memory, but you still have the
> issue with you do a fresh build or are building everything as you will need
> all the nodes loaded. This also messes up a nice feature in scons ( ie the
> –interactive ) as the tree is messed up, ie gone as we removed nodes. I
> believe an idea that I like a lot to help improve this, is to create a node
> tree of builders ( based on Gregs TNG that he never finished). As we find
> that an input nodes depend on other items, we make the link in the builder
> tree. This makes a much smaller tree than the current SCons node tree ( in
> which we build more off of a given node and see if we called the builder
> yet for it). This I think would save some memory and speed up builds. This
> information could also be tweak to store in a file cache or something to
> allow stuff like interactive build quietly reload and go when it is needed
> so overall memory can be reduced to what is needed at a given moment.
>
>
>
> Jason
>
>
>
> *From:* Scons-dev [mailto:scons-dev-bounces at scons.org] *On Behalf Of *
> Constantine
> *Sent:* Tuesday, January 13, 2015 4:28 PM
> *To:* dl9obn at darc.de; SCons developer list
> *Subject:* Re: [Scons-dev] SCons-like build system
>
>
>
>
>
> Thank you Dirk for the very interesting response.
>
> I will review closely all the links.
>
>
>
> I definitely see improvements in SCons performance during the last years.
>
>
>
> I think that moving to __slots__ is good way, it will improve memory
> consumption, but for the large projects it still could consume a lot of
> memory.
>
> Aqualid uses __slots__ since initial version, but it used a lot of memory
> on large projects in the past. Fortunately, I resolved this issue with
> quite simple fix. I added a function Node.shrink() (see
> https://github.com/aqualid/aqualid/blob/master/aql/nodes/aql_node.py)
>
> It simply resets all already unneeded attributes to None.
>
>
> *>> We're also currently redesigning the ways we handle Tools and
> Toolchains, trying to make things more flexible for the user. Maybe you
> want to join this discussion?*
>
> Yes. As example I can explain how it works in Aqualid. I don't have a link
> to documentation yet. I'm still writing user guide.
>
>
> *>> We could try to provide a "stack" of build systems, where users can
> migrate from "aqualid" -> "scons" -> "parts" whenever their requirements
> grow.*
>
> I fully support this idea. I remember there was a discussion of similar
> ideas about seven years ago. Email thread "SCons Future Directions and
> Thought"
>
>
>
> Thanks!
>
> Best regards,
>
> Konstantin,
>
>
>
> On Tue, Jan 13, 2015 at 9:55 PM, Dirk Bächle <tshortik at gmx.de> wrote:
>
> Hi Constantine,
>
> thank you very much for the note about your project "Aqualid". I already
> had a short look at your benchmark, and plan to also check the actual
> sources soon. Allow me to give you a few pointers as well:
>
> In the recent past we have done some extensive performance analysis and
> profilings for SCons, you can find the details and results at
> http://www.scons.org/wiki/WhySconsIsNotSlow . I also gave a talk about
> this ( http://www.pycon.fr/2014/schedule/presentation/25/ , slides at
> https://bitbucket.org/dirkbaechle/scons_talks ), unfortunately the video
> hasn't been edited/published yet. As a follow-up of these investigations I
> have prepared a patch that switches SCons (especially the Node class) to
> using slots, in order to reduce the overall memory consumption. If you're
> interested in trying it out with your benchmark, you can get a copy with
>
>   hg clone http://bitbucket.org/dirkbaechle/scons_experimental -r
> switch_node_to_slots
>
> . See also
> https://pairlist2.pair.net/pipermail/scons-dev/2014-December/002107.html
> for more infos about this.
>
> Once these changes are in, we'll integrate the stubprocess.py wrapper for
> speeding up clean builds...as described on the WhySconsIsNotSlow page.
>
> We're also currently redesigning the ways we handle Tools and Toolchains,
> trying to make things more flexible for the user. Maybe you want to join
> this discussion?
>
> Given the upcoming improvements in our codebase it might also make sense
> that you, Jason Kenny from the Parts project (parts.tigris.org) and our
> project managers Gary and Bill stick their heads together. I have a vision
> of using the SCons framework as building block for: the classic "scons"
> script, "parts" and "aqualid" as well. We could try to provide a "stack" of
> build systems, where users can migrate from "aqualid" -> "scons" -> "parts"
> whenever their requirements grow.
> Just as an idea...and we already have some plans to move stuff from parts
> into SCons. If we want to sort of combine our forces, now's the time to
> talk about it. ;)
>
> Best regards,
>
> Dirk
>
>
>
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20150114/29c7a754/attachment-0001.html>