[Scons-dev] SCons-like build system

[Kenny, Jason L]

In general yes. The idea is that the Parts makes a smaller dag. It does not do exactly as you state, it create one DAG based on the Parts that are not out of date, not lots of smaller DAGs. Parts check a number of things to see if a Part might be out of date, and if it could be, we load it, else we don’t read the file, and instead load cached information if and only if there is a component that depends on it.

In general Parts uses the notion of a component ( or a file that defines something) to make a “component” Dag. It checks to see if anything modified that state ( parts file changed, Sconstruct changed, configuration/toolchains was modified, source  or output files for component changed etc… which are mostly a Scons timestamp-MD5 check. If any component is out of date or might be out of sync we mark it as such and load any dependent component ( no need to check them as we know there is “some” modification). This loads the items into SCons to be processed. The resulting time to load and process is *much* smaller on average.

Currently Parts is looking at removing our “mapper” logic, which is an abuse of the subst() system in SCons to share data between components, and moving to what is called “continuous loader” which will load the Components in dependency order (making data sharing simpler). This should also make the current logic for checking state simpler and faster.

From there the idea to look at is to store the actions defined in the Component and replay them as part of the rebuild, as this can be done once we know we have a sub tree that has nothing new added to it. While those items build we continue to load more data ( as the build would be waiting for build actions to complete). Given the common case for a incremental build this should work out. As any relationships defined explicitly or implicitly will not change. The most common change will be in the source files, which will be seem as part of a change that effect loading a component build file, or by the component build file being changed. At any rate the dependent information gets loaded as needed as processed to see what is safe or not safe to say. This idea need more flushing out as there are details that could mess up a correct build, or it might show to not speed anything up enough.

Jason

From: Scons-dev [mailto:scons-dev-bounces at scons.org] On Behalf Of Constantine
Sent: Sunday, January 25, 2015 3:24 AM
To: SCons developer list; Tom Tanner
Subject: Re: [Scons-dev] SCons-like build system

Do I understand correctly?
That the main idea is that using the Parts a big project is split into many small parts.
And then SCons builds these parts more-or-less independently.
I.e. SCons processes many small DAGs instead of one big DAG.

Thank you.
Best regards,
Constantine.
On 01/23/15 18:03, Kenny, Jason L wrote:
I understand this problem. This is one main thing Parts tries to address. It uses information about the different component to figure out what not to load. This requires the build to be laid out in some sort of component based like build, vs a recursive make, which may not be a big deal, depending on how your project is laid out. The result for example of a massive build I have here is a “do nothing” build takes ~7-20 second vs the normal ~15-20 minutes it would normally take. Incremental build also are reduced to minutes ( depending on what changed of course)

I honestly believe this can be better yet.. but we requires more work.  The main issue with spawn issue that we pushed, was found when a build for this large product moved from rhel4 to rhel5. The time increased from 2 hours to 4+. It was the spawning issue. Something changed that really made this worse. Once we had this fixed rhel5 build went to 2 hours and the old rhel4 based build went to ~1.5 hours.

Jason

From: Scons-dev [mailto:scons-dev-bounces at scons.org] On Behalf Of Tom Tanner (BLOOMBERG/ LONDON)
Sent: Thursday, January 22, 2015 2:49 AM
To: scons-dev at scons.org<mailto:scons-dev at scons.org>
Subject: Re: [Scons-dev] SCons-like build system

Having been poking around in this, I see that 'posix_spawn' cleans to be availble on both aix and solaris (at least they have man pages). Which is a relief for me. However, my current grief is that a 'do nothing' build is still taking 20 minutes to traverse the tree. *part* of this is because I check the md5sum on each file to make sure someone hasn't gone and edit/patched/hacked a generated target (which happens more often than is desirable), but that isn't the bulk of the time taken.

I suspect I could leverage the fact that we use git to get the sha1 of certain repositories that are known not to be changed by us and if we find a file therein, to use the combined sha1 of the repos might improve that at the cost of potentially spurious rebuilds if one of the repos changed but the others didn't as i wouldn't be reading all the files.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20150125/b5d0b03f/attachment-0001.html>