[Scons-dev] scons-2.5.0 performance issue?

[William Blevins]

Thomas,

The prune option keeps a group of dependencies from being printed more than
once. For example, you see the dependency tree for a c++ library L, but
executable that files linking against L will only show that it depends on L
directly and give a line stating that the dependencies of L itself have
been pruned since they have already been listed previously. Considering
this reduction of output, it is more likely that the 4-fold increase is
exponential rather than linear.

It might be wise to evaluate some of the new subtrees by diffing the two
outputs. Chances are what's happening is that you have a N * M dependency
relationship which you may or may not be able to untangle.

To give you an idea of this change please see the original pull request
<https://bitbucket.org/scons/scons/pull-requests/244/issue-2264-cross-language-scanner-support/diff>.
Pay attention to the test updates to understand what dependencies were
previously missing and pay attention to src/engine/SCons/executor.py and
src/engine/SCons/Node/__init__.py for information on the functional changes.

I'm not sure that I can give you a 1-liner to disable the functionality
sadly. Take a look first and see if you have any questions about why the
new dependencies exist.

V/R,
William



On Tue, Apr 12, 2016 at 11:57 AM, Thomas Berg <merlin66b at gmail.com> wrote:

> William,
>
> I ran scons --tree=prune on a small subset of our build, and the text
> output goes up from 7.5MB to 35MB for that subset. I guess that means
> the dependency graph is four times bigger now.
>
> Cheers,
> Thomas
>
> On Tue, Apr 12, 2016 at 12:43 PM, Thomas Berg <merlin66b at gmail.com> wrote:
> > Bill,
> >
> > Here is the output of --debug=objects:
> >
> https://dl.dropboxusercontent.com/u/40097540/misc/scons/scons_objects_2_4_1.txt
> >
> https://dl.dropboxusercontent.com/u/40097540/misc/scons/scons_objects_2_5_0.txt
> >
> > Both are around 22MB, so here's a 3MB zip containing the two files:
> >
> https://dl.dropboxusercontent.com/u/40097540/misc/scons/scons_objects_output.zip
> >
> > The 2.5.0 output is around 2000 lines longer than the 2.4.1 output,
> > both are around 430 000 lines.
> >
> > As a test, is it possible to disable the cross language scanner
> > somehow, either by passing something to the environment or patching
> > the scons code?
> >
> > Dirk: thanks for the link. I have tested some but not all the tricks
> > used there, and I will definitely look closer at it after we
> > (hopefully) get to the bottom of this.
> >
> > I would at least like to understand what's going on in our case, so
> > thanks for all the help. Could it be that just adding a small number
> > of dependencies in the wrong place can, due to recursive effects,
> > cause such a significant increase in the processing time?
> >
> > Thanks,
> > Thomas
> >
> >
> >
> >
> > On Mon, Apr 11, 2016 at 6:02 PM, Dirk Bächle <tshortik at gmx.de> wrote:
> >> Hi Thomas,
> >>
> >> On 11.04.2016 16:59, Thomas Berg wrote:
> >>>
> >>> Bill, below is the output of --debug=count, it is identical with
> >>> scons-2.4.1 and scons-2.5.0.
> >>>
> >>> Since my case was about the no-op build (nothing is built), less
> >>> parallelization should not be an issue. ...
> >>
> >>
> >> if you haven't already done so, you might want to try out my "fastcpp"
> >> extension at:
> >>
> >>   https://bitbucket.org/dirkbaechle/scons_fastcpp
> >>
> >> . It may give you some speedup, but please regard its disclaimer.
> >>
> >> Best regards,
> >>
> >> Dirk
> >>
> >>
> >> _______________________________________________
> >> Scons-dev mailing list
> >> Scons-dev at scons.org
> >> https://pairlist2.pair.net/mailman/listinfo/scons-dev
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20160412/fa419acc/attachment-0001.html>