[Scons-dev] scons-2.5.0 performance issue?

Tue Apr 12 08:44:33 EDT 2016

Thomas,

There are no additional nodes only additional edges, so this becomes a bit
tricky. The problem is that not all edges are equal. A new edge at the
bottom of a tree makes almost no difference, but a single edge near the
root (like your manifest file), can double the size of the tree.

V/R,
William

On Tue, Apr 12, 2016 at 1:26 PM, Thomas Berg <merlin66b at gmail.com> wrote:

> William,
>
> Thank you, I think you are onto something here. I was about to send
> the following observations when your last e-mail arrived:
>
> ---
>
> I did some diffing of the tree output, and found the following:
> - part of the growth of the tree is indeed for Qt moc files. All the
> correct header dependencies are listed here now.
> - a big part of the grows is actually from Visual Studio manifest
> files. We have been using env['WINDOWS_INSERT_MANIFEST'] = 1. Now,
> with scons-2.5.0, the entire dependency-subtree for a dll is
> duplicated for the corresponding manifest file. While this is correct,
> I'm able to cut the tree size in half by turning this off.
>
> The only reason we us this flag is for the Program and SharedLibrary
> builders then return the file in the target list, so it is easy to
> copy the file to our target folder. While the new dependencies are
> correct, it is of limited use for us, since the manifest file will
> always be rebuilt if the dll/executable is rebuilt.
>
> So the growth of the dependency tree looks correct. It seems to me
> that there is a huge potential for optimization of scons here though,
> assuming that my understanding is correct: if scons could recognize
> identical sub-trees in the dependency graph, and not process them
> multiple times, speedup would be substantial. This would also bring us
> far below our original 5 minute build time, because I see a lot of
> duplicated subtrees in there too.
>
> ---
>
> After your mail I realize I didn't understand what --prune does.
> Perhaps the dependency subtrees are not 100% identical between the
> manifest files and the corresponding dlls, and this is the problem,
> rather than identical sub-trees being processed multiple times?
>
> I will look more closely at what you suggested.
>
>
>
> On Tue, Apr 12, 2016 at 2:22 PM, William Blevins <wblevins001 at gmail.com>
> wrote:
> > Thomas,
> >
> > The prune option keeps a group of dependencies from being printed more
> than
> > once. For example, you see the dependency tree for a c++ library L, but
> > executable that files linking against L will only show that it depends
> on L
> > directly and give a line stating that the dependencies of L itself have
> been
> > pruned since they have already been listed previously. Considering this
> > reduction of output, it is more likely that the 4-fold increase is
> > exponential rather than linear.
> >
> > It might be wise to evaluate some of the new subtrees by diffing the two
> > outputs. Chances are what's happening is that you have a N * M dependency
> > relationship which you may or may not be able to untangle.
> >
> > To give you an idea of this change please see the original pull request.
> Pay
> > attention to the test updates to understand what dependencies were
> > previously missing and pay attention to src/engine/SCons/executor.py and
> > src/engine/SCons/Node/__init__.py for information on the functional
> changes.
> >
> > I'm not sure that I can give you a 1-liner to disable the functionality
> > sadly. Take a look first and see if you have any questions about why the
> new
> > dependencies exist.
> >
> > V/R,
> > William
> >
> >
> >
> > On Tue, Apr 12, 2016 at 11:57 AM, Thomas Berg <merlin66b at gmail.com>
> wrote:
> >>
> >> William,
> >>
> >> I ran scons --tree=prune on a small subset of our build, and the text
> >> output goes up from 7.5MB to 35MB for that subset. I guess that means
> >> the dependency graph is four times bigger now.
> >>
> >> Cheers,
> >> Thomas
> >>
> >> On Tue, Apr 12, 2016 at 12:43 PM, Thomas Berg <merlin66b at gmail.com>
> wrote:
> >> > Bill,
> >> >
> >> > Here is the output of --debug=objects:
> >> >
> >> >
> https://dl.dropboxusercontent.com/u/40097540/misc/scons/scons_objects_2_4_1.txt
> >> >
> >> >
> https://dl.dropboxusercontent.com/u/40097540/misc/scons/scons_objects_2_5_0.txt
> >> >
> >> > Both are around 22MB, so here's a 3MB zip containing the two files:
> >> >
> >> >
> https://dl.dropboxusercontent.com/u/40097540/misc/scons/scons_objects_output.zip
> >> >
> >> > The 2.5.0 output is around 2000 lines longer than the 2.4.1 output,
> >> > both are around 430 000 lines.
> >> >
> >> > As a test, is it possible to disable the cross language scanner
> >> > somehow, either by passing something to the environment or patching
> >> > the scons code?
> >> >
> >> > Dirk: thanks for the link. I have tested some but not all the tricks
> >> > used there, and I will definitely look closer at it after we
> >> > (hopefully) get to the bottom of this.
> >> >
> >> > I would at least like to understand what's going on in our case, so
> >> > thanks for all the help. Could it be that just adding a small number
> >> > of dependencies in the wrong place can, due to recursive effects,
> >> > cause such a significant increase in the processing time?
> >> >
> >> > Thanks,
> >> > Thomas
> >> >
> >> >
> >> >
> >> >
> >> > On Mon, Apr 11, 2016 at 6:02 PM, Dirk Bächle <tshortik at gmx.de> wrote:
> >> >> Hi Thomas,
> >> >>
> >> >> On 11.04.2016 16:59, Thomas Berg wrote:
> >> >>>
> >> >>> Bill, below is the output of --debug=count, it is identical with
> >> >>> scons-2.4.1 and scons-2.5.0.
> >> >>>
> >> >>> Since my case was about the no-op build (nothing is built), less
> >> >>> parallelization should not be an issue. ...
> >> >>
> >> >>
> >> >> if you haven't already done so, you might want to try out my
> "fastcpp"
> >> >> extension at:
> >> >>
> >> >>   https://bitbucket.org/dirkbaechle/scons_fastcpp
> >> >>
> >> >> . It may give you some speedup, but please regard its disclaimer.
> >> >>
> >> >> Best regards,
> >> >>
> >> >> Dirk
> >> >>
> >> >>
> >> >> _______________________________________________
> >> >> Scons-dev mailing list
> >> >> Scons-dev at scons.org
> >> >> https://pairlist2.pair.net/mailman/listinfo/scons-dev
> >> _______________________________________________
> >> Scons-dev mailing list
> >> Scons-dev at scons.org
> >> https://pairlist2.pair.net/mailman/listinfo/scons-dev
> >
> >
> >
> > _______________________________________________
> > Scons-dev mailing list
> > Scons-dev at scons.org
> > https://pairlist2.pair.net/mailman/listinfo/scons-dev
> >
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20160412/5969cf8f/attachment-0001.html>