[Scons-dev] scons-2.5.0 performance issue?

Tue Apr 12 08:26:42 EDT 2016

William,

Thank you, I think you are onto something here. I was about to send
the following observations when your last e-mail arrived:

---

I did some diffing of the tree output, and found the following:
- part of the growth of the tree is indeed for Qt moc files. All the
correct header dependencies are listed here now.
- a big part of the grows is actually from Visual Studio manifest
files. We have been using env['WINDOWS_INSERT_MANIFEST'] = 1. Now,
with scons-2.5.0, the entire dependency-subtree for a dll is
duplicated for the corresponding manifest file. While this is correct,
I'm able to cut the tree size in half by turning this off.

The only reason we us this flag is for the Program and SharedLibrary
builders then return the file in the target list, so it is easy to
copy the file to our target folder. While the new dependencies are
correct, it is of limited use for us, since the manifest file will
always be rebuilt if the dll/executable is rebuilt.

So the growth of the dependency tree looks correct. It seems to me
that there is a huge potential for optimization of scons here though,
assuming that my understanding is correct: if scons could recognize
identical sub-trees in the dependency graph, and not process them
multiple times, speedup would be substantial. This would also bring us
far below our original 5 minute build time, because I see a lot of
duplicated subtrees in there too.

---

After your mail I realize I didn't understand what --prune does.
Perhaps the dependency subtrees are not 100% identical between the
manifest files and the corresponding dlls, and this is the problem,
rather than identical sub-trees being processed multiple times?

I will look more closely at what you suggested.

On Tue, Apr 12, 2016 at 2:22 PM, William Blevins <wblevins001 at gmail.com> wrote:
> Thomas,
>
> The prune option keeps a group of dependencies from being printed more than
> once. For example, you see the dependency tree for a c++ library L, but
> executable that files linking against L will only show that it depends on L
> directly and give a line stating that the dependencies of L itself have been
> pruned since they have already been listed previously. Considering this
> reduction of output, it is more likely that the 4-fold increase is
> exponential rather than linear.
>
> It might be wise to evaluate some of the new subtrees by diffing the two
> outputs. Chances are what's happening is that you have a N * M dependency
> relationship which you may or may not be able to untangle.
>
> To give you an idea of this change please see the original pull request. Pay
> attention to the test updates to understand what dependencies were
> previously missing and pay attention to src/engine/SCons/executor.py and
> src/engine/SCons/Node/__init__.py for information on the functional changes.
>
> I'm not sure that I can give you a 1-liner to disable the functionality
> sadly. Take a look first and see if you have any questions about why the new
> dependencies exist.
>
> V/R,
> William
>
>
>
> On Tue, Apr 12, 2016 at 11:57 AM, Thomas Berg <merlin66b at gmail.com> wrote:
>>
>> William,
>>
>> I ran scons --tree=prune on a small subset of our build, and the text
>> output goes up from 7.5MB to 35MB for that subset. I guess that means
>> the dependency graph is four times bigger now.
>>
>> Cheers,
>> Thomas
>>
>> On Tue, Apr 12, 2016 at 12:43 PM, Thomas Berg <merlin66b at gmail.com> wrote:
>> > Bill,
>> >
>> > Here is the output of --debug=objects:
>> >
>> > https://dl.dropboxusercontent.com/u/40097540/misc/scons/scons_objects_2_4_1.txt
>> >
>> > https://dl.dropboxusercontent.com/u/40097540/misc/scons/scons_objects_2_5_0.txt
>> >
>> > Both are around 22MB, so here's a 3MB zip containing the two files:
>> >
>> > https://dl.dropboxusercontent.com/u/40097540/misc/scons/scons_objects_output.zip
>> >
>> > The 2.5.0 output is around 2000 lines longer than the 2.4.1 output,
>> > both are around 430 000 lines.
>> >
>> > As a test, is it possible to disable the cross language scanner
>> > somehow, either by passing something to the environment or patching
>> > the scons code?
>> >
>> > Dirk: thanks for the link. I have tested some but not all the tricks
>> > used there, and I will definitely look closer at it after we
>> > (hopefully) get to the bottom of this.
>> >
>> > I would at least like to understand what's going on in our case, so
>> > thanks for all the help. Could it be that just adding a small number
>> > of dependencies in the wrong place can, due to recursive effects,
>> > cause such a significant increase in the processing time?
>> >
>> > Thanks,
>> > Thomas
>> >
>> >
>> >
>> >
>> > On Mon, Apr 11, 2016 at 6:02 PM, Dirk Bächle <tshortik at gmx.de> wrote:
>> >> Hi Thomas,
>> >>
>> >> On 11.04.2016 16:59, Thomas Berg wrote:
>> >>>
>> >>> Bill, below is the output of --debug=count, it is identical with
>> >>> scons-2.4.1 and scons-2.5.0.
>> >>>
>> >>> Since my case was about the no-op build (nothing is built), less
>> >>> parallelization should not be an issue. ...
>> >>
>> >>
>> >> if you haven't already done so, you might want to try out my "fastcpp"
>> >> extension at:
>> >>
>> >>   https://bitbucket.org/dirkbaechle/scons_fastcpp
>> >>
>> >> . It may give you some speedup, but please regard its disclaimer.
>> >>
>> >> Best regards,
>> >>
>> >> Dirk
>> >>
>> >>
>> >> _______________________________________________
>> >> Scons-dev mailing list
>> >> Scons-dev at scons.org
>> >> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>> _______________________________________________
>> Scons-dev mailing list
>> Scons-dev at scons.org
>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>
>
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>