[Scons-dev] Need for speed

[Andrew C. Morrow]

Hi All -

With the SCons 3.0 release and GitHub migration completed (huge
congratulations to everyone involved), I'd like to revisit the discussion
from the summer about SCons performance.

This email is a follow up to
http://two.pairlist.net/pipermail/scons-dev/2017-July/004371.html

While there was a lot of good discussion in that thread, it drifted into
several interesting side conversations. So I would like to narrow the scope
of discussion back to performance. I would also like to propose that the
next SCons release, (possibly named 3.1), focus primarily on SCons
performance, to the exclusion of feature evolution or refactoring.

As was noted in the earlier thread, SCons has many unique, flexible, and
powerful features, and those features come with a performance cost. I think
it unrealistic to expect that SCons will ever match the performance of,
say, Ninja, and I think that is just fine. But I also believe that SCons
could be made much faster than it currently is without compromising its
power or flexibility, and I further think that failing to improve its
performance jeopardizes its future. There are many competitors to SCons
emerging, and the current SCons performance for incremental rebuilds is so
poor as to make a busy developer reach for almost anything else to make the
bleeding stop.

I would like to propose the following as highlight areas for improvement in
a performance focused SCons release:

- Profile and fix low-hanging performance issues in no-op builds: Bill has
found a few using the MongoDB build as a testbed, but we know there are
more. Aggressively profile and address these issues as they are found.

- Tune for performance over resource utilization: There are places where
information is discarded in an effort to prune resource utilization, but
developer and CI machines are typically generously provisioned. Identify
locations where SCons has been optimized for resource utilization over
performance, and reverse.

- Implement variable Substitution caching: In complex builds we have
discovered that a large fraction of time is spent repeatedly evaluating the
same Substs. Caching these results when safe should offer significant
speedups. This has a larger visible impact on incremental builds where the
recreation of command lines to compare against existing built objects can
eat up to 50% of the run time.

- Re-write the SConsign mechanism to use a new opt-in format that avoids
the cost of pickle, and can do incremental updates. This would speed up
both the load of the SConsign data out of the gate, and also dramatically
improve the end-of-build delay encountered in large builds as the entire
SConsign file is re-written, even though only a few files changed.

- Avoid re-computing MD5 signatures when extracting items from the SCons
Cache. The end-to-end caching behavior of SCons is one of its highlights,
but enabling it causes signature re-computation when items are pulled out
of the cache, undermining its performance. The cache should be augmented to
cache not just the objects themselves, but the signatures that they
possess. When objects are extracted from the cache, the required signature
comes with and does not need to be re-computed.

- Fix the issue where the MD5-Timestamp decider cannot be safely combined
with the SCons Cache without spurious rebuilds
<http://scons.tigris.org/issues/show_bug.cgi?id=2980>. This again defeats
the otherwise fantastic behavior of the cache by requiring users to MD5
every file when building with the cache. By fixing this issue, and
implementing the above CacheDir improvements, the SCons cache will both be
intrinsically fast, and not force a degradation in the normal build path.

- Implement a parallel TaskMaster, such that the DAG walk can utilize
multiple cores. While this is clearly the most technically challenging
proposal, the benefits would be huge.

I believe that a SCons 3.1 release that addressed the above issues would
put SCons in a position to again become competitive with the landscape of
build systems.

I’m looking forward to hearing your thoughts, but I would appreciate it if
we could try to keep focused on performance and its prioritization in the
SCons release roadmap, rather than other discussions of potential future
SCons features or refactorings.

Thanks,
Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20170922/3f51ccff/attachment.html>