[Scons-dev] Excessive exec calls at startup

[Johan Holmberg]

Hi!

While building a project with SCons with 144 different Environment()
objects (one for each of my libraries), I observed that a lot of external
processes were created even before SCons did build anything. I use MacOS,
so I turned to DTrace to snoop on new processes created:

  $ sudo newproc.d > newproc.log

Then I ran "scons non_existing_file" in another terminal, to see what SCons
did at startup. According to "newproc.d" there are 3486 exec calls (new
processes). By looking in detail in the log from "newproc.d", I see that
"gcc --version" is called 290 times, and "g++ --version" another 290 times
(each *two* times per Environment). And each of these 580 processes seem to
use 6 exec calls internally, like:

    gcc --version
    /Applications/Xcode.app/Contents/Developer/usr/bin/gcc --version
    sh -c /usr/bin/xcode-select -print-path 2> /dev/null
    /usr/bin/xcode-select -print-path
    /usr/bin/xcrun clang --version

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang
--version

I modified the code in "src/engine/SCons/Tool/gcc.py" to cache calls to
"detect_version" where the same gcc/g++ is asked about its version several
times. With this change the number of exec calls went from 3486 to 18. And
the startup time went down from 18 seconds to 8 seconds. (MacOS Yosemite on
an iMac from 2011).

I'm not very familiar with the SCons source code, so my patch is probably
not entirely correct. But I hope something similar could be added to SCons
to avoid wasting time at startup.

Regards,
/Johan Holmberg

--- a/src/engine/SCons/Tool/gcc.py Fri Nov 20 13:54:09 2015 -0800
+++ b/src/engine/SCons/Tool/gcc.py Sat Nov 14 17:09:59 2015 +0100
@@ -63,11 +63,22 @@
     # is executable, and is a GNU compiler (or accepts '--version' at
least)
     return detect_version(env, env.Detect(env.get('CC', compilers)))

+_detected_version_cache = dict()
+
 def detect_version(env, cc):
     """Return the version of the GNU compiler, or None if it is not a GNU
compiler."""
     cc = env.subst(cc)
     if not cc:
         return None
+    cc_path = env.WhereIs(cc)
+    try:
+        return _detected_version_cache[cc_path]
+    except KeyError:
+        version = _detect_version_1(env, cc)
+        _detected_version_cache[cc_path] = version
+        return version
+
+def _detect_version_1(env, cc):
     version = None
     #pipe = SCons.Action._subproc(env, SCons.Util.CLVar(cc) +
['-dumpversion'],
     pipe = SCons.Action._subproc(env, SCons.Util.CLVar(cc) + ['--version'],
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20151123/f5788158/attachment.html>