[Scons-dev] SCons tools refactoring progress

[Dirk Baechle]

Hi there, 

just wanted to say that I second Gary in all his points...especially the last one. ;)

Dirk

Am 21. Oktober 2015 02:59:02 MESZ, schrieb Gary Oberbrunner <garyo at oberbrunner.com>:
>Comments inline.
>
>On Tue, Oct 20, 2015 at 12:52 AM, anatoly techtonik
><techtonik at gmail.com>
>wrote:
>
>> On Tue, Oct 20, 2015 at 3:57 AM, Gary Oberbrunner
><garyo at oberbrunner.com>
>> wrote:
>>>
>>> On Sun, Oct 18, 2015 at 7:52 AM, anatoly techtonik
><techtonik at gmail.com>
>>> wrote:
>>>
>>>>
>>>> I see the implementation, but I don't see any use cases. I know it
>>>> sounds too formal, but I can't validate the assumptions we had
>towards the
>>>> new toolchain without a formal list. Do you have some notes or
>maybe BDD
>>>> tests for that?
>>>>
>>>
>>> A very good question. I don't have enough design notes in there;
>I'll
>>> write up some of my motivations and design goals here and add them
>to the
>>> README.rst.
>>>
>>> The basic idea is that the current concept of a Tool is too low
>level;
>>> the primary motivating use case is that users (SConscript authors)
>should
>>> be able to select groups of related tools, _as_ a group, with
>fallbacks. A
>>> Toolchain is that abstraction.
>>>
>>
>> The use case is already good to be recorded. But there are two cases:
>> 1. Select a group of related tools
>> 2. Fallback
>> Do we have a concrete groups of related tools already?
>>
>
>Yes, I described a few. (Compiler and linker, maybe assembler is the
>obvious one.)
>
>
>> The fallback story needs to be expanded. Are there fallback choices
>inside
>> of one group (with priority or other preference strategy) or the
>fallback
>> means "fallback to another group"? Maybe I am too detailed, but also
>- what
>> are the cases when fallbacks occur?
>>
>
>This is all implemented. Check the code and test cases
>(Toolchain-test.py).
>I think it's pretty solid but please review. Basically it's recursive
>AND
>and OR trees with optional or required elements.
>
>
>> It it not necessary to design everything now upfront - the primary
>goal of
>> my questions is to find *real world* stories (not even use cases) for
>which
>> new mechanism is needed. I am afraid to design a system that will be
>too
>> perfect for implementation in a free time.
>>
>
>Well, at least for the toolchain, I think it's mostly there
>implementation-wise (check it out). The interesting stuff is improving
>the
>Tool ideas (there's a sort of registry so you can give concrete tools
>names
>and look them up that way), adding Finders, and all the other stuff
>that
>we've been discussing. But if you have some real-world stories you'd
>like
>to see covered, this is a good time to get them out there.
>
>All the rest is just there to make that idea work well. The secondary
>goal,
>>> in service of the main one, is that Tools need to obey their
>abstraction,
>>> for instance always calling exists().
>>>
>>
>> What is the story behind this? Because I know the opposite story -
>> exists() for default tools is called even if I don't build anything
>with
>> those tools - this delays the build start and produces messages about
>> missing compiler to the screen.
>>
>
>exists() is how a tool knows whether its binaries (or whatever it needs
>to
>run) exist or not, so toolchains can't work without it. As for default
>tools, the idea here is that SConscript writers will (finally!) be able
>to
>specify exactly which tools they want. I hope to do away with the
>current
>default tool initialization system, though some of that still needs to
>be
>thought out. The current design is much "lazier" but still needs work.
>
>The new system also creates a distinction between an abstract tool,
>such as
>>> intelc, and a concrete instance of it, such as intelc v12 x86. This
>is
>>> needed so the user can create chains of either specific or general
>tools.
>>>
>>
>> I understand where this might be useful, but still - is there a real
>world
>> story where this was needed?
>>
>
>Absolutely - in my day job we specify very carefully what tool chain is
>used to build any given product. If the machine doesn't have the right
>version of the Intel compiler the build should fail. For open-source
>projects, on the other hand, they should try to build on as many
>configurations as possible so they want to keep things general -- use
>any
>Intel compiler, or any gcc, or any cc (for instance).
>
>
>> One restriction I'm imposing in the new system is that Tools have to
>be
>>> configurable outside of any Environment; I don't like the current
>system
>>> where tool-configuration variables ("tool args" basically) are mixed
>into
>>> the Environment. This poses some challenges for a fully
>generalizable
>>> system but I think I have a decent handle on that. The current Intel
>>> compiler has an initial attempt at such tool args.
>>>
>>
>> How to handle "tools args" is a question for a separate thread. We
>will
>> need to have a standard set for every *type* of tool and specific for
>every
>> tool supported. For the maintenance that means tables, and perhaps
>tests
>> for those args. And neither unit tests, nor BB wiki are good things
>for
>> human readable tests and human writable tables.
>>
>
>Maybe. I think it is _vital_ at this stage to separate mechanism from
>policy. We need a mechanism that can support many policies;
>table-driven
>args or standard arg sets are fine policies, and we'll probably have
>one
>for the SCons built-in tools, and users will probably develop their
>own,
>which is fine.
>
>
>> Again, for the design it needs concrete stories. Like "I want to use
>> intelc v12 x86 which uses its own args on Windows". My goal with all
>these
>> stories is to dig up the cases where new more complicated system just
>don't
>> worth it and simple Environment variable hack is good.
>>
>
>I sure hope this isn't complicated! It's just a few classes. Simplicity
>is
>an over-arching goal. "Simple Environment hacks" is how we got to the
>complexity we have now where even documenting what args a tool requires
>is
>error-prone. (I've added some self-documentation to Tools -- see the
>code.)
>
>>
>>
>>> Some use cases:
>>>  * a simple SConstruct should automatically select the "best"
>>> compiler/linker
>>>
>>
>> What is the best in real world projects? For example, pre-configured
>chain
>> for specific combination of OS, environment, flags, source file
>types? How
>> many criteria we may (which arguments should we) pass to
>> get_the_best_tool() function?
>>
>
>Again, separate mechanism (what can we support) from policy (what is
>best
>for a particular project). I'm concerned with building the mechanism.
>
>
>> Also, I see that it will be good to make it independent of SCons
>codebase
>> for now.
>>
>
>I don't see how that is possible. It will need to replace the current
>Tool
>logic.
>
>>
>>
>>>  * a non-C-related SConstruct (e.g. doc prep, asset management,
>>> scientific data handling) shouldn't have to care about compilers,
>linkers
>>> or other unrelated tools
>>>
>>
>> Good. Now we need to expand what this means and record into separate
>> story. "When I invoke Sphinx and SCSS tool, SCons initializes
>compilers,
>> linkers and other related tools and complains about missing Visual
>Studio
>> compliler. It also takes a long time."
>>
>
>Well, that's not a story about the requirements of the system, it's a
>story
>about how the current system is bad. Please come up with some user
>stories
>that describe desired behavior: "As a user, I want to be able to
>specify
>that the only tools I need are Sphinx and SCSS. Other tools should not
>be
>initialized nor slow down the build." I've captured that in the README;
>please send more.
>
>>
>>
>>>  * a SConstruct author should be able to specify toolchains and
>>> alternatives, and handle failures gracefully
>>>
>>
>> What is the failure? Only "tool does not exist"? Where it is needed
>right
>> now?
>>
>
>Yes, "tool does not exist" is the only one I foresee now -- I have a
>provision for an additional error message the tool can pass back ("not
>found in $PATH" or whatever).
>
>>
>>
>>>  * it should be possible to write a SConstruct where the tools and
>>> toolchains can be manipulated by cmd line args if desired
>>>  * it should be possible to specify desired tools generally
>>> ("gcc/gnulink") or very specifically ("gcc 4.3, x86 cross compiler,
>with
>>> gnulink 4.3 - and TeXLive")
>>>
>>
>> I don't see how that's better than just passing the absolute path for
>> specific binary. Making specific restriction for "gcc" and "4.3" in
>SCons
>> defined classes may make it harder to build stuff with compatible
>> toolchains that don't have SCons definitions.
>>
>
>Think about Mac: there are three different places gcc can be found
>(macports, built-in, and another one I can't remember right now). A
>SConscript for an open-source project shouldn't have to have any paths
>hardcoded.
>
>
>> There's a bunch of tests in that dir; they're mostly unit-test level
>at
>>> this point. There are also some examples of the new subsystem in use
>(see
>>> the README) which are closer to actual use cases.
>>>
>>
>> There is a risk for such use cases to become skewed towards easier
>> implementation, and in the end it may be not solving the original
>problem
>> or doing it in a very strange and complicated way. That's why in
>formalism
>> I prefer real world stories.
>>
>
>OK, please send some.
>
>>
>>
>>> My current task is to design a good way for Tools to find their
>>> executables and configure themselves. The current ad-hoc method is
>not
>>> consistent and doesn't encourage reuse. Jason has some ideas in
>Parts (see
>>> its concept of Finders); I don't intend to reuse that directly but
>at least
>>> take some inspiration from it.
>>>
>>
>> I haven't seen the code, but why? What is the problem with Jason
>approach?
>> And difference in finding executables and configuration?
>>
>
>It's too complicated to get into here. The Parts code is not quite
>complete, requires some heavy infrastructure, is a bit policy-oriented
>rather than mechanism, and other things.
>
>>
>>
>>> My current design I'm working on is something like this: ... a good
>>> architecture might be to have a list of finders, to be tried in
>order; each
>>> one can be any type (env, reg, path, fixed path, function,
>whatever); build
>>> a list of all the results, and then have a finder-picker that picks
>the
>>> best from that list (the simplest finder-picker might be return
>paths[0]).
>>> But where that list comes from and how it's manipulated is still
>TBD.
>>>
>>
>> So, you need a bunch of finder functions - findinenv, findinreg,
>> findinfixedpath, ... which can be quickly combined into a list and is
>tool
>> is needed, the first wins. You also need to cache the result - either
>for
>> every check, or for the tool in general. Then you will need to clean
>the
>> cache for the check one day.
>>
>
>Yes, that's basically the idea. See the code. If you have a user story
>for
>why cleaning the cache could be needed, please send it along.
>
>>
>> For example, my story for Wesnoth automated build, is that if tool
>does
>> not exist, I download the tool and then redo the check. So it
>resembles the
>> way SCons deals with sources. So, should SCons threat Tools as such
>nodes?
>> Should these nodes be isolated from the main FS and Build graph?
>>
>
>This may be beyond the capabilities of the system I propose; I think
>adding
>that would significantly complicate it. If you need to download
>something
>(to even see if it exists()) or have a dependency-graph to create the
>tools, I think a different system would be needed.
>
>
>>
>> To make this actionable, I propose to create a simple "finders"
>repository
>> that will concentrate on finding tools and will contain:
>>
>> 1. lookup helpers
>> 2. descriptions of tool locations and discovery for
>>    1. different tools
>>    2. different platforms
>> 3. attempt to design file format for expressing this info
>declaratively
>> 4. description of stories where declarative spec doesn't help or will
>be
>> too complex
>>
>
>OK, feel free to start something. I can't promise to use it but at
>least
>some of what you're describing sounds useful. I'm currently hung up on
>how
>to allow configurability of those "lookup helpers" (aka Finders: path,
>registry, function-call) from various levels (within a tool, by
>toolchain,
>by SConstruct, or ultimately by cmd line arg) in an elegant and
>extensible
>way, but of course writing the basic code is the first step.
>
>Even if we fail to patch the SCons to accommodate all use cases, the
>repo
>> will be useful for other build tools, who may bring better ideas in
>chain
>> management.
>>
>>
>>> Additionally I'd like to see how far it makes sense to go in having
>>> standard args to configure Tools; one way is just to leave args up
>to each
>>> Tool (but maybe have the Finders have some high-level control); the
>other
>>> is to define many standard ones (like ARCH, search in PATH or some
>custom
>>> path, and so on) - the latter will make it easier someday to pass
>these all
>>> the way down from the cmd line. I want Tools (and their Toolchains)
>to
>>> stand alone easily, but also have some way to override their
>decisions at a
>>> high level (globally, from cmd line, etc.)
>>>
>>> And yes, with this system it is a goal that there would be no more
>>> "missing Visual Studio" errors on Windows. :-)
>>>
>>
>> In that case we need to start with use case from a different end -
>after
>> reading SConstruct detect if a Visual Studio is needed. We would
>really
>> need to draw a diagrams for that. =)
>>
>> I explicitly reject that goal. Automatically analyzing a SConstruct
>to see
>what Tools it needs is much too complicated. My idea is users
>(SConstruct-writers) will either use pre-set tool chains, or will set
>up
>their own at the top of the SConstruct (before creating any
>Environment).
> "Explicit is better than implicit" as the saying goes.
>
>-- 
>Gary
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Scons-dev mailing list
>Scons-dev at scons.org
>https://pairlist2.pair.net/mailman/listinfo/scons-dev

-- 
Sent from my Android with K-9 Mail.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20151021/c6ec1335/attachment-0001.html>