[Scons-dev] SCons tools refactoring progress

[Gary Oberbrunner]

You talk a lot about what I consider "policy": specific types of args that
would be used to configure specific types of Tools, like compilers. This is
all fine, but my goal is to build a framework where such policies can be
applied, and then get into the specifics of platforms, cross-compilers,
versions, targets and all that. Building a game asset manager, or a
visual-effects pipeline, or a document-preparation system, or a
scientific-data analysis pipeline, will have very different _specifics_ but
the same general mechanism should be usable for all, or at least that's my
goal.

Jason, since you've already been down this path once, I'd very much
appreciate a code and design review of what I have so far. It's only 700
lines or so, including tests. I'm especially interested in use cases Parts
can handle that my proposed system wouldn't be able to. (Again, from a
mechanism or base framework standpoint; not that I don't specifically
define a target architecture argument for instance.)

One use case I see from your notes below is this one, which I'll capture:
  As a SConscript writer, I'd like to be able to enumerate all versions of
a tool on the system so I can choose which one to use.
If there are others, please send them along.

-- Gary

On Tue, Oct 20, 2015 at 11:39 AM, Jason Kenny <dragon512 at live.com> wrote:

> Hi Gary,
>
>
>
> From looking at the below this seems to be in line with what is going on
> in Parts.  I had hoped that what I did here would be used /improved on in
> Scons. That is the a goal of Parts.. to improve Scons.
>
>
>
> From what you talk about below I think we are in general on the same page.
> I know when I started this I was also of the view that the “args” should be
> external. At the time Steve Knight release pushed hard that all value could
> be defined in the environment. I see some value in this, however I was like
> you in which I wanted to have something separate to define arguments to
> setup a tool. I also wanted to have a standard set of values. For example
> we should use the same values to talk about archecture, version, etc.. when
> we can as this makes it easier to use and develop new tools. Given what I
> have seen in practice, I was moving towards the IATAP idea from Greg. This
> is what I call Settings in Parts. While I have not finished this object yet
> I think it goes toward this idea better. Here we have an object that given
> values creates an environment to be used to build stuff. I think at a high
> level we want to move this way.
>
>
>
> For a design of the tools I think these are items we want to understand
> and define some way.
>
>
>
> *Platform*- Parts define a system platform object, Scons defines a
> string, for the OS, it has nothing standard for the architecture. I think
> we should agree we need to define this better in Scons. I would like to see
> something in Scons defining this. I suggest what I have in Parts as a way
> to start. I define in a given environment a HOST_PLATFORM object and a
> TARGET_PLATFORM Object. This is to extend the PLATFORM string we have now.
> The way I did in in Parts allows these items to be synced and to extended
> with new os or archecture value at run time. I think having something like
> this in SCons will allow for a better more reusable tool design.
>
>
>
> *Finders*- In Parts I have Finders and Scanners. These are different in
> what they return, but in effect are the same thing. I believe in the case
> of what Gary is talking about in Scons they serve the same purpose. The
> goal is it to return if we found something at a given resource location,
> given that being a path, registry entries, or environment variable. I
> should point out in someway this is like a autoconf test to find a
> compiler, etc. I would like to think this way is better however. I Parts I
> have finders and scanners. The difference in Parts is that a scanner
> returns back a set of value with version information, where a finder is
> much more simple “We have a match as resource X”. The reason for the
> difference is that some tools are easier to find as a set vs defining every
> known case. For example when I worked at Intel, the Intel compiler would
> make lots of internal drops as one would expect, these drops would have
> different versions. People would want to test the new drops out. Instead of
> me making a new entry for each new drop, I scanned the disk for example for
> what was installed. And returned all versions installed on disk. This
> scaled better for this tool set for users. Another example ( which I should
> improve a bit more) is the GNU tool change. People can install different
> version of the compiler on a Linux system and they often install them in
> different ways, such as under /opt or in there home directory in a standard
> way. I tried to have the Gnu logic in Parts look for these common places
> and use these version of requested. The use of a scanner logic was a lot
> easier here than making lots of custom finders. The main value of a finder
> vs scanner for me however is that a finder can be more declarative. This
> makes it easier to read and understand. For a system like SCons this is
> important for people to add new tools in a common way to be reused.
>
>
>
> *Tool args*- I don’t define this at the moment as an object. I define
> some common values, but not an object. Originally I was going to define an
> object. However Steve knight convinced me otherwise. There is some value in
> that choice. Making it an object has some different value. I not against or
> for making a tools arg like object. What I am against if a notions of a
> **kw object which was something Steve Knight basically had and what seems
> to be defined here in the new tool form Gary. The reason I have issue with
> this is that this allows the utter chaos we have in the current tools.. ie
> there is no standard way to set a given tool up. I agree a tool should be
> able to define unique items if it makes sense, but we should strive for a
> common set of basic values that we reuse in all tools. Some value like a
> version may be ignored as it does not make sense for it. We still should
> define a common set as most tool have a common ground and people need a
> common way to set them up. Sure advance cases will exists, we have a way
> with **kw to add advance stuff as it happens. For me value that are useful
> here are an idea of ( This is just a base set I found useful…not the end
> all end all)
>
> 1)      *Root path* – this is the base path we use to find the tool and
> extend with extra info
>
> 2)      *Version* – this is the version of the tool we want, or selector
> of the best match of the tool we want. ( ie 3.2.2 is exact where a 3 finds
> the highest version of 3 as 3.4 on the system. Yes many time we want the
> latest, or the version does not matter for the tool. This should exist, but
> does not have to be defined to get a tool to work
>
> 3)      *Use_script* – for me this tells us who to setup the environment.
> I use it as a way to point a custom script to use to setup the environment,
> vs a set of known defaults or to point to a known script ( based on the
> root path) to setup the environment
>
> 4)      *Target platform* – this is the target platform we want to build
> on. Not always needed, as we default to the host. Having a common want to
> talk about the target is powerful and very useful. I should not that many
> tools have different tools based on what host-target combo exists. Many
> cases we have x86 –> x64 and x64 -> x64 like tools on a platform. I never
> have seen a case in which not preferring a native client is the not the
> correct answer. This can be an important details as some may have a native
> x64 and other only have the x86->x64 case even though they have a x64 OS.
>
>
>
> *Tool environment instance*- I Parts I called this ToolInfo. This is not
> really different from what Gary talks about concreate instance of the tool.
> This is the object that is the information for a given instance of a tool
> we may want to use. Has information about what to add to the Environment.
> For me at least these object are define in a file and given finders and
> other information to expand to the environment when it is asked to fill in
> the environment with needed information to all a given tool to run/work.
> The main interface for this object at run time is an exist call and a get
> environment kv to add to the environment  ( or a function to set the
> environment directly). Both of these functions would take some sort of
> tool_args. This object would use the finders and other information given to
> it to find if that instance exists. So for example it would have finders
> for getting information on the path, or shell variable to find a root
> directory to use to create the real path to the exe, which it can use to
> test that it really exists. A common case would be that you might have:
>
>
>
> Standard look up
>
> 1)      Finder to path finder finds pat
>
> 2)      This object uses that path and test that when we subst the full
> path value that the provided tool name exists
>
> More custom
>
> 1)      Finder to path finders fails to find anything
>
> 2)      Next finder in list is an environment finder, it find a the
> variable defined to it exists.
>
> 3)      We use the value of this shell variable to subst the full path,
> and finds it the tools
>
>
>
> Either way this allow a way to find and setup what we need. In Parts I did
> the setup to allow for setting the environment via define defaults, or by
> calling a predefined script or a custom script. Scripts are slower to call,
> and add risk of differences (as they could be modified), but they add a lot
> of flexibility for custom setup cases while reusing the main tools logic.
>
>
>
> *Tool info container*- The other object needed is something to hold all
> these different tool cases. I called this ToolSetting in Parts. I did not
> make it as one global object, but many different obejcts, such as MSVC,
> GCC, etc. There might be some value to make it that way given what Gary
> proposes that I think would be an improvement on my design. The main point
> of this object for me is that this is the guy we register a set of tool
> information with, ie the exact platform requirements ( ie host and target
> it is for), and the tool info itself. This interface is a simple register
> at load time. If done right/well I should allow for an easy declarative
> definition, that should make it easy for other to read, copy and paste and
> tweak if needed. The other function is a query function to get a tool
> instance, based on some tool_args and to test for exist of some tool based
> on tool args. The only major function I would add in Parts that I don’t
> have in this case is a function to return all know information. This can be
> great to allow people to easily test via a simple command what the build
> system knows about. In Parts have this working via a hack of defining
> something we know does not exist to get an error message about what it
> could not find, and what are valid option the systems knows about.. just to
> note what I would change here ( minus the internally ugly but working impl
> I have at the moment). I also think this is a place I would add logic to
> cache known information about known tools to avoid re-looking it up for
> startup time improvements as a polish step. Would be useful for the generic
> check a lot of tools toolchain setups as these could be defined.
>
>
>
> *Toolchain* – This is a key feature that we need to have added. I think
> core ideas of easily defining a toolchain in some way is critical. Some
> quick thoughts. In Parts I never really got this done. What I do have is a
> way to define something in Files under a toolchain directory. What is great
> about this is that it easy for users to define there own changes, local to
> there own build or systems. That is a plus. What is not good. I did not
> define a good interface for this ( lack of time and I never got back to it
> correctly…) . This mean I cannot easily define a toolchain in a Sconstruct.
> I hope to address this with the Setting (IAPAT) object going forward. It is
> needed that user can define a toolchain in a the Sconstruct, in independent
> files and possibly have some control on the command line. On the command
> line I think user really need and want a way to influence a toolchain
> setup, such as use version X of a tool or use this path instead to find the
> tool, vs what is defined by default. In Parts I allow this to a degree with
> –toolchain (--tc for short). The argument is in a  tool[_version],[…]
> format where tool can be a tool or toolchain. I would redo this ideally to
> have more of a --tc=toolchain[@key:value[@key:value],[...]] like format to
> pass in a kv value dict to help influence a toolchain setup. Beyond this we
> need a good way to define a toolchain. There are a lot of ideas out there
> for this. Common on include some sort of anyof(), oneof() combination to
> define more complex toolchains with fallbacks. Or having an idea of a
> common tool type such as CC and all tool define themselves as a subset of a
> “master” tool concept. I really open to ideas here myself.
>
>
>
>
>
> Ideas we can layer on top of this
>
>
>
> Once we do a better tools setup. We can layer an idea on top of it like
> configuration to allow applying common flags to the environment after the
> tools are configured. This is great as it easily allows common values to be
> used as a whole and easily changed to try a new set, without having to
> modify the build scripts. In the case of Parts this has been great for
> complex build and cross build cases as uses can easily define common flags,
> etc for a given tool/version/etc to be used based on the tool chain a
> environment is using. This should be separate at this point.. how I just
> point it out as it a layer to think about as it would reuse tool
> information in a toolchain to provide more value. Something to think about
> in the design of building up the tool correctly.
>
>
>
>
>
> I would hope that the concepts I talked about here are something we can
> find common ground on and talk about defining some common name for. I know
> this is a large e-mail, However I like very much some thoughts about what I
> suggest at high level to form a base of communication for use on the tool
> design. I am not saying my impl in Parts is what I would like it to be, but
> I think it does so common design points we want to discuss and the value of
> a reasonably easy to use and reusable system can have.
>
>
>
> Jason
>
>
>
>
>
>
>
> Sent from Mail <http://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>



-- 
Gary
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20151020/41d28404/attachment-0001.html>