[Scons-dev] MSVC 2017+ Toolset Support

[Joseph Brill]

This email is an attempt to focus the discussion concerning support for
MSVC toolsets moving forward in a rational, dispassionate discourse.

I am advocating a minor shift in orientation or "world view" for MSVC 2017
and later: the semantic transition from thinking of the msvc version as a
product to thinking of the msvc version as a toolset.  This actually makes
the implementation of toolset support easier.

There is a working version of the support for toolsets at varying levels of
precision that can be enabled and disabled.  The intent is that a
command-line switch could be added to enable the toolset specific
capabilities.  By default, the toolset specific capabilities would be
disabled in which case the code behaves exactly like the current master.
This would allow field testing and evaluation of the functionality in other
environments and to easily test the differences between the current master
and the toolset functionality.  The functional aspects are feature
complete.  The working version is a product of more than one man-month of
full-time development activity and the result of the third iteration of
implementation strategy.

Existing PR #3717 was closed and the branch renamed due in part to make it
easier to identify and maintain for myself as I fear that I might be time
limited in the near term and due in part to the existing discussion
unraveling fairly quickly with dim prospects of inclusion moving
forward.  Despite
being closed, I recommend viewing the "documentation" (the PR text and the
first comment) and the vc.py code difference in PR #3717.  At any time the
PR request can be reopened.  I would need a git guru to walk me through
associating the new branch name with the "old" PR.

The implementation knocks an item or two off the internal vc.py TODO list
such as a single vswhere query and processing the json results for all
known installations.  Ranked toolset lists are constructed by host/target.
The ranking takes into account the toolset version and product type.  A
single toolset instance may be a member of multiple toolset lists.
Expensive lookups are "runtime memoized/cached" so that they are only
computed once.  While there is a fair amount of initialization, the
processing burden is based on the number of products installed and the
number of toolsets for each product.  Typical environments likely have a
single product installation with a spartan number of different toolsets
installed. There is nothing particularly "clever" done in the
implementation and should not be a maintenance burden.  The bulk of the
toolset support implementation follows the code in the existing master
(i.e., the bottom of the file).  The modifications to support toolsets
weighs in at approximately 1400 lines of code.

Prior to MSVS 2017, there was a direct 1:1 mapping between the msvc
*product* and the msvc *toolset*. The toolset might be upgraded in an
update or service pack but there was no way to select a specific toolset as
side-by-side installations were not supported.  That changed with the
release of MSVC 2017 which allows multiple side-by-side toolset
installations by target.  MSVC 2017 and later, can be viewed as
*toolset* oriented
(i.e., a MSVC 2019 installation can be viewed as a container for 14.1X,
14.2X, and transparently the 14.0 toolsets).  Moving forward, I believe
that it will be advantageous to change from a *product* orientation to a
*toolset* orientation.  This change is only for the 2017 and later
installations and currently really only affects 2017.

With MSVC 2017 and MSVC 2019, an SCons version request of "14.1" or "14.2",
respectively, *is a toolset request* that cannot be specified at a finer
granularity. The current msvc behavior uses the *default toolset* which is
generally the newest installed *toolset* version when the "--vcvars_ver"
argument is not specified for the vcvars batch file.

For example, a version request of "14.1" is likely to use the "14.16.27023"
toolset when MSVC 2017 is installed. Effectively, the *product request* is
a *toolset request* identical to "find the newest toolset version installed
that starts with '14.1'". However, "14.2" (MSVC 2019) will return a
*toolset* based on the latest update which likely varies across users and
computers.  Currently, a request for "14.1" will fail if MSVC 2019 is
installed with the 14.1 toolset.  With a toolset orientation, a 14.1
toolset request would be satisfied first by the native product (i.e., MSVC
2017) for the toolset and then by the latest product that contains the user
specified toolset for a given host/target combination.  This two-stage
query, if necessary, is implemented in the working toolset oriented version.

Currently, the actual toolset versions used for 2017 and 2019 are based on
end-user's installation options and frequency of updates. While two
different users may be using the same *product* (e.g., 2019) they may not
be using the same *toolset*. Similarly, the installed toolset version is
not necessarily the same for all host/target combinations within a given
product installation.  The current implementation uses a default
*toolset* within
a product across different users and environments, which may not be the
same for all instances, but currently cannot use a *toolset* across product
installations.

There may be many reasons (e.g., enterprise and/or customer requirements)
that build scripts are tied to a specific toolset.  If a build depends on
the 14.1 *toolset* it hardly seems as important in which *product* container
it is installed from an end-user's perspective given the two-stage
selection rule described earlier.  Given the same toolset, the produced
binaries should be binary compatible independent of the product
installation.  The same issue will arise when the next MSVC *product* is
released and end-users desire to use the 14.2X *toolset* combination in the
new product container.

At present, the shift in toolset orientation would only affect user's that
desire the 14.1 toolset and do not have MSVC 2017 installed.  In this case,
the only other product that can be used is MSVC 2019.  This isolates the
effects of changes to a single version of MSVC.

This could have a tangible side-effect of making build scripts more
portable: the build scripts can be tied to a toolset rather than a
toolset/product combination.  Although if it is desired to specify the
toolset/product combination that is supported as well.

The MSVC_VERSION format was expanded to allow:

   - specifying an individual toolset at varying degrees of precision:
      - "14.1"
      - "14.16"
      - "14.16.27"
      - "14.16.27023"
   - specifying individual product types (attached to a specific toolset or a
   product):
      - Enterprise: "14.26Ent"
      - Professional: "14.2Pro"
      - Community: "14.26.27Com"
      - Build Tools: "14.2BT"
      - Express: "14.1Exp"
   - a right assignment operator ("->") that maps a specific toolset to a
   defined product version with an optional product type:
      - "14.0->14.2" use the the 14.0 toolset via a 2019 installation
      - "14.1->14.2BT" use the 14.1 toolset via a 2019 Build Tools
      installation
      - "14.1->14.2Com" use the 14.1 toolset via a 2019 Community
      installation

The translation of the extended version specification to a supplementary
internal data structure is done in exactly one source code location:
msvc_setup_env.  All existing code continues to operate on the "NN.M"
format for MSVC_VERSION as before.  Existing msvc/msvs test suites pass all
tests.  However, issues with the extended format being used prior to entry
in vc.py script have yet to be identified, if any.  The working version is
completely self-contained in the vc.py script.

Without a product qualifier (e.g., "BT"), there is a *subtle but
significant* change semantically: the product selected will be the
newest/latest toolset that matches the user specification within installed
toolsets for a given (host,target) combination. Effectively, without a
product qualifier, any product type could be returned based on the
installed toolsets. This only affects multiple installations of the same
product version (e.g., 14.1 and 14.Exp and/or 14.2Com and 14.2BT).  For end
users that only have one product installed, there is no reason to specify a
product type.

This means that a user does not need to specify "14.1Exp" explicitly if
there is a single 14.1 product installation. On the flip side, it also
means that "14.1Exp" may be returned for a "14.1" request. In local
testing, "14.1Exp" was returned for an "arm" target. In the local
installations of 14.1 Community and 14.1 Express, the "newest" toolset for
an arm target was in the 14.1 Express installation as it was installed the
most recently.  With a product qualifier, a specific product version and
type is selected.

To be clear, I do not need any fixes, enhancements, and am not reporting
any bugs.  The toolset version development was done in the spirit of
contributing to software that I have used since 2.X.  I had a development
window to contribute to the windows/msvc support and in process address two
of the outstanding msvc items on the issues list.  I was motivated by both
the post in the scons users mailing list shown below and issue #3664.  As I
am familiar with the msvc detection code, my initial thoughts were "how
hard could it be?"

Regards,

Joe

https://pairlist4.pair.net/pipermail/scons-users/2020-June/008216.html

>* The problem is scons don't find any 14.1(2017) installation, he only find
*>* 14.2(2019).
*> >* Is there something I could do to tell scons that 141 is
installed somewhere
*>* in vs2019 installation?
*
at the moment, no... it's a work in progress.

See https://github.com/SCons/scons/issues/3664

There wasn't really a "Windows native" person involved in the most
recent times, so we didn't end up understanding about the model of
multiple versions under one VS install - and in particular that people
would want to get 14.1 from VS2019 rather than keep VS2017 installed.
For the most recent VS installs, vswhere is used to detect them, but we
find only the "default version" under each product vswhere reports, so
14.2.latest for 2019, 14.1 for 2017.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20200809/4114d855/attachment-0001.html>