[Scons-dev] CI and testing in general

[Kenny, Jason L]

Ya

I am ... I have been adding an updated tool logic on Parts over the default Scons logic. The issue I have still been trying to figure out how to deal with is how to test that when I have icc 13 support added or cl 2012 support added how to do I automate the testing of these tools. Currently I just test it manually and hope that is good enough as the tools is not going to change, but it does not really test the tool well as for as much as I know there might be a small but subtle change in the OS or something that could break what we view as true some day tomorrow. Then there is the other issue of how to test the different versions etc. This leads to an issue in which I need a system with all or some of the tools on it, with systems without some of the tools to make sure I tested not finding it, or in different places to make sure certain logic provided by the users works to find tools in unknown places.

The way I have been looking at so far is that it best for me to test the finding logic on a "fake" tools. 
Given that this works, I should not really need to test each possible tools as logic for find it is just applying the correct meta value about how to find the tools.
The testing the builder comes up. I think that the builder falls under the same logic in general, however one needs to test complex tool relationships.. such as compiling Fortran with C++ code and linking. This can blow up again, but there is a view that the tools already tested and documented what they works with, is it really the point of the build system to point out bad combinations? I think here that it is not, and such bad combination would cause some warning or error message by the tool being used

Now for pure Scons the tools apply random logic in how the tool is setup and or found etc... I feel this is a testing a design nightmare. That I why I have the ToolSetting objects, and I have tried to get these item added to SCons ( I guess I should do my next round of clean up in these objects and push it more directly). These objects allow a common more declarative way for us to find a give tool and or version of the tool. It basically and engine that takes some meta data and allow for a tools to be found, and have the needed environment setup done in a common way, without having to rewrite a tool file like intelc or msvc, etc.. every time there is a major change version release or change in some logic. I think with a setup like what I propose ( or if you feel you can improve it great...) this would allow a more testable way for use to new tools, as the testing really is only verifying that the core engine logic works, and the metadata about how to find a tool is correct.

Just my two cents...

Jason