[Scons-dev] CI Roadmap: ( Was Enabling Travis CI service for SCons on github)

Bill Deegan bill at baddogconsulting.com
Wed Dec 20 12:20:27 EST 2017 

I'm using:
PYTHONPATH=$PWD/QMTest:$PWD/src/engine:$PWD/testing/framework python
src/engine/SCons/EnvironmentValuesSubstTests.py

In my current sandbox.

Most unit tests  (under src/engine) are runnable that way.

The tests under test/... may not be as easy to run without runtest.py.
Still depending on some SCons testing code.

-Bill

On Wed, Dec 20, 2017 at 12:13 PM, Andrew Featherstone <
andrew.featherstone at cantab.net> wrote:

> Are you able to provide a one liner that helps clarify this?
>
> python -m unittest discover -s QMTest --pattern "Test*.py" produces a very
> quick failure.
>
> Andrew
>
> On 20 December 2017 at 16:39, Bill Deegan <bill at baddogconsulting.com>
> wrote:
>
>> 4. We're already using unittest. QMtest is just the name of the directory
>> at this point. (You used to have to install qmtest to run the test suite)
>>
>> On Wed, Dec 20, 2017 at 11:28 AM, Daniel Moody <dmoody256 at gmail.com>
>> wrote:
>>
>>> 1. are you referring to the use of runtest.py command, or how the
>>> various builders and environments are started?
>>> 2. agreed
>>> 3. With the free CI services you are limited to whatever they offer, and
>>> at this point splitting up the tests is a workaround, instead of rewriting
>>> the entire test suite. Even then, scaling it up, it may still be necessary
>>> to split up tests across jobs.
>>> 4. agreed, a well known standard python testing framework would be
>>> great. I like pytest.
>>>
>>> My goal currently is to at least get basic CI on github working with as
>>> little rework as possible.
>>>
>>> On Wed, Dec 20, 2017 at 11:20 AM, Bill Deegan <bill at baddogconsulting.com
>>> > wrote:
>>>
>>>>
>>>>
>>>> On Wed, Dec 20, 2017 at 11:05 AM, Andrew Featherstone <
>>>> andrew.featherstone at cantab.net> wrote:
>>>>
>>>>> My preferred way of working on this problem:
>>>>>
>>>>>
>>>>>    1. Get parity with what happens at http://buildbot.scons.org/#/. I
>>>>>    can't work out from there how to read the actual commands being run in the
>>>>>    CI job, so I'm mostly replicating what happens in the Travis-based builds.
>>>>>
>>>>>
>>>> Basically:
>>>> 1) git clone
>>>> 2) python2.7 runtest.py -a
>>>>
>>>> Current docs and packages aren't being built (but should be)
>>>>
>>>>>
>>>>>    1. Get all of scons' build processes onto cloud-based CI.
>>>>>    Everything that is at http://scons.org/pages/download.html should
>>>>>    come out of a CI job.
>>>>>
>>>>>  Sounds good.
>>>> We used to build dev packages an make available a while back.
>>>> setup.py needs a rewrite and a few other pieces of logic can be removed.
>>>>
>>>>
>>>>>    1.
>>>>>    2. Get the tests running faster. This needs some profiling, but
>>>>>    from the logs it reads like every test is run in a separate process, which
>>>>>    feels pretty expensive to me. There's also references to 2008-era blog
>>>>>    posts which I take with a large grain of salt. We need the tests running as
>>>>>    a pre-requisite of course. Devising way to slice up long running tests
>>>>>    feels like solving the wrong problem.
>>>>>
>>>>> My guess is that most of the tests could have
>>>> DefaultEnvironment(tools=[]), and all Environment()'s could have a more
>>>> limited list of tools. That would make the test faster to much faster
>>>> depending on the test. Likely more speedup on win32 than on linux due to
>>>> msvc detection logic.
>>>>
>>>>
>>>>>
>>>>>    1. Move away from a QMTest to something more commonplace. The
>>>>>    canonical home of QMTest seems to be https://github.com/MentorEmbed
>>>>>    ded/qmtest, last updated in 2011. I think it's time to look at
>>>>>    something else, e.g. plain old unittest or pytest.
>>>>>
>>>>> QMtest is basically gone. Tests are basically unittests now.
>>>>
>>>>
>>>>>
>>>>>    1.
>>>>>
>>>>>  Perhaps it's worth splitting off another thread for roadmapping?
>>>>>
>>>>
>>>> Sure.
>>>>
>>>>
>>>>
>>>>>
>>>>> Andrew
>>>>>
>>>>> On 20 December 2017 at 14:00, Daniel Moody <dmoody256 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> What about this:
>>>>>>
>>>>>>      --interval-start FLOAT   Percentile as float (0.0 - 1.0) of what
>>>>>> test to start on from the
>>>>>>                                            determined list of all
>>>>>> tests
>>>>>>      --interval-end FLOAT     Percentile as float (0.0 - 1.0) of what
>>>>>> test to start on from the
>>>>>>                                            determined list of all
>>>>>> tests
>>>>>>
>>>>>> interval-start default is 0
>>>>>> interval-end default is 1
>>>>>>
>>>>>> A little more flexible I think.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Dec 20, 2017 at 8:41 AM, Bill Deegan <
>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>
>>>>>>> Maybe add a flag directly to runtest.py to split up the tests.
>>>>>>> --test_mod 4 --test_index 0..3 ?
>>>>>>>
>>>>>>> Or something to that affect.
>>>>>>>
>>>>>>> On Wed, Dec 20, 2017 at 8:27 AM, Daniel Moody <dmoody256 at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hey Andrew,
>>>>>>>>
>>>>>>>> I also was working on this. I think in your case the builds timed
>>>>>>>> out due to issues with vswhere:
>>>>>>>> https://github.com/Microsoft/vswhere/issues/87
>>>>>>>> https://github.com/Microsoft/vswhere/issues/91
>>>>>>>>
>>>>>>>> In this case we will need to use the Visual Studio 2017 image.
>>>>>>>>
>>>>>>>> Also just to note; the issue mentioned above is also an issue with
>>>>>>>> SCons MSVS detection in general now for SCons 3 and newer since that is
>>>>>>>> when it was switched to use vswhere.
>>>>>>>>
>>>>>>>> I liked some of the things you had in your script so I took those
>>>>>>>> and merge them into my script. I also implemented a script that will split
>>>>>>>> the build up into multiple jobs like the travis script does, however I have
>>>>>>>> not been able to get appveyor to do multi-line scripts correctly so it is
>>>>>>>> all in a one liner at the moment:
>>>>>>>> https://github.com/dmoody256/scons/blob/AppveyorCI/.appveyor.yml
>>>>>>>> https://ci.appveyor.com/project/dmoody256/scons/build/1.0.53
>>>>>>>>
>>>>>>>> Currently python 3 will fail from several tests so I have them
>>>>>>>> commented out.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Dec 20, 2017 at 8:00 AM, Bill Deegan <
>>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>>
>>>>>>>>> Parallel should help.
>>>>>>>>>
>>>>>>>>> On my buildbot worker (with 2 other builds running single threaded
>>>>>>>>> tests) it takes 2:05.
>>>>>>>>> So on a reasonably modern machine, -j2 should finish in under an
>>>>>>>>> hour if not, try -j3?
>>>>>>>>>
>>>>>>>>> Or we can split up runs as we've done with the travis run..
>>>>>>>>>
>>>>>>>>> -Bill
>>>>>>>>>
>>>>>>>>> On Wed, Dec 20, 2017 at 5:54 AM, Andrew Featherstone <
>>>>>>>>> andrew.featherstone at cantab.net> wrote:
>>>>>>>>>
>>>>>>>>>> I've been trying to get AppVeyor working for Windows-based CI,
>>>>>>>>>> but we're hitting their 1 hour time limit (see
>>>>>>>>>> https://github.com/ajf58/scons/blob/appveyor/.appveyor.yml and
>>>>>>>>>> https://ci.appveyor.com/project/ajf58/scons.
>>>>>>>>>>
>>>>>>>>>> My next pass at this will be trying to run the unit tests in
>>>>>>>>>> parallel (as the Windows VM has two cores available). Either that, or we
>>>>>>>>>> split the job matrix a different way, e.g. run tests grouped by something
>>>>>>>>>> else other than Python version.
>>>>>>>>>>
>>>>>>>>>> Thoughts?
>>>>>>>>>>
>>>>>>>>>> Andrew
>>>>>>>>>>
>>>>>>>>>> On 18 December 2017 at 22:51, Bill Deegan <
>>>>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Daniel,
>>>>>>>>>>>
>>>>>>>>>>> Can we get travis to test with py2.7, 3.5, and 3.6 ?
>>>>>>>>>>>
>>>>>>>>>>> -Bill
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Dec 6, 2017 at 12:03 AM, Bill Deegan <
>>>>>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>> That's pretty cool.
>>>>>>>>>>>> I'll try to get the coverage hooked up soon.
>>>>>>>>>>>> That'll also be very useful..
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Dec 5, 2017 at 8:27 PM, Jonathon Reinhart <
>>>>>>>>>>>> jonathon.reinhart at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, it should automatically do that.
>>>>>>>>>>>>>
>>>>>>>>>>>>> See this (merged) PR from one of my projects:
>>>>>>>>>>>>> https://github.com/JonathonReinhart/scuba/pull/98
>>>>>>>>>>>>>
>>>>>>>>>>>>> Towards the bottom you'll see a "View Details" button.
>>>>>>>>>>>>> Clicking that will expand a box showing the results of all the
>>>>>>>>>>>>> "checks" that ran.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Dec 5, 2017 at 11:13 PM, Bill Deegan <
>>>>>>>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is there a way to get travis to post the results back into
>>>>>>>>>>>>>> the pull request?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Scons-dev mailing list
>>>>>>>>>>>>> Scons-dev at scons.org
>>>>>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Scons-dev mailing list
>>>>>>>>>>> Scons-dev at scons.org
>>>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Scons-dev mailing list
>>>>>>>>>> Scons-dev at scons.org
>>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Scons-dev mailing list
>>>>>>>>> Scons-dev at scons.org
>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Scons-dev mailing list
>>>>>>>> Scons-dev at scons.org
>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-dev mailing list
>>>>>>> Scons-dev at scons.org
>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-dev mailing list
>>>>>> Scons-dev at scons.org
>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-dev mailing list
>>>>> Scons-dev at scons.org
>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-dev mailing list
>>>> Scons-dev at scons.org
>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-dev mailing list
>>> Scons-dev at scons.org
>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>
>>>
>>
>> _______________________________________________
>> Scons-dev mailing list
>> Scons-dev at scons.org
>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>
>>
>
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20171220/e9456519/attachment-0001.html>

More information about the Scons-dev mailing list