[Scons-dev] CI Roadmap: ( Was Enabling Travis CI service for SCons on github)

Andrew Featherstone andrew.featherstone at cantab.net
Wed Dec 20 12:13:29 EST 2017 

Are you able to provide a one liner that helps clarify this?

python -m unittest discover -s QMTest --pattern "Test*.py" produces a very
quick failure.

Andrew

On 20 December 2017 at 16:39, Bill Deegan <bill at baddogconsulting.com> wrote:

> 4. We're already using unittest. QMtest is just the name of the directory
> at this point. (You used to have to install qmtest to run the test suite)
>
> On Wed, Dec 20, 2017 at 11:28 AM, Daniel Moody <dmoody256 at gmail.com>
> wrote:
>
>> 1. are you referring to the use of runtest.py command, or how the various
>> builders and environments are started?
>> 2. agreed
>> 3. With the free CI services you are limited to whatever they offer, and
>> at this point splitting up the tests is a workaround, instead of rewriting
>> the entire test suite. Even then, scaling it up, it may still be necessary
>> to split up tests across jobs.
>> 4. agreed, a well known standard python testing framework would be great.
>> I like pytest.
>>
>> My goal currently is to at least get basic CI on github working with as
>> little rework as possible.
>>
>> On Wed, Dec 20, 2017 at 11:20 AM, Bill Deegan <bill at baddogconsulting.com>
>> wrote:
>>
>>>
>>>
>>> On Wed, Dec 20, 2017 at 11:05 AM, Andrew Featherstone <
>>> andrew.featherstone at cantab.net> wrote:
>>>
>>>> My preferred way of working on this problem:
>>>>
>>>>
>>>>    1. Get parity with what happens at http://buildbot.scons.org/#/. I
>>>>    can't work out from there how to read the actual commands being run in the
>>>>    CI job, so I'm mostly replicating what happens in the Travis-based builds.
>>>>
>>>>
>>> Basically:
>>> 1) git clone
>>> 2) python2.7 runtest.py -a
>>>
>>> Current docs and packages aren't being built (but should be)
>>>
>>>>
>>>>    1. Get all of scons' build processes onto cloud-based CI.
>>>>    Everything that is at http://scons.org/pages/download.html should
>>>>    come out of a CI job.
>>>>
>>>>  Sounds good.
>>> We used to build dev packages an make available a while back.
>>> setup.py needs a rewrite and a few other pieces of logic can be removed.
>>>
>>>
>>>>    1.
>>>>    2. Get the tests running faster. This needs some profiling, but
>>>>    from the logs it reads like every test is run in a separate process, which
>>>>    feels pretty expensive to me. There's also references to 2008-era blog
>>>>    posts which I take with a large grain of salt. We need the tests running as
>>>>    a pre-requisite of course. Devising way to slice up long running tests
>>>>    feels like solving the wrong problem.
>>>>
>>>> My guess is that most of the tests could have
>>> DefaultEnvironment(tools=[]), and all Environment()'s could have a more
>>> limited list of tools. That would make the test faster to much faster
>>> depending on the test. Likely more speedup on win32 than on linux due to
>>> msvc detection logic.
>>>
>>>
>>>>
>>>>    1. Move away from a QMTest to something more commonplace. The
>>>>    canonical home of QMTest seems to be https://github.com/MentorEmbed
>>>>    ded/qmtest, last updated in 2011. I think it's time to look at
>>>>    something else, e.g. plain old unittest or pytest.
>>>>
>>>> QMtest is basically gone. Tests are basically unittests now.
>>>
>>>
>>>>
>>>>    1.
>>>>
>>>>  Perhaps it's worth splitting off another thread for roadmapping?
>>>>
>>>
>>> Sure.
>>>
>>>
>>>
>>>>
>>>> Andrew
>>>>
>>>> On 20 December 2017 at 14:00, Daniel Moody <dmoody256 at gmail.com> wrote:
>>>>
>>>>> What about this:
>>>>>
>>>>>      --interval-start FLOAT   Percentile as float (0.0 - 1.0) of what
>>>>> test to start on from the
>>>>>                                            determined list of all tests
>>>>>      --interval-end FLOAT     Percentile as float (0.0 - 1.0) of what
>>>>> test to start on from the
>>>>>                                            determined list of all
>>>>> tests
>>>>>
>>>>> interval-start default is 0
>>>>> interval-end default is 1
>>>>>
>>>>> A little more flexible I think.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Dec 20, 2017 at 8:41 AM, Bill Deegan <
>>>>> bill at baddogconsulting.com> wrote:
>>>>>
>>>>>> Maybe add a flag directly to runtest.py to split up the tests.
>>>>>> --test_mod 4 --test_index 0..3 ?
>>>>>>
>>>>>> Or something to that affect.
>>>>>>
>>>>>> On Wed, Dec 20, 2017 at 8:27 AM, Daniel Moody <dmoody256 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hey Andrew,
>>>>>>>
>>>>>>> I also was working on this. I think in your case the builds timed
>>>>>>> out due to issues with vswhere:
>>>>>>> https://github.com/Microsoft/vswhere/issues/87
>>>>>>> https://github.com/Microsoft/vswhere/issues/91
>>>>>>>
>>>>>>> In this case we will need to use the Visual Studio 2017 image.
>>>>>>>
>>>>>>> Also just to note; the issue mentioned above is also an issue with
>>>>>>> SCons MSVS detection in general now for SCons 3 and newer since that is
>>>>>>> when it was switched to use vswhere.
>>>>>>>
>>>>>>> I liked some of the things you had in your script so I took those
>>>>>>> and merge them into my script. I also implemented a script that will split
>>>>>>> the build up into multiple jobs like the travis script does, however I have
>>>>>>> not been able to get appveyor to do multi-line scripts correctly so it is
>>>>>>> all in a one liner at the moment:
>>>>>>> https://github.com/dmoody256/scons/blob/AppveyorCI/.appveyor.yml
>>>>>>> https://ci.appveyor.com/project/dmoody256/scons/build/1.0.53
>>>>>>>
>>>>>>> Currently python 3 will fail from several tests so I have them
>>>>>>> commented out.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Dec 20, 2017 at 8:00 AM, Bill Deegan <
>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>
>>>>>>>> Parallel should help.
>>>>>>>>
>>>>>>>> On my buildbot worker (with 2 other builds running single threaded
>>>>>>>> tests) it takes 2:05.
>>>>>>>> So on a reasonably modern machine, -j2 should finish in under an
>>>>>>>> hour if not, try -j3?
>>>>>>>>
>>>>>>>> Or we can split up runs as we've done with the travis run..
>>>>>>>>
>>>>>>>> -Bill
>>>>>>>>
>>>>>>>> On Wed, Dec 20, 2017 at 5:54 AM, Andrew Featherstone <
>>>>>>>> andrew.featherstone at cantab.net> wrote:
>>>>>>>>
>>>>>>>>> I've been trying to get AppVeyor working for Windows-based CI, but
>>>>>>>>> we're hitting their 1 hour time limit (see
>>>>>>>>> https://github.com/ajf58/scons/blob/appveyor/.appveyor.yml and
>>>>>>>>> https://ci.appveyor.com/project/ajf58/scons.
>>>>>>>>>
>>>>>>>>> My next pass at this will be trying to run the unit tests in
>>>>>>>>> parallel (as the Windows VM has two cores available). Either that, or we
>>>>>>>>> split the job matrix a different way, e.g. run tests grouped by something
>>>>>>>>> else other than Python version.
>>>>>>>>>
>>>>>>>>> Thoughts?
>>>>>>>>>
>>>>>>>>> Andrew
>>>>>>>>>
>>>>>>>>> On 18 December 2017 at 22:51, Bill Deegan <
>>>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>>>
>>>>>>>>>> Daniel,
>>>>>>>>>>
>>>>>>>>>> Can we get travis to test with py2.7, 3.5, and 3.6 ?
>>>>>>>>>>
>>>>>>>>>> -Bill
>>>>>>>>>>
>>>>>>>>>> On Wed, Dec 6, 2017 at 12:03 AM, Bill Deegan <
>>>>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>> That's pretty cool.
>>>>>>>>>>> I'll try to get the coverage hooked up soon.
>>>>>>>>>>> That'll also be very useful..
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Dec 5, 2017 at 8:27 PM, Jonathon Reinhart <
>>>>>>>>>>> jonathon.reinhart at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Yes, it should automatically do that.
>>>>>>>>>>>>
>>>>>>>>>>>> See this (merged) PR from one of my projects:
>>>>>>>>>>>> https://github.com/JonathonReinhart/scuba/pull/98
>>>>>>>>>>>>
>>>>>>>>>>>> Towards the bottom you'll see a "View Details" button.
>>>>>>>>>>>> Clicking that will expand a box showing the results of all the
>>>>>>>>>>>> "checks" that ran.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Dec 5, 2017 at 11:13 PM, Bill Deegan <
>>>>>>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Is there a way to get travis to post the results back into the
>>>>>>>>>>>>> pull request?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Scons-dev mailing list
>>>>>>>>>>>> Scons-dev at scons.org
>>>>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Scons-dev mailing list
>>>>>>>>>> Scons-dev at scons.org
>>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Scons-dev mailing list
>>>>>>>>> Scons-dev at scons.org
>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Scons-dev mailing list
>>>>>>>> Scons-dev at scons.org
>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-dev mailing list
>>>>>>> Scons-dev at scons.org
>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-dev mailing list
>>>>>> Scons-dev at scons.org
>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-dev mailing list
>>>>> Scons-dev at scons.org
>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-dev mailing list
>>>> Scons-dev at scons.org
>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-dev mailing list
>>> Scons-dev at scons.org
>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>
>>>
>>
>> _______________________________________________
>> Scons-dev mailing list
>> Scons-dev at scons.org
>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>
>>
>
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20171220/3d28d836/attachment-0001.html>

More information about the Scons-dev mailing list