[Scons-dev] CI Roadmap: ( Was Enabling Travis CI service for SCons on github)

Daniel Moody dmoody256 at gmail.com
Wed Dec 20 11:28:16 EST 2017 

1. are you referring to the use of runtest.py command, or how the various
builders and environments are started?
2. agreed
3. With the free CI services you are limited to whatever they offer, and at
this point splitting up the tests is a workaround, instead of rewriting the
entire test suite. Even then, scaling it up, it may still be necessary to
split up tests across jobs.
4. agreed, a well known standard python testing framework would be great. I
like pytest.

My goal currently is to at least get basic CI on github working with as
little rework as possible.

On Wed, Dec 20, 2017 at 11:20 AM, Bill Deegan <bill at baddogconsulting.com>
wrote:

>
>
> On Wed, Dec 20, 2017 at 11:05 AM, Andrew Featherstone <
> andrew.featherstone at cantab.net> wrote:
>
>> My preferred way of working on this problem:
>>
>>
>>    1. Get parity with what happens at http://buildbot.scons.org/#/. I
>>    can't work out from there how to read the actual commands being run in the
>>    CI job, so I'm mostly replicating what happens in the Travis-based builds.
>>
>>
> Basically:
> 1) git clone
> 2) python2.7 runtest.py -a
>
> Current docs and packages aren't being built (but should be)
>
>>
>>    1. Get all of scons' build processes onto cloud-based CI. Everything
>>    that is at http://scons.org/pages/download.html should come out of a
>>    CI job.
>>
>>  Sounds good.
> We used to build dev packages an make available a while back.
> setup.py needs a rewrite and a few other pieces of logic can be removed.
>
>
>>    1.
>>    2. Get the tests running faster. This needs some profiling, but from
>>    the logs it reads like every test is run in a separate process, which feels
>>    pretty expensive to me. There's also references to 2008-era blog posts
>>    which I take with a large grain of salt. We need the tests running as a
>>    pre-requisite of course. Devising way to slice up long running tests feels
>>    like solving the wrong problem.
>>
>> My guess is that most of the tests could have
> DefaultEnvironment(tools=[]), and all Environment()'s could have a more
> limited list of tools. That would make the test faster to much faster
> depending on the test. Likely more speedup on win32 than on linux due to
> msvc detection logic.
>
>
>>
>>    1. Move away from a QMTest to something more commonplace. The
>>    canonical home of QMTest seems to be https://github.com/MentorEmbed
>>    ded/qmtest, last updated in 2011. I think it's time to look at
>>    something else, e.g. plain old unittest or pytest.
>>
>> QMtest is basically gone. Tests are basically unittests now.
>
>
>>
>>    1.
>>
>>  Perhaps it's worth splitting off another thread for roadmapping?
>>
>
> Sure.
>
>
>
>>
>> Andrew
>>
>> On 20 December 2017 at 14:00, Daniel Moody <dmoody256 at gmail.com> wrote:
>>
>>> What about this:
>>>
>>>      --interval-start FLOAT   Percentile as float (0.0 - 1.0) of what
>>> test to start on from the
>>>                                            determined list of all tests
>>>      --interval-end FLOAT     Percentile as float (0.0 - 1.0) of what
>>> test to start on from the
>>>                                            determined list of all tests
>>>
>>> interval-start default is 0
>>> interval-end default is 1
>>>
>>> A little more flexible I think.
>>>
>>>
>>>
>>> On Wed, Dec 20, 2017 at 8:41 AM, Bill Deegan <bill at baddogconsulting.com>
>>> wrote:
>>>
>>>> Maybe add a flag directly to runtest.py to split up the tests.
>>>> --test_mod 4 --test_index 0..3 ?
>>>>
>>>> Or something to that affect.
>>>>
>>>> On Wed, Dec 20, 2017 at 8:27 AM, Daniel Moody <dmoody256 at gmail.com>
>>>> wrote:
>>>>
>>>>> Hey Andrew,
>>>>>
>>>>> I also was working on this. I think in your case the builds timed out
>>>>> due to issues with vswhere:
>>>>> https://github.com/Microsoft/vswhere/issues/87
>>>>> https://github.com/Microsoft/vswhere/issues/91
>>>>>
>>>>> In this case we will need to use the Visual Studio 2017 image.
>>>>>
>>>>> Also just to note; the issue mentioned above is also an issue with
>>>>> SCons MSVS detection in general now for SCons 3 and newer since that is
>>>>> when it was switched to use vswhere.
>>>>>
>>>>> I liked some of the things you had in your script so I took those and
>>>>> merge them into my script. I also implemented a script that will split the
>>>>> build up into multiple jobs like the travis script does, however I have not
>>>>> been able to get appveyor to do multi-line scripts correctly so it is all
>>>>> in a one liner at the moment:
>>>>> https://github.com/dmoody256/scons/blob/AppveyorCI/.appveyor.yml
>>>>> https://ci.appveyor.com/project/dmoody256/scons/build/1.0.53
>>>>>
>>>>> Currently python 3 will fail from several tests so I have them
>>>>> commented out.
>>>>>
>>>>>
>>>>> On Wed, Dec 20, 2017 at 8:00 AM, Bill Deegan <
>>>>> bill at baddogconsulting.com> wrote:
>>>>>
>>>>>> Parallel should help.
>>>>>>
>>>>>> On my buildbot worker (with 2 other builds running single threaded
>>>>>> tests) it takes 2:05.
>>>>>> So on a reasonably modern machine, -j2 should finish in under an hour
>>>>>> if not, try -j3?
>>>>>>
>>>>>> Or we can split up runs as we've done with the travis run..
>>>>>>
>>>>>> -Bill
>>>>>>
>>>>>> On Wed, Dec 20, 2017 at 5:54 AM, Andrew Featherstone <
>>>>>> andrew.featherstone at cantab.net> wrote:
>>>>>>
>>>>>>> I've been trying to get AppVeyor working for Windows-based CI, but
>>>>>>> we're hitting their 1 hour time limit (see
>>>>>>> https://github.com/ajf58/scons/blob/appveyor/.appveyor.yml and
>>>>>>> https://ci.appveyor.com/project/ajf58/scons.
>>>>>>>
>>>>>>> My next pass at this will be trying to run the unit tests in
>>>>>>> parallel (as the Windows VM has two cores available). Either that, or we
>>>>>>> split the job matrix a different way, e.g. run tests grouped by something
>>>>>>> else other than Python version.
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> Andrew
>>>>>>>
>>>>>>> On 18 December 2017 at 22:51, Bill Deegan <bill at baddogconsulting.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Daniel,
>>>>>>>>
>>>>>>>> Can we get travis to test with py2.7, 3.5, and 3.6 ?
>>>>>>>>
>>>>>>>> -Bill
>>>>>>>>
>>>>>>>> On Wed, Dec 6, 2017 at 12:03 AM, Bill Deegan <
>>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>> That's pretty cool.
>>>>>>>>> I'll try to get the coverage hooked up soon.
>>>>>>>>> That'll also be very useful..
>>>>>>>>>
>>>>>>>>> On Tue, Dec 5, 2017 at 8:27 PM, Jonathon Reinhart <
>>>>>>>>> jonathon.reinhart at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Yes, it should automatically do that.
>>>>>>>>>>
>>>>>>>>>> See this (merged) PR from one of my projects:
>>>>>>>>>> https://github.com/JonathonReinhart/scuba/pull/98
>>>>>>>>>>
>>>>>>>>>> Towards the bottom you'll see a "View Details" button.
>>>>>>>>>> Clicking that will expand a box showing the results of all the
>>>>>>>>>> "checks" that ran.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Dec 5, 2017 at 11:13 PM, Bill Deegan <
>>>>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Is there a way to get travis to post the results back into the
>>>>>>>>>>> pull request?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Scons-dev mailing list
>>>>>>>>>> Scons-dev at scons.org
>>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Scons-dev mailing list
>>>>>>>> Scons-dev at scons.org
>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-dev mailing list
>>>>>>> Scons-dev at scons.org
>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-dev mailing list
>>>>>> Scons-dev at scons.org
>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-dev mailing list
>>>>> Scons-dev at scons.org
>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-dev mailing list
>>>> Scons-dev at scons.org
>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-dev mailing list
>>> Scons-dev at scons.org
>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>
>>>
>>
>> _______________________________________________
>> Scons-dev mailing list
>> Scons-dev at scons.org
>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>
>>
>
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20171220/3e053a29/attachment-0001.html>

More information about the Scons-dev mailing list