[Scons-dev] Migrating issues from Tigris to GitHub
Andrew Featherstone
andrew.featherstone at cantab.net
Fri Jan 26 18:21:06 EST 2018
The design intent is that the script overwrites existing issues, and
assumes that PRs are the first n contiguous issue/PRs in the project. We
didn't stop more PRs being recreated, so now issue/pr 1-36 exist, and
additionally there are PRs at, for example
https://github.com/SCons/scons/pull/3044. The script as is will try and
write Tigris issue 1 to Github as issue 37, 2 to 38, and so on. Now we've
got PRs after the block of issues, the problem becomes more complicated.
Some ideas off the top of my head:
1. These high numbered PRs get trashed, and we accept that has happened.
2. tigris2github.py is modified to handle this situation. That's not very
appealing to me at this point, as it's more effort for some single-use code.
3. Err... something else?
Andrew
The script overwrites existing issues. Because the PRs exist already, we
want to edit the existing issues, and create an additional few issues.
On 26 January 2018 at 22:46, Bill Deegan <bill at baddogconsulting.com> wrote:
> Is there any way to enumerate all the existing PR/Issues and overwrite
> only the issues? (and/or items without corresponding changelists)?
>
> That'd allow us to re-use the existing 3k or so aborted migrated issues.
>
> -Bill
>
> On Fri, Jan 26, 2018 at 1:34 PM, Andrew Featherstone <
> andrew.featherstone at cantab.net> wrote:
>
>> Hi All,
>>
>> I've addressed all the issues that I'm aware of with the importing of
>> issues into GitHub. I don't kniow if there's anything we can do to fix the
>> existing changes to the merge requests, but AFAICT we're ready to give this
>> another go.
>>
>> Regards,
>> Andrew
>>
>> On 3 January 2018 at 18:37, Bill Deegan <bill at baddogconsulting.com>
>> wrote:
>>
>>> Crap.
>>> It looks like the migration script overwrote existing pull requests.
>>> They are all gone and replaced with migrated bugs.
>>>
>>> Let me see if I can get ahold of github and get them restored..
>>>
>>> -Bill
>>>
>>> On Tue, Jan 2, 2018 at 1:39 PM, Bill Deegan <bill at baddogconsulting.com>
>>> wrote:
>>>
>>>> o.k. I'm going to disable the issues and remove the lock on the
>>>> scons/scons repo for now. I'll fork out another for use trying out any
>>>> fixes before rolling out to the main repo.
>>>> -Bill
>>>>
>>>> On Tue, Jan 2, 2018 at 12:57 PM, Andrew Featherstone <
>>>> andrew.featherstone at cantab.net> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> Yes you're right, it seems that "every pull request is an issue, but
>>>>> not every issue is a pull request." as noted at
>>>>> https://developer.github.com/v3/issues/#list-issues. I've raised a
>>>>> couple of issues at https://github.com/SCons/tigris-to-github/issues
>>>>> to submit my fixes against. I'm not sure at this point how to fix the
>>>>> comments in the pull requests, but one step at a time.
>>>>>
>>>>> Andrew
>>>>>
>>>>> On 2 January 2018 at 19:11, Daniel Moody <dmoody256 at gmail.com> wrote:
>>>>>
>>>>>> Yes the 36 Pull Request we had got overwritten with the bug
>>>>>> information for 1-36.
>>>>>>
>>>>>> If possible i think it would be best to move the pull request to the
>>>>>> newest numbers so all the cross references to issue numbers can stay the
>>>>>> same.
>>>>>>
>>>>>> Also another issue I saw was that # sign will link to other issues,
>>>>>> which in some of the text of our issue, that was not intended. If possible
>>>>>> you can replace '#[number]' with 'choice [number]' in the text when they
>>>>>> get reprocessed?
>>>>>>
>>>>>> I read about choice here:
>>>>>> https://github.com/github/markup/issues/303
>>>>>>
>>>>>> On Tue, Jan 2, 2018 at 12:03 PM, Bill Deegan <
>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>
>>>>>>> Andrew,
>>>>>>>
>>>>>>> Also.. there's 36 bugs which didn't get migrated.
>>>>>>> I'm guessing they're the ones which yielded:
>>>>>>> 422 {'message': 'Validation Failed', 'errors': [],
>>>>>>> 'documentation_url': 'https://developer.github.com/
>>>>>>> v3/issues/#edit-an-issue'}
>>>>>>>
>>>>>>> ...
>>>>>>> 8
>>>>>>> 422 {'message': 'Validation Failed', 'errors': [],
>>>>>>> 'documentation_url': 'https://developer.github.com/
>>>>>>> v3/issues/#edit-an-issue'}
>>>>>>> 422 {'message': 'Validation Failed', 'errors': [],
>>>>>>> 'documentation_url': 'https://developer.github.com/
>>>>>>> v3/issues/#edit-an-issue'}
>>>>>>> 422 {'message': 'Validation Failed', 'errors': [],
>>>>>>> 'documentation_url': 'https://developer.github.com/
>>>>>>> v3/issues/#edit-an-issue'}
>>>>>>> 422 {'message': 'Validation Failed', 'errors': [],
>>>>>>> 'documentation_url': 'https://developer.github.com/
>>>>>>> v3/issues/#edit-an-issue'}
>>>>>>> 422 {'message': 'Validation Failed', 'errors': [],
>>>>>>> 'documentation_url': 'https://developer.github.com/
>>>>>>> v3/issues/#edit-an-issue'}
>>>>>>> 422 {'message': 'Validation Failed', 'errors': [],
>>>>>>> 'documentation_url': 'https://developer.github.com/
>>>>>>> v3/issues/#edit-an-issue'}
>>>>>>> 422 {'message': 'Validation Failed', 'errors': [],
>>>>>>> 'documentation_url': 'https://developer.github.com/
>>>>>>> v3/issues/#edit-an-issue'}
>>>>>>> 422 {'message': 'Validation Failed', 'errors': [],
>>>>>>> 'documentation_url': 'https://developer.github.com/
>>>>>>> v3/issues/#edit-an-issue'}
>>>>>>> 422 {'message': 'Validation Failed', 'errors': [],
>>>>>>> 'documentation_url': 'https://developer.github.com/
>>>>>>> v3/issues/#edit-an-issue'}
>>>>>>> 422 {'message': 'Validation Failed', 'errors': [],
>>>>>>> 'documentation_url': 'https://developer.github.com/
>>>>>>> v3/issues/#edit-an-issue'}
>>>>>>> 9
>>>>>>> ...
>>>>>>>
>>>>>>> So I'm guessing
>>>>>>> Can you take a look at that?
>>>>>>> For now I'm going to disable issues on scons/scons and then I'll try
>>>>>>> again once you're pull request is ready.
>>>>>>>
>>>>>>> Also looks like the lowest issue # created is 37 (notable that the
>>>>>>> highest pull request # is 36).
>>>>>>> It looks like they share the same "namespace" for lack of a better
>>>>>>> term.
>>>>>>> So cross references to other bugs will be off..
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Bill
>>>>>>>
>>>>>>> On Tue, Jan 2, 2018 at 1:32 AM, Andrew Featherstone <
>>>>>>> andrew.featherstone at cantab.net> wrote:
>>>>>>>
>>>>>>>> Hi Bill,
>>>>>>>>
>>>>>>>> I'm glad you persevered. No I've not put any effort into making
>>>>>>>> this work with Python 2. There's some crude retry mechanisms in the
>>>>>>>> implementation. By the look of https://github.com/scons/scons
>>>>>>>> /issues this is still trundling on, as the issue count keeps going
>>>>>>>> up when I refresh the browser.
>>>>>>>>
>>>>>>>> Unfortunately, I've spotted a meta-issue, which can be seen at
>>>>>>>> https://github.com/SCons/scons/issues/1345, for example. There's
>>>>>>>> an attachment for this issue that should be at
>>>>>>>> https://github.com/scons/tigris-issue-attachments/blob/maste
>>>>>>>> r/45/SCons-PkgCheckModules.diff, but is pointing to
>>>>>>>> https://github.com/bdbaddog/scons/tigris-issue-attachments/b
>>>>>>>> lob/master/45/SCons-PkgCheckModules.diff instead.
>>>>>>>>
>>>>>>>> I think the sensible thing is for me to raise an issue against
>>>>>>>> https://github.com/SCons/tigris-to-github and submit a fix via a
>>>>>>>> pull request. The import process will merrily overwrite existing issues so
>>>>>>>> that's not a problem, but we will need to run the script again.
>>>>>>>>
>>>>>>>> Sorry, all.
>>>>>>>>
>>>>>>>> Andrew
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2 January 2018 at 04:48, Bill Deegan <bill at baddogconsulting.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Got past that. Requires python 3 and a few modules.
>>>>>>>>> Running now but just got the folllowing:
>>>>>>>>> GitHub repository for issues: scons/scons
>>>>>>>>> GitHub repository for attachments: scons/tigris-issue-attachments
>>>>>>>>> 1
>>>>>>>>> 2
>>>>>>>>> 3
>>>>>>>>> 4
>>>>>>>>> 5
>>>>>>>>> 6
>>>>>>>>> 7
>>>>>>>>> 8
>>>>>>>>> 422 {'message': 'Validation Failed', 'errors': [],
>>>>>>>>> 'documentation_url': 'https://developer.github.com/
>>>>>>>>> v3/issues/#edit-an-issue'}
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jan 1, 2018 at 8:05 PM, Bill Deegan <
>>>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>>>
>>>>>>>>>> No go.
>>>>>>>>>>
>>>>>>>>>> python tigris2github.py
>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>> File "tigris2github.py", line 11, in <module>
>>>>>>>>>> from github import Github, UnknownObjectException
>>>>>>>>>> ImportError: No module named github
>>>>>>>>>>
>>>>>>>>>> pip search github yields a long list.
>>>>>>>>>>
>>>>>>>>>> Can you add a requirements.txt with a list of required packages
>>>>>>>>>> to the repo and send a pull request?
>>>>>>>>>>
>>>>>>>>>> On Fri, Dec 29, 2017 at 3:37 AM, Andrew Featherstone <
>>>>>>>>>> andrew.featherstone at cantab.net> wrote:
>>>>>>>>>>
>>>>>>>>>>> I've just check this morning that it's able to work with an
>>>>>>>>>>> organization's repos and not just an individual's. If you don't want to
>>>>>>>>>>> solicit feedback on the users' mailing list, then the next step is to
>>>>>>>>>>> perform the migration. So, to move things forward:
>>>>>>>>>>>
>>>>>>>>>>> 1. Clone my repo.
>>>>>>>>>>> 2. Call python tigris2github.py and answer the questions.
>>>>>>>>>>>
>>>>>>>>>>> So it'd be something like this:
>>>>>>>>>>>
>>>>>>>>>>> > python .\tigris2github.py
>>>>>>>>>>> GitHub username: ajf58
>>>>>>>>>>> GitHub Password:
>>>>>>>>>>> GitHub repository for issues: scons/scons
>>>>>>>>>>> GitHub repository for attachments: scons/tigris-issue-attachments
>>>>>>>>>>>
>>>>>>>>>>> As mentioned previously, it'll take a couple of hours to churn
>>>>>>>>>>> through everything due to the rate limiting of GitHub.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Andrew
>>>>>>>>>>>
>>>>>>>>>>> On 28 December 2017 at 22:12, Bill Deegan <
>>>>>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Yes. Sorry I didn't see the link in the existing markup.
>>>>>>>>>>>> That looks good to me.
>>>>>>>>>>>> So to move forward (migrate bugs to scons/scons and attachments
>>>>>>>>>>>> to scons/tigris-issue-attachements/ ) what do I need to do?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Bill
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Dec 28, 2017 at 12:56 PM, Andrew Featherstone <
>>>>>>>>>>>> andrew.featherstone at cantab.net> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Bill,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm not sure I follow you. The issue numbers and the Tigris
>>>>>>>>>>>>> 'attachment IDs' (used as the folder names) aren't the same value. Where
>>>>>>>>>>>>> issues have attachments they are reffered to in the issue by name, and the
>>>>>>>>>>>>> URL is included in the Markdown. As an example, see
>>>>>>>>>>>>> https://github.com/ajf58/tigris-sandbox/issues/6, which has
>>>>>>>>>>>>> an attachment found at https://github.com/ajf58/tigri
>>>>>>>>>>>>> s-issue-attachments/blob/master/190/bug6.zip.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Have I understood your question correctly?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 28 December 2017 at 20:33, Bill Deegan <
>>>>>>>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you add to the github bug a link to the github attachment
>>>>>>>>>>>>>> repo where appropriate?
>>>>>>>>>>>>>> (so you don't have to go look for the repo and then the issue
>>>>>>>>>>>>>> number folder.)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Dec 28, 2017 at 11:51 AM, Daniel Moody <
>>>>>>>>>>>>>> dmoody256 at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think it's awesome, looks great fully support moving
>>>>>>>>>>>>>>> forward with this! Good work!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Dec 28, 2017 2:36 PM, "Andrew Featherstone" <
>>>>>>>>>>>>>>> andrew.featherstone at cantab.net> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Any comments or feedback on this?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 17 December 2017 at 18:58, Bill Deegan <
>>>>>>>>>>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Great work!
>>>>>>>>>>>>>>>>> I'll try and set aside some time to review it this week.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -Bill
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sat, Dec 16, 2017 at 6:44 PM, Andrew Featherstone <
>>>>>>>>>>>>>>>>> andrew.featherstone at cantab.net> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I've been working on how we can migrate the Tigris issues
>>>>>>>>>>>>>>>>>> to GitHub. You can see the output of this at
>>>>>>>>>>>>>>>>>> https://github.com/ajf58/tigris-sandbox/issues, To
>>>>>>>>>>>>>>>>>> ensure that we're not relying on issue attachments being hosted at Tigris,
>>>>>>>>>>>>>>>>>> which looks to be deserted by its owners (no tweets since 2013
>>>>>>>>>>>>>>>>>> https://twitter.com/tigrisdotorg?lang=en, for example),
>>>>>>>>>>>>>>>>>> the issue attachments are also migrated to GitHub (
>>>>>>>>>>>>>>>>>> https://github.com/ajf58/tigris-issue-attachments). The
>>>>>>>>>>>>>>>>>> source code used for doing this can be found at
>>>>>>>>>>>>>>>>>> https://github.com/ajf58/tigris-to-github.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I think the next steps are:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 1. Members of this list review the format of the
>>>>>>>>>>>>>>>>>> migrated issues and offer feedback.
>>>>>>>>>>>>>>>>>> 2. Act on feedback.
>>>>>>>>>>>>>>>>>> 3. Get feedback on the scons-user mailing list.
>>>>>>>>>>>>>>>>>> 4. Migrate issues to https://github.com/SCons/scons
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The migration process takes ~2 hours currently. This is
>>>>>>>>>>>>>>>>>> largely due to the rate limits GitHub imposees on their REST API. GitHub
>>>>>>>>>>>>>>>>>> allows temporary restrictions to be imposed on the repo, so we can do that
>>>>>>>>>>>>>>>>>> while the migration takes place.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>> Scons-dev mailing list
>>>>>>>>>>>>>>>>>> Scons-dev at scons.org
>>>>>>>>>>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> Scons-dev mailing list
>>>>>>>>>>>>>>>>> Scons-dev at scons.org
>>>>>>>>>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> Scons-dev mailing list
>>>>>>>>>>>>>>>> Scons-dev at scons.org
>>>>>>>>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> Scons-dev mailing list
>>>>>>>>>>>>>>> Scons-dev at scons.org
>>>>>>>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Scons-dev mailing list
>>>>>>>>>>>>>> Scons-dev at scons.org
>>>>>>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Scons-dev mailing list
>>>>>>>>>>>>> Scons-dev at scons.org
>>>>>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Scons-dev mailing list
>>>>>>>>>>>> Scons-dev at scons.org
>>>>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Scons-dev mailing list
>>>>>>>>>>> Scons-dev at scons.org
>>>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Scons-dev mailing list
>>>>>>>>> Scons-dev at scons.org
>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Scons-dev mailing list
>>>>>>>> Scons-dev at scons.org
>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-dev mailing list
>>>>>>> Scons-dev at scons.org
>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-dev mailing list
>>>>>> Scons-dev at scons.org
>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-dev mailing list
>>>>> Scons-dev at scons.org
>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-dev mailing list
>>> Scons-dev at scons.org
>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>
>>>
>>
>> _______________________________________________
>> Scons-dev mailing list
>> Scons-dev at scons.org
>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>
>>
>
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20180126/cf9936f3/attachment-0001.html>
More information about the Scons-dev
mailing list