[Scons-dev] Support for Tools via a pip package / python module in sys path

Wed Aug 2 13:03:30 EDT 2017

You might take a look at this page and install the listed doc packages and
see if any of the errors go away?

https://bitbucket.org/scons/scons/wiki/DeveloperGuide/Documentation

On Wed, Aug 2, 2017 at 2:26 AM, RW <garlicbready at googlemail.com> wrote:

> Thanks for the info that's quite useful
> it turns out I already had python-libxml2 installed but after installing
> python-libxslt1
> (apt-get install python-libxslt1) that seemed to resolve that problem
>
> I can now get as far as generating the user docs in html (which is good
> enough for what I need)
>
> For info there it's highlighted another problem further down the line when
> generating the pdf's. It's probably something I have or haven't got
> installed under Ubuntu mate but as long as I can get as far as html output
> that's good enough for me anyways
>
> There's quite a few of these
> ```
> +---------------------------------------------------------------------
> | In /home/ric/scons-temp/scons_patched/build/scons/engine/SCons/
> | Variables/PathVariable.py:
> | Import failed (but source code parsing was successful).
> |     Error: ImportError: No module named compat (line 43)
> |
>   [..............
> +---------------------------------------------------------------------
> | In /home/ric/scons-temp/scons_patched/build/scons/engine/SCons/
> | Warnings.py:
> | Import failed (but source code parsing was successful).
> |     Error: ImportError: No module named compat (line 43)
> |
>   [...................
> ```
>
> Followed by this at the end
> ```
> Warning: No information available for SCons.Util.UniqueList's base
>          collections.UserList
>   [..............................
> Warning: No information available for
>          SCons.Variables.ListVariable._ListVariable's base
>          collections.UserList
>   [......................................Traceback (most recent call
> last):
>   File "/usr/bin/epydoc", line 13, in <module>
>     cli()
>   File "/usr/lib/python2.7/dist-packages/epydoc/cli.py", line 971, in cli
>     main(options, names)
>   File "/usr/lib/python2.7/dist-packages/epydoc/cli.py", line 791, in main
>     write_latex(docindex, options, options.action)
>   File "/usr/lib/python2.7/dist-packages/epydoc/cli.py", line 870, in
> write_latex
>     latex_writer.write(options.target)
>   File "/usr/lib/python2.7/dist-packages/epydoc/docwriter/latex.py", line
> 199, in write
>     self._write(self.write_module, directory, filename, val_doc)
>   File "/usr/lib/python2.7/dist-packages/epydoc/docwriter/latex.py", line
> 217, in _write
>     write_func(f.write, *args)
>   File "/usr/lib/python2.7/dist-packages/epydoc/docwriter/latex.py", line
> 384, in write_module
>     self.write_class(out, var_doc.value)
>   File "/usr/lib/python2.7/dist-packages/epydoc/docwriter/latex.py", line
> 412, in write_class
>     out(self.base_tree(doc))
>   File "/usr/lib/python2.7/dist-packages/epydoc/docwriter/latex.py", line
> 504, in base_tree
>     width = self._find_tree_width(doc)+2
>   File "/usr/lib/python2.7/dist-packages/epydoc/docwriter/latex.py", line
> 538, in _find_tree_width
>     width = max(width, self._find_tree_width(base)+2)
>   File "/usr/lib/python2.7/dist-packages/epydoc/docwriter/latex.py", line
> 537, in _find_tree_width
>     for base in doc.bases:
> TypeError: iteration over non-sequence
> scons: *** [build/doc/scons-api/api.pdf] Error 1
> scons: building terminated because of errors.
> ```
>
> Many Thanks
> Richard
>
> On 1 August 2017 at 16:29, Bill Deegan <bill at baddogconsulting.com> wrote:
>
>> Once we merge your pull request I'll try building on my system.
>> I know there is occasionally some issue using lxml vs libxml2/libxslt.
>>
>> If you like I have a script to build python with libxml2 and libxslt.
>> Just change the version string at the top and it should build a python
>> which should work for the doc flow.
>>
>> https://github.com/bdbaddog/python_build_script
>>
>> -Bill
>>
>> On Tue, Aug 1, 2017 at 8:04 AM, RW <garlicbready at googlemail.com> wrote:
>>
>>> Just to follow on the below
>>> I've recently found I can do what I want to do without making any
>>> functional changes to scons
>>> but I put in a pull request for some changes to the user manual to make
>>> it clear on how to use toolpath since it doesn't seem to be documented all
>>> that much
>>> the trick I found was just to put sys.path into the toolpath argument so
>>> that it can pick up on tools installed via a package management system like
>>> pip
>>>
>>> https://bitbucket.org/scons/scons/pull-requests/493/updates-
>>> to-user-manual-tests-for-the-use/diff#comment-None
>>>
>>>
>>> I'm still having problems generating the docs under the head sources
>>> though
>>> I had to copy the xml files to a 2.5.1 tree just to build them into html
>>> to check how they would come out
>>>
>>> I had a root through the list of bugs but I don't think this one is
>>> listed yet
>>> Under the latest sources
>>>
>>> doc/user/tasks.xml is built to build/doc/generated/examples/t
>>> asks_ex1_1.xml
>>>
>>> tasks.xml is the same / no change between 2.5.1 and 3.0 latest
>>> but the resultant tasks_ex1_1.xml is different
>>>
>>> under 2.5.1 it comes out as
>>> ```
>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <screen xmlns="http://www.scons.org/dbxsd/v1.0" xmlns:xsi="
>>> http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
>>> http://www.scons.org/dbxsd/v1.0 http://www.scons.org/dbxsd/v1.
>>> 0/scons.xsd">% <userinput>scons -Q</userinput>
>>> cc -o app main.cpp
>>> cat < foo.bar2 > foo.cpp
>>> cc -o app2 main2.cpp foo.cpp
>>> cat < test.bar > test.h
>>> </screen>
>>> ```
>>>
>>> under 3.0 latest it comes out as
>>> ```
>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <screen xmlns="http://www.scons.org/dbxsd/v1.0" xmlns:xsi="
>>> http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
>>> http://www.scons.org/dbxsd/v1.0 http://www.scons.org/dbxsd/v1.
>>> 0/scons.xsd">% <userinput>scons -Q</userinput>
>>> [?1034hcat < test.bar > test.h
>>> cc -o app main.cpp
>>> cat < foo.bar2 > foo.cpp
>>> cc -o app2 main2.cpp foo.cpp
>>> </screen>
>>> ```
>>>
>>> I think the difference leads to an error of
>>> ```
>>> scons: Reading SConscript files ...
>>> scons: done reading SConscript files.
>>> scons: Building targets ...
>>> __xinclude_lxml(["scons_xi.xml"], ["main.xml"])
>>> __build_lxml(["scons_ex.xml"], ["scons_xi.xml"])
>>> scons: *** [scons_exi.xml] XSLTApplyError : Cannot resolve URI
>>> ../generated/examples/tasks_ex1_1.xml
>>> scons: building terminated because of errors.
>>> scons: *** [build/doc/user/index.html] Error 2
>>> scons: building terminated because of errors.
>>> ```
>>>
>>> Since the source file is the same I think it's something inbetween
>>> if you want me to file this as a bug then let me know
>>>
>>> Many Thanks
>>> Richard
>>>
>>> On 20 July 2017 at 21:21, RW <garlicbready at googlemail.com> wrote:
>>>
>>>> Hi Bill,
>>>>
>>>> # Current Search Mechanism
>>>>
>>>> From what I understand of the toolpath search mechanism it searches:
>>>> 1. within scons\engine\SCons\Tool for the tools bundled with scons
>>>> 2. A list of directories specified within the toolpath parameter
>>>> 3. A variable called DefaultToolpath within the scons source (by
>>>> default is empty)
>>>>
>>>> I checked experimentally and it looks as if the syspath is not included
>>>> in the search for tools so I don't think it can pick up on modules at
>>>> present when searching for tools.
>>>>
>>>> I've already added in the dot syntax in a prior commit to do something
>>>> different
>>>> The dot syntax is currently coded to search sub-directories within
>>>> toolpaths
>>>> so if you have a toolname like 'Test1.TestBuilder1'
>>>> and a list of toolpaths like
>>>> C:\Lib\toolpath1
>>>> C:\Lib\toolpath2
>>>> C:\Lib\toolpath3
>>>>
>>>> The places it's going to search will be something like
>>>> scons\engine\SCons\Tool\Test1\TestBuilder1
>>>> C:\Lib\toolpath1\Test1\TestBuilder1
>>>> C:\Lib\toolpath2\Test1\TestBuilder1
>>>> C:\Lib\toolpath3\Test1\TestBuilder1
>>>>
>>>> I could change that around so that the dot means something else
>>>> and use another character such as forward slash (pythonesque) as the
>>>> directory separator instead.
>>>> That might be a better idea
>>>>
>>>> # Overrinding
>>>>
>>>> For overriding, it's currently just currently adding to the toolpath
>>>> parameter behind the scenes
>>>> so the behaviour will be the same as that, I'm not sure how that works
>>>> currently
>>>> If toolpath overrides the inbuilt tools then this should work the same
>>>> way
>>>>
>>>>
>>>> # Suggested syntax
>>>>
>>>> There's a few different ways of changing the syntax around
>>>> To give some examples of what I'm thinking based on what you've said
>>>>
>>>> 1. With the below example I've change the dot to a forward slash within
>>>> the toolname
>>>> "TestBuilder1" - look for tool in root of toolpath directory
>>>> "ExampleDir1/TestBuilder1" - look for tool in sub-dir of toolpath
>>>> directory
>>>> "ExampleDir1/ExampleDir2/TestBuilder1" - look for tool in sub-dir of
>>>> toolpath directory
>>>>
>>>> 2. This example uses a # prefix in the toolname to specify to use a
>>>> python module as the base search directory (add to the toolpath)
>>>> "scons_toollib_example#TestBuilder1"
>>>> "scons_toollib_example.submod1#TestBuilder1"
>>>> "scons_toollib_example#ExampleDir1/TestBuilder1"
>>>> "scons_toollib_example#ExampleDir1/ExampleDir2/TestBuilder1"
>>>>
>>>> 3. We could also keep the toolmods option to avoid using a prefix as
>>>> well
>>>> all that will do is find the directory where the module is located and
>>>> add that directory to the toolpath list
>>>>
>>>>
>>>> Any thoughts?
>>>>
>>>>
>>>> Many Thanks
>>>> Richard
>>>>
>>>>
>>>> On 20 July 2017 at 20:05, Bill Deegan <bill at baddogconsulting.com>
>>>> wrote:
>>>>
>>>>> Wouldn't this work and not require any changes?
>>>>> Or if it doesn't already work, wouldn't that be easier syntax? no
>>>>> additional arguments, if it's got a '.' in it then you know it's part of a
>>>>> package?
>>>>>
>>>>> toollist = ['scons_toollib_example.TestBuilder1']
>>>>> env = Environment(ENV = os.environ, tools = toollist)
>>>>> env.TestBuilder1('testdest1.txt', 'testsrc1.txt')
>>>>>
>>>>> How would you handle overloaded tool names in a module?
>>>>> For example
>>>>> scons_toollib_example has a gcc tool?
>>>>>
>>>>> -Bill
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 20, 2017 at 2:53 PM, RW via Scons-dev <scons-dev at scons.org
>>>>> > wrote:
>>>>>
>>>>>> Hi,
>>>>>> One of the features I'm interested in is the ability to have tools
>>>>>> loaded externally via a package installed via pip.
>>>>>> This way if you wanted to, you could ether install tools as a
>>>>>> separate repo via pip, or move some of the existing tools into separate
>>>>>> repo's
>>>>>>
>>>>>> I think I've come up with a way to do it which doesn't involve much
>>>>>> code.
>>>>>> But before I add in any tests or additions to the man page / user
>>>>>> docs I wanted to check if this is an okay way to implement this
>>>>>>
>>>>>> I've put the commit here:
>>>>>> https://bitbucket.org/grbd/scons/commits/af3ea9f2d4468479f7f
>>>>>> 7be73afd455b0f2067409
>>>>>>
>>>>>> With this method there's an additional option on the environment
>>>>>> constructor called toolmods
>>>>>>
>>>>>> ```
>>>>>> toollist = ['TestBuilder1']
>>>>>> env = Environment(ENV = os.environ, tools = toollist, toolmods =
>>>>>> ['scons_toollib_example'])
>>>>>> env.TestBuilder1('testdest1.txt', 'testsrc1.txt')
>>>>>> ```
>>>>>>
>>>>>> This option looks up the directory where the module is located in the
>>>>>> sys path and adds it to the toolpath
>>>>>> (e.g. C:\Python27\Lib\site-packages\scons_toollib_example)
>>>>>>
>>>>>> A side effect is that the module file gets imported at the same time
>>>>>> (scons_toollib_example\__init__.py is run)
>>>>>> I just do this to get the full path to the module when adding to the
>>>>>> toolpath, although you could probably use that to modify scons or do
>>>>>> something else when the tool is referenced if you wanted to.
>>>>>>
>>>>>> Is this implementation is a good idea? if so I'll look into adding
>>>>>> some tests and user doc entries for it before doing a pull request
>>>>>>
>>>>>> Many Thanks
>>>>>> Richard
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-dev mailing list
>>>>>> Scons-dev at scons.org
>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20170802/2ad22181/attachment-0001.html>