[Scons-dev] SCons-like build system

Thu Jan 15 11:34:10 EST 2015

 Hi Jason,

*>> **I am interested in how you deal with target platforms and tool
section ( I see that you suggest that tool version selection exists).*
I will try to explain.

Aqualid tries to load all tools from following locations:
On Windows:
   X:\PythonXX\Lib\site-packages\aqualid\tools
   %USERPROFILE%\AppData\Roaming\Python\PythonXX\site-packages\aqualid\tools
   %USERPROFILE%\.config\aqualid\tools

 On Unix:
   /usr/lib/pythonX.Y/site-packages/aqualid/tools
   $PYTHONUSERBASE/lib/pythonX.Y/site-packages
   $HOME/.config/aqualid/tools

User may specify additional paths in defualt config file
(~/.config/aqualid/defualt.cfg) or via CLI option '-I'

An example of g++ tool:

@tool('c++', 'g++', 'gxx', 'cpp', 'cxx')
class ToolGcc( ToolCommonCpp ):

  @classmethod
  def   setup( cls, options, env ):
    # it usually searches for programs and sets programs' options
    # if tools can't be found then it raises NotImplementedError

  @classmethod
  def   options( cls ):
    # returns tool's specific options
    returns options

  def   __init__(self, options ):
    # intinitializes other tool's options after setup(..) successfully
finished

  #//-------------------------------------------------------//
  # tool's methods, which can be called from scripts

  def   Compile( self, options ):
    return GccCompiler( options )

  def   CompileResource( self, options ):
    return GccResCompiler( options )

  def   LinkStaticLibrary( self, options, target ):
    return GccArchiver( options, target )

  def   LinkSharedLibrary( self, options, target, def_file = None ):
    return GccLinker( options, target, shared = True )

  def   LinkProgram( self, options, target ):
    return GccLinker( options, target, shared = False )

Decorator @tool is used to set names for the tool.
Any name can used from scripts to get access to tool's methods.

Different tools may share names. For example MS Visual C++:

@tool('c++', 'msvcpp','msvc++', 'cpp', 'cxx')
class ToolMsVCpp( ToolMsvcCommon ):

Assume the following build script:
  cpp = Tool('c++')

In this line Aqualid calls *setup(..)* methods of each tool.
If no exceptions were raised then corresponding tool instance will be
returned to the build script.

Aqualid also checks that key properties do not conflict with user settings.
For example, if user set target_os to 'windows' but we have only gcc
compiler for Linux then this tool will be skipped.

Aqualid supports customization different environments for different
versions of a tool.
User may define special pre-setup methods for a tool.
For example below is pre-setup script for GCC on my system:
~/.config/aqualid/tools/setup_gcc.py

@toolSetup('g++', 'gxx', 'gcc')
def   setupGcc( options ):
  if options.cc_name.isSetNotTo( 'gcc' ):
    raise NotImplementedError()

  if options.target_os == 'windows':

    target_arch = options.target_arch

    if not target_arch.isSet() or target_arch == 'x86-64':
      options.gcc_prefix="x86_64-w64-mingw32-"

    elif target_arch == 'x86-32':
      options.gcc_prefix="i686-w64-mingw32-"

    else:
      raise NotImplementedError()

If I set target_arch to 'arm' via command line or in a build script then no
C++ tool will be found.
Pre-setup scripts can be global (for all users) or local (only for current
user).

The main purpose of pre-setup scripts to set OS environment (like correct
PATH),
But tool's setup method use the OS environment to find tool itself.
And Aqualid checks that key options don't conflict.

Tool selection logic:

0. Get a list of tools with the same name
1. Pick the next tool from the list
2. Get list of pre-setup methods for the tool
3. If exists pick the next pre-setup method from the list (2).
4. Run pre-setup method for a tool. if failed go to the step 3 or step 1
5.  Run setup method of the tool. if failed go to the step 3 or step 1
6. Compare tool's key options with user's settings. if there is a conflict
go to step 3 or step 1
7. Create the tool instance. if failed go to the step 3 or step 1
8. Return the tool

*>> **As the build gets larger the time to rebuild when one file changed
take much longer.*
I agree. Running Aqualid from the top directory could take much time, but
it supports sub-projects.
You may build only specific targets running Aqualid from a sub-project
directory.

*>> **I have number of other suggestion on what could be done in you DB to
store certain states to allow rebuild to get going very fast. Some other
these I have not done in Parts yet ( as I don’t have “easy” control over
the main execution logic )*
I'm very interested. Can you share you suggestions?

Thank you.
Best regards,
Constantine

On 01/14/15 17:42, Kenny, Jason L wrote:

 HI,

Looks nice. It will be nice to see more documentation on this. I am
interested in how you deal with target platforms and tool section ( I see
that you suggest that tool version selection exists).

Another test which would be useful for me is how well does this scale. You
show an example with lots of node in a given aql file. The issue I see with
this is that in my case I have 1000+ build (parts) files that need to be
loaded. The some of the total targets (objs,so|dll, etc) would be about
200K. One issue with this is that it takes time to load each file. What I
have seem with SCons, and looks like you have a similar issue, is that to
load a build file, means it has to be processed/executed by python. This
takes time as user tend to scan the disk to get the source files and python
code is executed. As the build gets larger the time to rebuild when one
file changed take much longer. Parts ( currently) does some work to avoid
loading build files to SCons to reduce the startup time to get a build
going. What would take Scons 2-3 minute to get started with a build, now
takes 3 to30 second ( depending on states…).

Just something to think about. I have number of other suggestion on what
could be done in you DB to store cetain states to allow rebuild to get
going very fast. Some other these I have not done in Parts yet ( as I don’t
have “easy” control over the main execution logic )

Jason

*From:* Scons-dev [mailto:scons-dev-bounces at scons.org
<scons-dev-bounces at scons.org>] *On Behalf Of *Constantine
*Sent:* Wednesday, January 14, 2015 1:38 AM
*To:* SCons developer list
*Subject:* Re: [Scons-dev] SCons-like build system

Hi Jason,

>> *What I would like to know as I did not see it in the example right off
is, how do you deal with lots of components.*

I guess you mean this example:
https://github.com/aqualid/aqualid/wiki/An-introduction-to-Aqualid#build-sub-projects

lib = Script('../lib/make.aql')['lib']

dll = tools.cpp.LinkSharedLibrary( 'tool_api.cpp', lib, target = 'toolapi',

                                   cpppath = lib.options.api_cpppath,

                                   api_cpppath = '.',

                                   cppdefines = 'MAKING_LIBRARY' )

We call function 'Script()' to build component 'lib'.
This function returns a dictionary of all local variables defined inside
the executed script. Then we get the 'lib' Node.

To build a shared library we use the lib's CPPPATH:
  cpppath = lib.options.api_cpppath,

Option 'api_cpppath' will have correct CPPPATH to use headers of this
library.

Inside component 'lib' we are free to change directories layout whenever we
want.

*>> **Also your benchmarks… is this 32-bit or 64-bit.*

I used 64-bit Linux, 2 cores CPU and available 1.3 GiB of RAM, no swap

Thank you.

Best regards,

Konstantin.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20150115/a9eb39cc/attachment.html>