[Scons-dev] Schedule for redesign?

[Kenny, Jason L]

Sounds good... Technically I can open up what I have for the Scons nodes rewrite as well.. I just trying to get the interfaces fixed and clear and clean.

I have with Parts two node trees. One for the SCons nodes, and one for the component nodes. Ideally I would want to merge these together as one, with one node manager, not many ( like we have in Scons). We need to get Symlinks in the node tree for file based nodes. I have it working in Parts (yaaa...) It works well, and allows us to define a gnu style symlink setup in a cross platform way. I believe one thing I found with this is that we only need symlink for file. Directory links are a can of worms.

For subst() stuff it is the memory size. There is a set of special objects that are buggy and wasteful in memory. There is very little caching going on here. This can be improved.

I should add from what I learned in what I have for Scons speed up with Parts there are few things that can be done in SCons. So some thoughts here to think about.

I store in my parts cache for a given cache key the same info SCons stores, plus a lot more. For some reason the size I was seeing stored is less that the master scons db file. I looked at it a little and honestly got lost in some code that tries to mess with the pickled data.


>From that one area of speed in SCons is that all scons files are executed when they are read. From a reclusive makefile point of view this might be needed, but when you have a more component load model ( as in Parts) this is a BIG waste if we don't need to build that component. Currently Parts tries to figure out what it needs to load, and loads only that. This gives a huge speed up. When mixed in to some no depends/unsafe loading logic it is really fast ( and useful is you are only trying to rebuild you top-level component... no need to load all the dependent components as these are already built and we know we did not modify them, or have the system to check to see if they are) This is the main reason why I would like to have the ability have a new format for Parts as this would allow for delayed execution.. ie something like ( very simple example):


PartName("foo")
PartVerson("1.2" 
@config
def config(env):
     ... setup.. test stuff..

@emit:
Def emit(env)
    Files=env.Glob(*.c)
    Env.SharedLibrary("$PARTNAME_$PARTVERSION",Files)


In this case the need to configure a component or call emit ( lack of better name) can be delayed unit we need to do it. For example here the Glob() call is slow, as well as the SharedLibrary() call. The setting a name or adding a CPPFLAG is fast.

I also believe that we can do a lot more with the DB/cache files. Given you skip the python only builder issues. Most actions are command line ones. These actions can be stored in the DB. Doing a set of Checks like Parts.. Scons can easily detected ( in less than a second) if any files that define context of what we would build. Given that is not changed, we load information about what we last built instead of load it all from files and executing code to make a tree of what to do again and instead just start building the nodes from the stored action string without loading any build files. This is ideally the most common case for developers as they are modifying code, not adding/removing file or modifying build files. ( even in the case of adding removing files, we could do a lot to reduce what is loaded)

I should say that Given some tweaks, I do believe SCons can do a correct safe build much faster, vs doing trick to get speed up but with risk of an broken build.

I should add that I have seen a lot of SCons files are more scripts. They call execute() to build stuff vs letting SCons do what it does best. These seems to be happen because the developer does not understand what SCons does. They still used SCons because the Environment object provides and easy to under model to control different tools chains and cross build in a single pass. This notion seems to be much hard to deal with in other many other systems. 

I am sure I have more stuff/experience to add.. but I need to get some work done :-)

Jason


-----Original Message-----
From: scons-dev-bounces at scons.org [mailto:scons-dev-bounces at scons.org] On Behalf Of Dirk Bächle
Sent: Monday, October 28, 2013 1:45 PM
To: SCons developer list
Subject: Re: [Scons-dev] Schedule for redesign?

Hi Jason,

and thanks for chiming in.

On 28.10.2013 17:07, Kenny, Jason L wrote:

> I hope I would not be a wrench in engine :-)

>

> Honestly The issue that this is work and I have to do this at home is 

> the main reason I have not pushed anything at this point. ( And having 

> twin 3 years-old means I don't have the go power at the end of the 

> day...)

>

> I would agree with what is being stated. I would go a bit farther however.


I understand that you have done some work on redesigning/refactoring parts of SCons, and you certainly don't want to lose it for nothing. So, I'd like to make the following suggestion: I'll factor out some attributes of the Node class into their own module and commit these changes (together with the other rewritings I mentioned) to a named branch in my personal "experimental" SCons fork.
Then you can have a look at it and check whether you'd be able to base your further work on that. If not, you can tell me what should be changed such that you can continue properly with your work. If I get your "okay", the changes find their way into the SCons core and you can take it from there whenever you find the time. How does that sound?


> I would redesign the Node API. But also redo the executioner logic. Ideally we want I have learned is that the task logic needs to be updated. Greg Noel was on the correct path with TNG in that we want to execute "builders" not nodes. We also need the API ( internal) to decouple the relationships with the tasks and the nodes.

>

> I would also look at fixing up the subst engine. I found that this is another area of memory waste at the moment ( and not in the good way). This might be more of an issue with Parts as I currently use the subst engine to share data between different "parts"/ components. Until I do a new part file format and or a "continuous loader" I have no way to pass data correctly.


During my investigations I tried to speedup subst() by caching some pre-expanded strings, but it didn't work out. My current impression is that you either have to compile it (including optimizations by hand, simply running Cython over it is not enough!) or reduce functionality (like cutting down the support for callables) to get it faster.

But don't let me stop you from trying... ;)

Best regards,

Dirk

_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev