[Scons-dev] Some thoughts

[Kenny, Jason L]

Hi

I had some thought from what I have done in Parts about Scons.

Doing the work I have done with Parts and SCons I have found a various set of thoughts.

What do we build?
SCons has changed the way it handle stuff being out of date. When I started SCons would do logic with the MD5 check that I really liked. I could change a foo.c file by adding a comment. SCons would rebuild this file and when it look at the linking it would see the foo.o was that same as it was before and not link a new SO file. This was correct as only the differences would be built. At this moment Scons assume all dependent file are out of date if a child is out of date and rebuilds them implicitly. This mean it build more than it should. I am not sure why we want this. I pretty sure anything that needs this to build, is not defining what relationship to the build system correctly.

Speed issues at startup!
Scons as a build system is nice as in we can say I want to do something and SCons does what we want normally. We want a build system to only build what it has to, nothing else.

SCons is slow with large builds. However given the correctness, I don't know any build system that is while making it as easy to build large projects ( ie build 100K+ files across 1000 different Components .. ie Parts for me). This is not say there is a lot we can improve on. One of the thing I have tried to make better ( currently finishing off a massive improvement to this thinking)  is that we can do a quick set of checks on files that define content to see if the node it would define might be out of date. This means that the node information about the node SCons stored is out of date. If these files are not out of date in theory we could have SCons check the files on disk and build anything out of date without loading a single SConstruct file. This is simpler said that done, as many actions are Python functions, which are not easy to store like some CLI actions string. However in theory this could be done. What I do know from what I do in Parts is that we can do a check of files that define content for nodes based on the target to see if what set of files need to be loaded again, mix that with a check to see what node are out of date we can get a set of file that need to be loaded to build what is out of date. This logic shows a very large improvement in startup time in Scons.

Parts defines an idea of a Sconstruct being a file like a master project or solution file, while a Part file is like project file. Scons by definition tries to be a Make replacement, so it out of the box supports a recursive make idea. I believe what I stated above about reduce loading can be applied in both cases.

Going forward, better integration of these checks with task master means the SCons could start build section of the tree before it reads in all the files, as at certain points we know that there is nothing we can add to build a set of node. This is easier said in Parts as we have nice Partitions of what to build.

Dynamic Node issue.

Given what was said about Java, I have I believe a similar but different issue with SCons. There are often issue with wanting to glob for files to do something with. However with what I tried to do with Pattern in Parts or using Scons with Glob, they both have one common fault. That is they produce a set of node based on what exists ( or known) at that point of time. This mean we get issue with saying copy all the files in directory X of a certain pattern Y to a certain location Z if we don't have the order correct. This is because unless all the files are defined to SCons before we do the Glob() we get an incomplete set of files returned to pass to a different builder. This become an issue in large build where it is hard to make sure the order of everything is defined in correct order.

What I think we need is notion of filler/dynamic node that says something like "there are some node here" but we can fill them in till all dependents of the node are done. This also means the user can see not what these node will explain to are during the "reading" of the Sconscript files.

Other build systems

My last thought is dealing with other build systems. One issue I get a lot at work, is how to do I take this configure/make file hand have it build correctly under SCons. My first answer is why not user Parts/SCons as the end result is normally 10 line of code or less and it will build different system easily. However I normally get a "Well, we don't know what this does to build the code, we assume that since they used autoconf it must be incredibly complex, and any translation to anything else will build it incorrectly. On the windows side I tend to get why can't we make MSBUILD files outputs, and or read them in to get what should build ( and maybe override certain build settings). it seems to me that it would be useful to think about builders that can process files in other build systems so it can easily be added to a SCons build to help with migration. Or to help with developers that use IDE to write the code.

Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://two.pairlist.net/pipermail/scons-dev/attachments/20120904/f7ef253b/attachment-0001.html>