[Scons-dev] Some thoughts on SCons based on my Parts works

[Kenny, Jason L]

Hi guys,

Well this year I will not be able to work on Parts as much as my projects at work are changing. Part of this comes from that with the latest drop ( while not what I would like to call 1.0) I was able to take a 100-200K node build that would take about 2-3 hour to do a full build on a box, with about 2-3 GB of memory for the SCons and was able to reduce the incremental start up time for the build from a number such as 600 second to a number that is much more reasonable of about 7-20 seconds (disk load variation seems to be the biggest factor for the number range), and viewed by many as good enough.

The raw SCons times to start up and start building would be between 120-400 seconds. This was because SCons would have to read all the data files ( which was about 1000 different files), make a big tree ( about 150K nodes), and would have to allocate ( based on current code) about 2.5-3.6 GB. This forced us as some point to move to a 64-bit python as the product grew large enough, we hit a 32bit 2GB barrier that is common on most system. This was not a major deal as most systems that are not tablets or a cell phone are a 64-bit OS anyways, and one has to cross build for tablets or Phone OS today anyways on a "real" computer as tablets and phones are for consuming not creation. The

The latest code in Parts as I have it at the moment. Was able to take most of the rebuild cases (ie the one in which you change a source file or removed a target file on disk), and was able to get the startup time down to 20 second or less ( most cases it was 8 seconds). Since Parts would have SCons load less stuff, I was also able to keep usage memory down in the 1 to 1.6 GB range. I believe this shows a great improvement and that we can get SCons to work much better.

Some of the memory hogs in SCons that we seem to know about would be the Node objects. Some other ones I have seen are the internal objects used for the subst engine. From what I can tell the internal objects used to help with holding string objects in special ways is why a lot of memory gets used and seems to be part of the reason for memory growth during the build.

Another oddity I found when I was looking at how to use the DB better. I was found that it was easier to store a copy of about everything in the build info signature object in my own Parts cache files. One of my stretch goal was to bi-pass SCons load logic completely on rebuild, but use information about the build in the cache to allow me to get building much faster. Not to go into the details here on what would need to happen, but as I added this extra information into my own cached and stored these objects in a big pickle file, I notices that my files are a lot smaller for some reason. All the data I had stored for the build ( which includes for our build information about where we checked out a given component from, node relationships including Alias and Value nodes, component relationships, etc..) was 30.2 MB. The SCons DB was 129.6MB. I am not sure what SCons is doing to require the information to take more space on disk, but I did find some odd code in SCons that messes with the pickle information before it is stored on disk. I wonder if this has anything to do with it. I believe that given better usage of the information about the build in a DB/cache file. We can improve start up time of SCons for very large builds by even a few more second as well as cut out a lot of the memory usage we currently have.

Going forward I hope to make some more fixes and additions to Parts ( mostly clean up work, and keeping the tools I override in Parts up to date with new version of Microsoft and Intel compilers) , and start a rewrite/cleanup of SCons core code to merge what I learned ( I am currently looking at the subst engine). I do believe there is a lot of room for improvement in SCons engine.

Feel free to ask me any questions. I would hope that Parts, and what it can do will help improve SCons going forward.

Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://two.pairlist.net/pipermail/scons-dev/attachments/20130107/1c329f48/attachment.html>