[Scons-dev] How to traverse the graph after files are read

[Kenny, Jason L]

Let me give you an understanding of this as I understand it based on my work and talks I had with Steve Knight...
Starting with the basics.. some of this you already know..

Scons has nodes. These node are used to represent items. We have 
*Value nodes that can holds some value like a string, list, some python object, 
*Alias nodes that in general are used to represent a set of other nodes ( but you can give an alias an action to run ). This does not have a state that is stored in SCons and since different target can cause alias to have a different set of children, it being out of date is based on the current read state of the build files.
*file nodes which are nodes that are based on what is on disk. In this set we have currently:
** Dir node which represent some directory on disk
** File node which is some file
** Entry node which is either a file or directory, but is assumed to be unknown at the current moment. It is morphed into a Dir or File node later. In general if it is not a directory Scons assumes it is a file. Sometimes this is wrong and will lead to an error about looking up node X as a file but it is really a directory error message. In this cases you normally work around it by forcing it to Dir node when you provide the item to the builder.
** in Parts we have added a FileSymbolicLink node object to help deal with symlinks on disk.

Given this any node can have a builder associated to it. Given the node does not exist SCons will call the builder to make it ( directory can be funny here.. as I noted a bug I have found to Dirk with directory nodes and copy logic). Any node that does not have a builder and does not exist leads to a "Don't know how to build XYZ" message from SCons. Normally a node without a builder is source/dependent node that is added as an explicit dependency via a builder call, Depends() call or via a scanner.

The FS object only holds file based nodes ( Dir, File, Entry and FileSymbolicLink [given Parts]). When you iterate the FS object you can generally view it as a virtual file system. When I had talked to Steve knight years ago, he told me that he had hoped to make the nodes a portable way to process files independent of the Python file API. Currently SCons is not there, but much of the plumbing for this exists and is used internally. Anyways you can iterate the FS object as a virtual file system and see that directory X contain a set of files and or directories or entry objects. These items may or may not exists on disk yet. This is allows Glob to return object that don't exist on disk yet as before the glob call happened something defined a build action that was to output some file based node. Call to add a node after a Glob call means Globs does not see it as a dependent, or least not until it exists on disk from a previous build run. Whenever you make a node such as File(readme.txt) scons will try to lookup this code in the FS object and return the existing node that represents this file if it already exists, if it does not exist it make a new node adds it to the FS object in the correct place and returns the reference to that node. ( given a variant directory, etc are being used this may be many nodes for the different locations)

To get the tree you want all you have to do is grab the target list ie... SCons.Script.BUILD_TARGETS and for each item in the list, turn it into a node ( as it might be a string still) and then you start to iterate it children. As pointed out the main code for this in SCons.Script.Main.py. you can look at TreePrinter as a starting point and search for the code that uses it.

Hope this helps

Jason



As a side note.. the main point of the Gregs taskmaster TNG was to change the logic in SCons to make a builder tree based on the node tree and use the builder tree to build node as this tree was much smaller and removes lots of wasted iteration we have when iterating node directly. This is really seen in cases such as package builders that depend on children as SCons has to iterate the children node, vs just understand that all the builders for the children are up to date ( which is a difference of a few builder objects vs ideally 1000s...  100,000 of node objects.


-----Original Message-----
From: Scons-dev [mailto:scons-dev-bounces at scons.org] On Behalf Of anatoly techtonik
Sent: Wednesday, May 20, 2015 9:25 AM
To: SCons developer list
Subject: Re: [Scons-dev] How to traverse the graph after files are read

On Wed, May 20, 2015 at 4:59 PM, Kenny, Jason L <jason.l.kenny at intel.com> wrote:
>>>Are there two separate FS objects for that?
>
> No there is one global fs that is shared with all environment objects.
>
> After the Sconstruct are read in this will hold all nodes that are known to the SCons and their relationships. The scanner have not been run yet, so there might be some node that are still to be added as dependent children. I think a call to get_children() will call the scanners however and return all explicit and implicit node dependents for a given node.

So, I thought that FS object is just used to access filesystem.  Not it looks like an FS is not only a namespace and cache for filesystem access, but also a data/state storage for the build graph. Perhaps that makes SCons more memory efficient, but it is hard to wrap my head around it. Now the parsing code from Main.py start to make sense:

            SCons.Script._SConscript._SConscript(fs, script)

So, this one parses the script and updates fs tree, right?
_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev