[Scons-dev] VariantDir() and #paths to files

[Jason Kenny]

Parts defines structure that the user can overrides.

The reason I was talking about this in SCons is that accessing node outside the directory tree of the SConstruct or sconscript defining a VariantDir() was always viewed as something you could not or should not do. At the very least it was unclear. I was hoping that at SCons level something could be done to make this accessible to everyone else. My finding seem to be that it probably not that easy or wise to do so.

In the end I have a solution that allows the UnitTest to work as well as general builders provided by SCons or Parts to access files outside the default directory tree and build correctly in the VariantDir() as desired no, instead of build in the srcnode() area.

Thanks for help in discussing this with me.
Jason

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Bill Deegan<mailto:bill at baddogconsulting.com>
Sent: Friday, March 11, 2016 5:41 PM
To: Jason Kenny<mailto:dragon512 at live.com>
Cc: SCons developer list<mailto:scons-dev at scons.org>; scons-users at scons.org<mailto:scons-users at scons.org>
Subject: Re: [Scons-dev] VariantDir() and #paths to files

Jason,

>From a brief look at this it seems like a Parts specific issue to me.
In terms of the solution you are suggesting may make sense in the
predefined configuration Parts is supporting.
(From what you're saying Parts makes a bunch of decisions on layout of the
tree for the users).

Also if this is specifically about your UnitTest builder, change it into a
psuedo method and monkey with the source nodes there?

-Bill

On Fri, Mar 11, 2016 at 6:07 PM, Jason Kenny <dragon512 at live.com> wrote:

> Bill,
>
>
>
> >>SCons only allows one variantdir processed per target dir, but not per
> Environment().
>
>
>
> In Parts, each Part has its own variant directory define by the BUILD_DIR
> variable. So this is not an issue. Making mapping to # for each part would
> make a different directory.
>
>
>
> >> How would this work when there are N variants (and thus N variant
> directory trees)?
>
>
>
> I guess how one uses Variant directories could make this complex. A common
> case shown is to use VariantDir to control build configurations, such as
> Debug and Release. This could be done in Raw Scons, however I found this
> for larger projects very complex to do ( much like using make VPATH is). I
> deal with this in Parts via defining a configuration concept that is bound
> the toolchain improvements I have made. This allows me to define a CONFIG
> variable that can be used per environment (or Part/Component) to control
> what setting are loaded. For the VariantDir point of view the CONFIG value
> is used in compose BUILD_DIR finial value. This mean the Parts has a
> VariantDir for each Part. Adding the idea of out of tree files for the
> Sconstruct or a given Parts ( as I have working now ) is just adding a two
> new variants for each Parts before I read in the file. The new Variants are
> controlled by new BUILD_DIR variable like ROOT_BUILD_DIR which would
> control path outside the Sconstruct tree.
>
>
>
> For to get this to work I have to do two things:
>
> 1)      Add a AbsDirNode() and AbsFileNode() to correctly translate the
> string and returns to the correctly formed Node so Scons will make a
> variant node correctly for the source. (I had a AbsDir and AbsFile, but the
> return strings and are useful for adding values to variables like CPPATH)
>
> 2)      Tweak my UnitTest function to process certain string correctly
> based on the new values
>
>
>
> My thought was that it would be nice is this was build into the Node API
> to just do this. So if a user said:
>
> env.File(‘#somedir/sub1/file2’)
>
>
>
> it would be mapped to a “$ROOT_BULD_DIR/somedir/sub1/file2” automatically
> when making a node. I however understand such a change could have high
> risk, and would need to be tested. I also understand that for me in Parts I
> sort of have a state because of the BUILD_DIR variables I have setup, so
> any Part would have three VariantDir() being “active” at a time because of
> what BUILD_DIR, ROOT_BUILD_DIR, etc would map to when a Part file is read.
> SCons does not have at the core a component notion of any kind so this
> probably would not map well in SCons in hindsight. SCons would need some
> “state” API to allow for this in an order dependent way. It might be
> possible if the SConscript could be updated to take a list of VariantDir
> and src_dir. I would have to test and see what would happen if I tried to
> modified the Nodes via a monkey patch in Parts to see if I could have the
> Variant “path” translation happen naturally in the nodes in a safe way.
> Till then I think you are right in that this does not make sense yet in raw
> SCons.
>
>
>
> >>You might be better to use Repository pointing targets into the
> variantdir?
>
> I am not sure how this would help. Honestly I was looking at using the
> Repository feature to help speed up large build, by allowing the user to
> point to a build on a shared drive and build off of that. The idea here is
> different from CacheDir() in that it would point to a given build to allow
> a developer to quick debug and fix an issue seen in the field without
> having to build that revision locally. How did you think Repository() could
> help with this VariantDir() discussion?
>
>
>
>
>
> I have a pull request open for my fix in Parts to this issue. I added you
> Bill in case you wanted to peak at it and provide any thoughts to what I am
> doing in Parts.
>
>
>
> Jason
>
>
>
>
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
> *From: *Bill Deegan <bill at baddogconsulting.com>
> *Sent: *Thursday, March 10, 2016 12:37 PM
> *To: *Jason Kenny <dragon512 at live.com>
> *Cc: *SCons developer list <scons-dev at scons.org>; scons-users at scons.org
>
> *Subject: *Re: [Scons-dev] VariantDir() and #paths to files
>
>
> Jason,
>
>
>
>
> On Thu, Mar 10, 2016 at 12:23 PM, Jason Kenny <dragon512 at live.com> wrote:
>
> Bill,
>
>
>
> First, let me add this to user group.
>
>
>
> So maybe I am doing something wrong, but..
>
>
>
> I think the core issue here is that SCons only allows one active
> VariantDir at a time to be actively translated.
>
>
> SCons only allows one variantdir processed per target dir, but not per
> Environment().
> So src/a -> output/a, src/b -> output_b/b, etc is fine
>
> src/a -> output/a
> src/bin -> output/b # Fails..
>
>
>
> When a user does a
>
>
>
> Env.VariantDir(“debug”,”src”)
>
> Env.Sconscript(“debug/Sconscript”)
>
>
>
> The user has to refer to files in the Sconscript as “file1” or
> “sub1/file2” and these value get changed into “debug/file1” or
> “debug/sub1/file2”.
>
>
>
> If I was to refer to a file outside the src directory such as a
> “src2/file3”. This requires active translation by the user for file not
> under the source directory of the sconscript.
>
>
>
> I can do one of two things.
>
>
>
> 1.       add a another VariantDir(“#debug/outoftree”,”.”) and then monkey
> path env.File() to auto translate any string that is not under the current
> source directory. This way a #src2/file3 would be translated to a
> “#debug/outoftree/src2/file3”, much like the file “sub1/file2” is turned in
> to “debug/sub1/file2” inside the Sconscript with out user intervention.
>
> 2.       Try to setup funny relative path, however if this tends to fail
> if the Variant directory and source directory don’t have the same level of
> subdirectories.
>
>
>
> I think what would be useful for SCons is to have the logic in the Nodes
> to allow for 1) to work. While one can argue about what the current way
> Variant Directory works is correct, I believe it is clear that users need
> the translation to work for usability. In complex builds it is not always
> obvious what the Variant Directory is.
>
>
> How would this work when there are N variants (and thus N variant
> directory trees)?
>
>
>
>
> I think I going to have to setup some form of 1) to address this problem
> in Parts. This means I setup a set of 1 or 2 more active directory’s that I
> would look for when the File, Dir, etc node is made and translate the path
> if it start with correct source path. This will allow different parts such
> as PartA and PartB refer to a common file out these parts subtree and built
> it correctly for each Part case.
>
>
>
> Or I could add a new variable like $ROOTNODE_BUILD_DIR and change the
> Parts AbsFile() api to translate these files to correct Variant Directory
> based on it being under the Sconstruct directory or outside it.
>
>
>
> I think making a way to do this more directly without a monkey patch would
> be useful. I need to look at the second option with modifying the Part
> AbsFile API to see if this could work. This might be an easy addition that
> could be moved to SCons.
>
>
> You might be better to use Repository pointing targets into the variantdir?
>
> -Bill
>
>
>
> Jason
>
>
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
> *From: *Bill Deegan <bill at baddogconsulting.com>
> *Sent: *Wednesday, March 9, 2016 9:29 PM
> *To: *SCons developer list <scons-dev at scons.org>
> *Subject: *Re: [Scons-dev] VariantDir() and #paths to files
>
>
> Jason,
>
> The problem you are seeing ( believe) is that calling SConscript() with a
> variant_dir argument performs the function
>
> VariantDir(whatever the variant dir was specified, directory of the
> SConscript file).
>
> Where you'd likely need also
>
> VariantDir(whatever the variant dir was specified, "#/myprog/test/unit")
> And then to have the source file referenced via the variant dir and not
> the source dir #/myprog/test/unit.
>
> So then the question would be is it correct to automatically apply the
> variantdir behavior to any source file specified in the SConscript being
> read when it is specified with variant_dir?
> One thing for sure, doing so would be a break with current functionality
> and would trigger a deprecation cycle.
>
> Would it make sense to provide (an)other flag(s) to tell SConscript to do
> this?
>
> These are just some items to consider from a quick look at the issue.
>
> Also, I'd say this would be safe/better for users mailing list since this
> is about how to use/proper functioning of user visible SCons logic.
> -Bill
>
> On Wed, Mar 9, 2016 at 10:15 PM, Jason Kenny <dragon512 at live.com> wrote:
>
>
>
> Hi guys,
>
>
>
> There has been a constant issue I have had with using SCons with
> VariantDir logic. I was wondering if I am missing something.
>
>
>
> The issue shows up a number of time when people use Parts. The issue is
> when the user wants to use a file outside the directory containing the part
> file ( From a pure Scons point of view a part file is what would be called
> a SConscript). I believe this is a known issue with SCons, however it seems
> this is a common problem I have seen many times, enough for me to think
> there should be some better solution.
>
>
>
> I have a example of this with an issue in Parts just opened.
> https://bitbucket.org/sconsparts/parts/issues/1/unit-test-object-build-files-can-appear-in
>
>
>
> The easy answer for me is to say move the parts file up to the root. And
> tweak a few paths in a part file, to allow all files to be at the same
> level or below the location of the parts file. While this will solve the
> problem, I know it will be seen again as users have a reason they want to
> locate the “build” files in different locations from the all or some of the
> source. In this example they seem to have a pattern in which they have some
> “common” test helpers files that they want other Parts to use when creating
> there unit test. It makes sense, I would like to find a way to allow this
> functionality to the user in a painless way ( ie I do something under the
> covers 😊 ). There is an attached example of the issue.
>
>
>
> Does anyone have any thought on this?
>
>
>
>
>
> Jason
>
>
>
>
>
>
>
>
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20160311/b0d177a3/attachment-0001.html>