[Scons-dev] VariantDir() and #paths to files

[Bill Deegan]

Jason,

>From a brief look at this it seems like a Parts specific issue to me.
In terms of the solution you are suggesting may make sense in the
predefined configuration Parts is supporting.
(From what you're saying Parts makes a bunch of decisions on layout of the
tree for the users).

Also if this is specifically about your UnitTest builder, change it into a
psuedo method and monkey with the source nodes there?

-Bill

On Fri, Mar 11, 2016 at 6:07 PM, Jason Kenny <dragon512 at live.com> wrote:

> Bill,
>
>
>
> >>SCons only allows one variantdir processed per target dir, but not per
> Environment().
>
>
>
> In Parts, each Part has its own variant directory define by the BUILD_DIR
> variable. So this is not an issue. Making mapping to # for each part would
> make a different directory.
>
>
>
> >> How would this work when there are N variants (and thus N variant
> directory trees)?
>
>
>
> I guess how one uses Variant directories could make this complex. A common
> case shown is to use VariantDir to control build configurations, such as
> Debug and Release. This could be done in Raw Scons, however I found this
> for larger projects very complex to do ( much like using make VPATH is). I
> deal with this in Parts via defining a configuration concept that is bound
> the toolchain improvements I have made. This allows me to define a CONFIG
> variable that can be used per environment (or Part/Component) to control
> what setting are loaded. For the VariantDir point of view the CONFIG value
> is used in compose BUILD_DIR finial value. This mean the Parts has a
> VariantDir for each Part. Adding the idea of out of tree files for the
> Sconstruct or a given Parts ( as I have working now ) is just adding a two
> new variants for each Parts before I read in the file. The new Variants are
> controlled by new BUILD_DIR variable like ROOT_BUILD_DIR which would
> control path outside the Sconstruct tree.
>
>
>
> For to get this to work I have to do two things:
>
> 1)      Add a AbsDirNode() and AbsFileNode() to correctly translate the
> string and returns to the correctly formed Node so Scons will make a
> variant node correctly for the source. (I had a AbsDir and AbsFile, but the
> return strings and are useful for adding values to variables like CPPATH)
>
> 2)      Tweak my UnitTest function to process certain string correctly
> based on the new values
>
>
>
> My thought was that it would be nice is this was build into the Node API
> to just do this. So if a user said:
>
> env.File(‘#somedir/sub1/file2’)
>
>
>
> it would be mapped to a “$ROOT_BULD_DIR/somedir/sub1/file2” automatically
> when making a node. I however understand such a change could have high
> risk, and would need to be tested. I also understand that for me in Parts I
> sort of have a state because of the BUILD_DIR variables I have setup, so
> any Part would have three VariantDir() being “active” at a time because of
> what BUILD_DIR, ROOT_BUILD_DIR, etc would map to when a Part file is read.
> SCons does not have at the core a component notion of any kind so this
> probably would not map well in SCons in hindsight. SCons would need some
> “state” API to allow for this in an order dependent way. It might be
> possible if the SConscript could be updated to take a list of VariantDir
> and src_dir. I would have to test and see what would happen if I tried to
> modified the Nodes via a monkey patch in Parts to see if I could have the
> Variant “path” translation happen naturally in the nodes in a safe way.
> Till then I think you are right in that this does not make sense yet in raw
> SCons.
>
>
>
> >>You might be better to use Repository pointing targets into the
> variantdir?
>
> I am not sure how this would help. Honestly I was looking at using the
> Repository feature to help speed up large build, by allowing the user to
> point to a build on a shared drive and build off of that. The idea here is
> different from CacheDir() in that it would point to a given build to allow
> a developer to quick debug and fix an issue seen in the field without
> having to build that revision locally. How did you think Repository() could
> help with this VariantDir() discussion?
>
>
>
>
>
> I have a pull request open for my fix in Parts to this issue. I added you
> Bill in case you wanted to peak at it and provide any thoughts to what I am
> doing in Parts.
>
>
>
> Jason
>
>
>
>
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
> *From: *Bill Deegan <bill at baddogconsulting.com>
> *Sent: *Thursday, March 10, 2016 12:37 PM
> *To: *Jason Kenny <dragon512 at live.com>
> *Cc: *SCons developer list <scons-dev at scons.org>; scons-users at scons.org
>
> *Subject: *Re: [Scons-dev] VariantDir() and #paths to files
>
>
> Jason,
>
>
>
>
> On Thu, Mar 10, 2016 at 12:23 PM, Jason Kenny <dragon512 at live.com> wrote:
>
> Bill,
>
>
>
> First, let me add this to user group.
>
>
>
> So maybe I am doing something wrong, but..
>
>
>
> I think the core issue here is that SCons only allows one active
> VariantDir at a time to be actively translated.
>
>
> SCons only allows one variantdir processed per target dir, but not per
> Environment().
> So src/a -> output/a, src/b -> output_b/b, etc is fine
>
> src/a -> output/a
> src/bin -> output/b # Fails..
>
>
>
> When a user does a
>
>
>
> Env.VariantDir(“debug”,”src”)
>
> Env.Sconscript(“debug/Sconscript”)
>
>
>
> The user has to refer to files in the Sconscript as “file1” or
> “sub1/file2” and these value get changed into “debug/file1” or
> “debug/sub1/file2”.
>
>
>
> If I was to refer to a file outside the src directory such as a
> “src2/file3”. This requires active translation by the user for file not
> under the source directory of the sconscript.
>
>
>
> I can do one of two things.
>
>
>
> 1.       add a another VariantDir(“#debug/outoftree”,”.”) and then monkey
> path env.File() to auto translate any string that is not under the current
> source directory. This way a #src2/file3 would be translated to a
> “#debug/outoftree/src2/file3”, much like the file “sub1/file2” is turned in
> to “debug/sub1/file2” inside the Sconscript with out user intervention.
>
> 2.       Try to setup funny relative path, however if this tends to fail
> if the Variant directory and source directory don’t have the same level of
> subdirectories.
>
>
>
> I think what would be useful for SCons is to have the logic in the Nodes
> to allow for 1) to work. While one can argue about what the current way
> Variant Directory works is correct, I believe it is clear that users need
> the translation to work for usability. In complex builds it is not always
> obvious what the Variant Directory is.
>
>
> How would this work when there are N variants (and thus N variant
> directory trees)?
>
>
>
>
> I think I going to have to setup some form of 1) to address this problem
> in Parts. This means I setup a set of 1 or 2 more active directory’s that I
> would look for when the File, Dir, etc node is made and translate the path
> if it start with correct source path. This will allow different parts such
> as PartA and PartB refer to a common file out these parts subtree and built
> it correctly for each Part case.
>
>
>
> Or I could add a new variable like $ROOTNODE_BUILD_DIR and change the
> Parts AbsFile() api to translate these files to correct Variant Directory
> based on it being under the Sconstruct directory or outside it.
>
>
>
> I think making a way to do this more directly without a monkey patch would
> be useful. I need to look at the second option with modifying the Part
> AbsFile API to see if this could work. This might be an easy addition that
> could be moved to SCons.
>
>
> You might be better to use Repository pointing targets into the variantdir?
>
> -Bill
>
>
>
> Jason
>
>
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
> *From: *Bill Deegan <bill at baddogconsulting.com>
> *Sent: *Wednesday, March 9, 2016 9:29 PM
> *To: *SCons developer list <scons-dev at scons.org>
> *Subject: *Re: [Scons-dev] VariantDir() and #paths to files
>
>
> Jason,
>
> The problem you are seeing ( believe) is that calling SConscript() with a
> variant_dir argument performs the function
>
> VariantDir(whatever the variant dir was specified, directory of the
> SConscript file).
>
> Where you'd likely need also
>
> VariantDir(whatever the variant dir was specified, "#/myprog/test/unit")
> And then to have the source file referenced via the variant dir and not
> the source dir #/myprog/test/unit.
>
> So then the question would be is it correct to automatically apply the
> variantdir behavior to any source file specified in the SConscript being
> read when it is specified with variant_dir?
> One thing for sure, doing so would be a break with current functionality
> and would trigger a deprecation cycle.
>
> Would it make sense to provide (an)other flag(s) to tell SConscript to do
> this?
>
> These are just some items to consider from a quick look at the issue.
>
> Also, I'd say this would be safe/better for users mailing list since this
> is about how to use/proper functioning of user visible SCons logic.
> -Bill
>
> On Wed, Mar 9, 2016 at 10:15 PM, Jason Kenny <dragon512 at live.com> wrote:
>
>
>
> Hi guys,
>
>
>
> There has been a constant issue I have had with using SCons with
> VariantDir logic. I was wondering if I am missing something.
>
>
>
> The issue shows up a number of time when people use Parts. The issue is
> when the user wants to use a file outside the directory containing the part
> file ( From a pure Scons point of view a part file is what would be called
> a SConscript). I believe this is a known issue with SCons, however it seems
> this is a common problem I have seen many times, enough for me to think
> there should be some better solution.
>
>
>
> I have a example of this with an issue in Parts just opened.
> https://bitbucket.org/sconsparts/parts/issues/1/unit-test-object-build-files-can-appear-in
>
>
>
> The easy answer for me is to say move the parts file up to the root. And
> tweak a few paths in a part file, to allow all files to be at the same
> level or below the location of the parts file. While this will solve the
> problem, I know it will be seen again as users have a reason they want to
> locate the “build” files in different locations from the all or some of the
> source. In this example they seem to have a pattern in which they have some
> “common” test helpers files that they want other Parts to use when creating
> there unit test. It makes sense, I would like to find a way to allow this
> functionality to the user in a painless way ( ie I do something under the
> covers 😊 ). There is an attached example of the issue.
>
>
>
> Does anyone have any thought on this?
>
>
>
>
>
> Jason
>
>
>
>
>
>
>
>
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20160311/09583d67/attachment-0001.html>