[Scons-dev] Subst() replacement?

[Bill Deegan]

There's a pull request on scons repo with the current state of code.
There's a wiki page on the approach as well.
The approach should remove all the recursion, add caching, and pre-tokenize
(so strings aren't tokenized every time they're evaluated).

In the meantime if you want to shortcut some of your recursion. Walk you're
envs and env.Subst() them where it make sense.

I see chains of strings like this

x='$y', y="$z', z='$a' z="$b" b=".obj"

There's likely lots of those which could reduce your recursion.

It's unfortunate that currently SCons has no way of representing makes
":="  and all variables are equivalent to "=" where they're evaluated every
time their referenced..

What do you  expect ${${}} to do?


On Mon, Feb 18, 2019 at 3:14 PM Jason Kenny <dragon512 at live.com> wrote:

> Do you have a link for me to look at?
>
> Else I was going to start by making a expression parser to allow the
> breaking up of the $ ${} $() $$ and ${${}} ( this last one is a feature
> improvement) style expression and then make tweaked versions of the
> existing classes in subst module to try to avoid recursion when it can.
>
> Jason
> ------------------------------
> *From:* Scons-dev <scons-dev-bounces at scons.org> on behalf of Bill Deegan <
> bill at baddogconsulting.com>
> *Sent:* Monday, February 18, 2019 5:05 PM
> *To:* SCons developer list
> *Subject:* Re: [Scons-dev] Subst() replacement?
>
> Jason,
>
> The code I have is still a work in progress.
> Putting it in parts first doesn't make sense to me as when it's ready I'll
> be dropping it into SCons itself, though likely as a switchable alternative
> initially.
>
> I'll be back to working on it in about a month I expect. Maybe sooner.
>
> -Bill
>
> On Mon, Feb 18, 2019 at 11:18 AM Jason Kenny <dragon512 at live.com> wrote:
>
> Hi Bill,
>
> I know in the past we have talked about various things to do to fix up
> Scons. One of the items was the subst engine we have. Currently for me the
> issue with Parts is that I use the Subst engine to pass data around between
> different components in a automated way. This works well minus two issues:
>
> 1) it can take time. However I have various tweaks in the object I pass to
> the subst call to help cache a number of cases.
> 2) stack depth. This is the major issue for me at the moment.
>
> The issue with with 2) is seen with -j based builds which can lead to seg
> faults. Depending on the build, it is common to have some depth of 10-20+
> components. This can lead to a stack of depth per thread of 250 to 300+.
> When doing a -j based build this blows the stack in a bad way.
>
> one can easily get stack with this pattern:
>
> ...
> File "/build-env/lib/python3.6/site-packages/scons/SCons/Subst.py", line
> 543, in substitute
>     return self.expand(args, lvars)
>   File "/build-env/lib/python3.6/site-packages/scons/SCons/Subst.py", line
> 495, in expand
>     return list(map(func, s))
>   File "/build-env/lib/python3.6/site-packages/scons/SCons/Subst.py", line
> 494, in func
>     return conv(substitute(l, lvars))
>   File "/build-env/lib/python3.6/site-packages/scons/SCons/Subst.py", line
> 526, in substitute
>     result = _dollar_exps.sub(sub_match, args)
>   File "/build-env/lib/python3.6/site-packages/scons/SCons/Subst.py", line
> 525, in sub_match
>     return self.conv(self.expand(match.group(1), lvars))
>   File "/build-env/lib/python3.6/site-packages/scons/SCons/Subst.py", line
> 491, in expand
>     return self.substitute(s, lv)
>   File "/build-env/lib/python3.6/site-packages/scons/SCons/Subst.py", line
> 543, in substitute
>     return self.expand(args, lvars)
>   File "/build-env/lib/python3.6/site-packages/scons/SCons/Subst.py", line
> 509, in expand
>     return self.substitute(s, lvars)
>   File "/build-env/lib/python3.6/site-packages/scons/SCons/Subst.py", line
> 543, in substitute
>     return self.expand(args, lvars)
>   File "/build-env/lib/python3.6/site-packages/scons/SCons/Subst.py", line
> 495, in expand
>     return list(map(func, s))
>   File "/build-env/lib/python3.6/site-packages/scons/SCons/Subst.py", line
> 494, in func
>     return conv(substitute(l, lvars))
>   File "/build-env/lib/python3.6/site-packages/scons/SCons/Subst.py", line
> 543, in substitute
>     return self.expand(args, lvars)
>   File "/build-env/lib/python3.6/site-packages/scons/SCons/Subst.py", line
> 495, in expand
>     return list(map(func, s))
>   File "/build-env/lib/python3.6/site-packages/scons/SCons/Subst.py", line
> 494, in func
>     return conv(substitute(l, lvars))
>   File "/build-env/lib/python3.6/site-packages/scons/SCons/Subst.py", line
> 526, in substitute
>     result = _dollar_exps.sub(sub_match, args)
>   File "/build-env/lib/python3.6/site-packages/scons/SCons/Subst.py", line
> 525, in sub_match
> ...
>
> I had mentioned in the past I want to not use the subst call as much, but
> change this with something else is non trivial at the moment. I do recall
> you had done some work on this. I am looking at trying to fix the subst
> logic in Scons . What I propose to do it to override the subst module in
> SCons with Parts. The goal would to be to show a better version we could
> the move in to Scons with minimum effort as well as allow me to move
> forward with out having to block on a new SCons drop.
>
> I believe you had done work on this already. I was hoping I could try to
> start with that with hope this can be used to reduce the recursion issues
> in SCons. This seemed to be better than start from scratch.
>
> Oh course if you have a better idea I am
>
> Jason
>
>
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
> <https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpairlist2.pair.net%2Fmailman%2Flistinfo%2Fscons-dev&data=02%7C01%7C%7Cde0bea34038744b51bbc08d695f59d7b%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636861279472300731&sdata=75yXbVakjeXi61d0cgZwJSSHZzYjgS3wSViRtqR0lBw%3D&reserved=0>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20190218/53fe5ce7/attachment-0001.html>