[Scons-dev] Directory node up_to_date changes

[Manish Vachharajani]

Ok, what about the following fix for the issue:

If a kid.is_up_to_date returns true, then check to see if kid is derived.
If it is *not* derived then check the signature.  I think (correct me if I
am wrong) this will only check the directory signature when you have a
directory as a source and have given scons no information on how to build
it.  In this case, a correct build requires scanning the whole directory
tree anyway since the builder may use it (otherwise what does it mean to
list a directory as a source).  If it is derived, we are just assuming the
dependence information computed by scons is correct.

The one slow corner case I can think of is if a builder adds new targets
after the build has started and one of those targets moves a directory from
not derived to derived (because on the first evaluation of the kid, scons
didn't know the directory was derived because another scanner didn't know
it until the build progressed a bit.  Here scons will iterate the whole
directory hierarchy to compute the signature only to later find out that
was not necessary.

Manish



On Wed, Mar 22, 2017 at 5:09 PM, Manish Vachharajani <
manishv at unbounded.systems> wrote:

> Knowing how --tree=all,status works it really can't be a misnomer, all
> those nodes are in children, and we will check all the directory
> signatures.  Fortunately, I think the check is only linear in the number of
> listed source files since the directory hierarchy nests.  Moreover, the
> changed since last build check only applies to the local node.
>
> Ugh, why do things always have to be harder than they should be.  I have
> to go for the day but tomorrow I'll look into how to address this issue too
> (or try to decide if the time complexity is due to real work that must be
> done for correctness).  Also, I'll try to figure out what the added time
> complexity really is.
>
> Manish
>
> On Wed, Mar 22, 2017 at 4:59 PM, William Blevins <wblevins001 at gmail.com>
> wrote:
>
>> Manish,
>>
>> Yeah, I would assume sources depend on directories, but the output of
>> --tree seems to imply otherwise? http://scons.org/doc/HTML/scon
>> s-user.html#idm139837639486464
>>
>> # SConstruct contents
>> Command("aaa/bbb/ccc/ddd/eee", "ccc.txt",
>>     [
>>         Mkdir("$TARGET"),
>>         Copy("$TARGET", "$SOURCE")
>>     ]
>> )
>>
>> Program('program', 'qqq/main.c')
>>
>> $ scons --tree=status
>>
>> scons: Reading SConscript files ...
>> scons: done reading SConscript files.
>> scons: Building targets ...
>> scons: *** [aaa/bbb/ccc/ddd/eee] Source `ccc.txt' not found, needed by
>> target `aaa/bbb/ccc/ddd/eee'.
>>  E         = exists
>>   R        = exists in repository only
>>    b       = implicit builder
>>    B       = explicit builder
>>     S      = side effect
>>      P     = precious
>>       A    = always build
>>        C   = current
>>         N  = no clean
>>          H = no cache
>>
>> [E b      ]+-.
>> [E     C  ]  +-SConstruct
>> [  b      ]  +-aaa
>> [  b      ]  | +-aaa/bbb
>> [  b      ]  |   +-aaa/bbb/ccc
>> [  b      ]  |     +-aaa/bbb/ccc/ddd
>> [  B      ]  |       +-aaa/bbb/ccc/ddd/eee
>> [         ]  |         +-ccc.txt
>> [         ]  +-ccc.txt
>> [  B      ]  +-program
>> [  B      ]  | +-qqq/main.o
>> [E     C  ]  | | +-qqq/main.c
>> [E     C  ]  | | +-/usr/bin/gcc
>> [E     C  ]  | +-/usr/bin/gcc
>> [E b   C  ]  +-qqq
>> [E     C  ]    +-qqq/main.c
>> [  B      ]    +-qqq/main.o
>> [E     C  ]      +-qqq/main.c
>> [E     C  ]      +-/usr/bin/gcc
>>
>> You are looking at the internals currently? Is the tree a misnomer?
>>
>> V/R,
>> William
>>
>>
>> On Wed, Mar 22, 2017 at 6:49 PM, Manish Vachharajani <
>> manishv at unbounded.systems> wrote:
>>
>>> I haven't even tried it yet, I accidentally pushed it, I'm trying to
>>> figure out how to revert that push.
>>>
>>> I was hoping that it would cache the check if the source was used
>>> elsewhere.  Also, I'm not sure why it is O(N*M).  I'm check the sources not
>>> the targets, right?  If there was a directory as a source dependency, then
>>> yes we'd have a problem of O(N*M) but those checks would be necessary,
>>> right?
>>>
>>> In any case, that change was a premature commit, I haven't even tested
>>> to see if it does the right thing yet, much less what performance impact it
>>> might have.
>>>
>>> Manish
>>>
>>> On Wed, Mar 22, 2017 at 4:41 PM, William Blevins <wblevins001 at gmail.com>
>>> wrote:
>>>
>>>> Manish,
>>>>
>>>> Have you thought about the performance penalty for the directory node
>>>> up_to_date change? I assume this is a reason the code operates as it
>>>> currently does...
>>>>
>>>> This will require, O(N * M) checks per directory node where N = max
>>>> directory depth and M = the number of files within the directory chain.
>>>>
>>>> V/R,
>>>> William
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20170323/952158ce/attachment-0001.html>