[Scons-dev] [Scons-users] Bug in up_to_date for directory targets
William Blevins
wblevins001 at gmail.com
Tue Mar 21 18:09:12 EDT 2017
Moving to scons-dev list. Notice email change ^
When I look at the "sconsign -c .scons.dblite ", I noticed that the source
file does not have a csig, so the default decider will not cause a rebuild
(MD5-only).
$ sconsign .sconsign.dblite
> === .:
> SConstruct: None 1490133556 120
> permissions: None 1490133631 5
> === /bin:
> chmod: None 1487766225 60296
> mkdir: None 1487766225 81032
>
The dependency tree looks right:
$ scons --tree=all,status
> scons: Reading SConscript files ...
> scons: done reading SConscript files.
> scons: Building targets ...
> scons: `.' is up to date.
> E = exists
> R = exists in repository only
> b = implicit builder
> B = explicit builder
> S = side effect
> P = precious
> A = always build
> C = current
> N = no clean
> H = no cache
>
> [E b C ]+-.
> [E C ] +-SConstruct
> [E B C ] +-aaa
> [E C ] | +-permissions
> [E C ] | +-/bin/mkdir
> [E C ] | +-/bin/chmod
> [E C ] +-permissions
>
Even if I change the decider I still don't get a rebuild:
env = DefaultEnvironment()
>
> env.Decider('MD5-timestamp')
> env.Command(Dir('aaa'), File('permissions'), ['mkdir $TARGET', 'chmod `cat
> permissions` $TARGET'])
>
Information:
$ scons --version
> SCons by Steven Knight et al.:
> script: v2.5.1.rel_2.5.1:3735:9dc6cee5c168[MODIFIED], 2016/11/03
> 14:02:02, by bdbaddog on mongodog
> engine: v2.5.1.rel_2.5.1:3735:9dc6cee5c168[MODIFIED], 2016/11/03
> 14:02:02, by bdbaddog on mongodog
> engine path: ['/usr/lib/scons/SCons']
> Copyright (c) 2001 - 2016 The SCons Foundation
>
>
$ python --version
>
Python 2.7.13
>
SCons distribution per Debian Apt 2.5.1-1
Thoughts?
William
On Tue, Mar 21, 2017 at 5:39 PM, William Blevins <wblevins001 at gmail.com>
wrote:
> I see what you are saying. I need to look into it some. Did this work in a
> previous version of SCons? What version did it stop working in? What
> version of Python are you using?
>
> It's a bit strange. I needed to fix your example, but I got it to
> replicate. At a glance, the dependency tree looks correct, but nothing
> happens when the permissions file is updated.
>
> V/R,
> William
>
> On Tue, Mar 21, 2017 at 2:09 PM, Manish Vachharajani <
> manishv at unbounded.systems> wrote:
>
>> Let me try to get the discussion back to my main point. There is a bug
>> in scons and there should be a ticket for it and a fix (unless I've
>> misunderstood something, which from the discussion I don't think I have).
>> The nature of the bug is this:
>>
>> 1) The user tells scons that there is a target (foo in my example)
>> 2) The target depends on a source (foo-contents.txt in my example)
>> 3) When the target is out of date with respect to the source, rerun the
>> builder
>>
>> It isn't scons' job, in my opinion, to second guess me and decide that it
>> doesn't really need to run the builder, or that the target isn't the
>> precise output set of the builder etc. The point is that there is a target
>> which is out of date with respect to the source and scons is not running
>> the builder. Everything else, IMHO, is an aside. Does that make it
>> clear? I'm not trying to be rude but something is getting lost when I'm
>> trying to be indirect.
>>
>> I'm glad to discuss what I believe is the root cause based on my
>> investigations in the python debugger. 99% of the time this doesn't matter
>> because there is almost never a directory only set of targets with a
>> builder that isn't simply a mkdir builder or equivalent. Note, however, if
>> the builder were something like:
>>
>> env.Command(['/a/b/c'],
>> ['permissions.txt'],
>> [mkdir $TARGET,
>> chmod `permissions.txt` $TARGET])
>>
>> scons would fail to update the permissions on the directory when
>> permissions.txt changed if /a/b/c already existed. I hope that makes it
>> more clear the exact nature of the issue. My example is simple because it
>> is the smallest SConstruct I could create to demonstrate the issue.
>>
>> For my specific case, I have workarounds, and a custom builder that uses
>> them. A custom builder doesn't help without the workarounds because the
>> problem is in the up to date logic, not the scanner and action logic in
>> scons. Oh, and while package managers let you see content, it doesn't help
>> if the package is going to fetched from the network and I need to calculate
>> dependencies before the package is fetched.
>>
>> Manish
>>
>>
>>
>> On Tue, Mar 21, 2017 at 10:53 AM, William Blevins <wblevins001 at gmail.com>
>> wrote:
>>
>>> Manish,
>>>
>>> Ah, ok. I was confused when your original example seemed so simple. You
>>> should probably create a custom builder then:
>>> https://bitbucket.org/scons/scons/wiki/ToolsForFools
>>>
>>> It isn't a bug that SCons doesn't rebuild targets it doesn't have. The
>>> issue is that you need to tell SCons about said targets. Do this via a
>>> custom builder.
>>>
>>> I haven't used npm, but most package managers have a way to list the
>>> contents of a package, so just convert that list into a target set. I would
>>> need more information about the source -> target dependency chain in terms
>>> of actual vs expected in order to say much more.
>>>
>>> Sorry for the fix response and run, but normally I don't get time to
>>> answer emails this early in the day. Let us know if you think a custom
>>> builder will not solve your problem.
>>>
>>> V/R,
>>> William
>>>
>>> On Tue, Mar 21, 2017 at 12:22 PM, Manish Vachharajani <
>>> manishv at unbounded.systems> wrote:
>>>
>>>> If you look at my original message, the problem is that in my real use
>>>> case, I have no idea what the file targets will be. The only target I know
>>>> will be built by the builder is a set of directories. Your example simply
>>>> causes scons to have a file target in the second Command builder which
>>>> works around the bug as I describe in my original email. In real life, my
>>>> builder looks something like this;
>>>>
>>>> env.Command(['node_modules/minimist', 'node_modules/@types/node, ...],
>>>> #these are directories
>>>> ['package.json'],
>>>> ['npm install --global-style'],
>>>> chdir=Dir('.')) #chdir because npm doesn't have a way to
>>>> install elsewhere
>>>>
>>>> In this case, I have no idea what files npm install will create. The
>>>> targets array is actually created by reading package.json and looking at
>>>> the dependencies and devDependencies fields of the main object, so those
>>>> aren't even hard-coded in the SConstruct/SConscript.
>>>>
>>>> Again, if I fake out scons by have a 'node_modules/.built' File target
>>>> in the list and add a 'touch node_modules/.built' command to the builder
>>>> after 'npm install', all will work as expected because there is a File
>>>> target. In your example, things work because the second builder has a file
>>>> target and SCons will only run the first builder in your example when foo
>>>> does not exist, regardless of the state of foo-contents.txt in relation to
>>>> what is in .sconsign. That is the essence of the bug, if the target
>>>> directory exists and it is the only target, the builder will not run
>>>> regardless of the state of the sources.
>>>>
>>>> Manish
>>>>
>>>>
>>>> On Mon, Mar 20, 2017 at 9:09 PM, William Blevins <wblevins001 at gmail.com
>>>> > wrote:
>>>>
>>>>> Manish,
>>>>>
>>>>> * Why does my example not work?
>>>>>
>>>>> SCons Comand Builder parameters are target(s), source(s), command(s):
>>>>> http://scons.org/doc/HTML/scons-user.html#chap-builders-commands
>>>>>
>>>>> Foo is the target parameter, and foo-contents.txt is the source
>>>>> parameter. Targets explicitly depend on sources. In your example, directory
>>>>> foo depends on source file foo-contents.txt; target file copy does not
>>>>> depend on source file.
>>>>>
>>>>> * How do I fix this issue?
>>>>>
>>>>> 1. Call Command Builder correctly: Command('foo', 'foo-contents',
>>>>> 'mkdir $TARGET') which creates file dependency on directory, and then
>>>>> Command('foo/foo-contents.txt', 'foo-contents.txt', 'cp $SOURCE
>>>>> $TARGET') which creates file target dependency on file source.
>>>>>
>>>>> 2: [PREFERRED] Use the SCons FileSystem factory
>>>>> http://scons.org/doc/HTML/scons-user.html#chap-factories
>>>>>
>>>>> V/R,
>>>>> William
>>>>>
>>>>>
>>>>> On Mon, Mar 20, 2017 at 1:47 PM, Manish Vachharajani <
>>>>> manishv at unbounded.systems> wrote:
>>>>>
>>>>>> I believe there is a bug in how up to date is computed for file nodes
>>>>>> and directory targets. Below is a SConstruct and run output illustrating
>>>>>> the bug.
>>>>>>
>>>>>> ##############
>>>>>> # SConstruct
>>>>>> env = Environment()
>>>>>> targets = env.Command('foo',
>>>>>> 'foo-contents.txt',
>>>>>> ['mkdir foo',
>>>>>> 'cp foo-contents.txt foo'])
>>>>>> env.Default(targets)
>>>>>>
>>>>>> ##############
>>>>>> # Shell log
>>>>>>
>>>>>> # Create the file that will be installed into foo/foo-contents.txt
>>>>>> $ echo "Hello, " >> foo-contents.txt
>>>>>> # All is well, scons runs the command builder as expected
>>>>>> $ scons
>>>>>>
>>>>>> scons: Reading SConscript files ...
>>>>>> scons: done reading SConscript files.
>>>>>> scons: Building targets ...
>>>>>> mkdir foo
>>>>>> cp foo-contents.txt foo
>>>>>> scons: done building targets.
>>>>>>
>>>>>> # All is well, the target is up to date as expected
>>>>>> $ scons
>>>>>>
>>>>>> scons: Reading SConscript files ...
>>>>>> scons: done reading SConscript files.
>>>>>> scons: Building targets ...
>>>>>> scons: `foo' is up to date.
>>>>>> scons: done building targets.
>>>>>>
>>>>>> # Uh-oh, source is updated but the target is considered up to date!
>>>>>> $ echo "World!" >> foo-contents.txt
>>>>>> $ scons
>>>>>>
>>>>>> scons: Reading SConscript files ...
>>>>>> scons: done reading SConscript files.
>>>>>> scons: Building targets ...
>>>>>> scons: `foo' is up to date.
>>>>>> scons: done building targets.
>>>>>>
>>>>>> For the bug to trigger, all targets for a builder must be
>>>>>> directories. The target directory must exist and the source file must
>>>>>> exist and be out of date due to a content update (resulting in a signature
>>>>>> mismatch).
>>>>>>
>>>>>> I think the problem is that is_up_to_date for File nodes only checks
>>>>>> the signatures of its children, which avoids this bug as long as one of the
>>>>>> builder targets is a File and not a Directory. However, I haven't done
>>>>>> enough analysis to be sure that this is the issue.
>>>>>>
>>>>>> A work around is to make sure that you have a file (or create a dummy
>>>>>> .built file) and include it as a target.
>>>>>>
>>>>>> This is showing up specifically for us when trying to calculate
>>>>>> dependencies for a package manager (npm) where we have no insight into the
>>>>>> contents of the resulting directory.
>>>>>>
>>>>>> Manish
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20170321/431e85d5/attachment-0001.html>
More information about the Scons-dev
mailing list