[Scons-dev] [Scons-users] Bug in up_to_date for directory targets
Bill Deegan
bill at baddogconsulting.com
Tue Mar 21 18:55:01 EDT 2017
Likely this has to do with the target being a directory, which has never
worked quite the way people expect.
SCons is primarily a file based build system. Directories are just
containers which those files are stored in.
By default they depend on their contents.
Thus allowing directories to be created prior to file generated into them
being created.
If you make the source a value node with reasonable contents and then
change it does it make a difference?
On Tue, Mar 21, 2017 at 6:33 PM, William Blevins <wblevins001 at gmail.com>
wrote:
> I also checked scons 2.4.1 and 2.3.0, so this goes back a while
> apparently? Or maybe this never worked right...
>
> Can someone confirm?
>
> V/R,
> William
>
> On Tue, Mar 21, 2017 at 6:09 PM, William Blevins <wblevins001 at gmail.com>
> wrote:
>
>> Moving to scons-dev list. Notice email change ^
>>
>> When I look at the "sconsign -c .scons.dblite ", I noticed that the
>> source file does not have a csig, so the default decider will not cause a
>> rebuild (MD5-only).
>>
>> $ sconsign .sconsign.dblite
>>> === .:
>>> SConstruct: None 1490133556 120
>>> permissions: None 1490133631 5
>>> === /bin:
>>> chmod: None 1487766225 60296
>>> mkdir: None 1487766225 81032
>>>
>>
>> The dependency tree looks right:
>>
>> $ scons --tree=all,status
>>> scons: Reading SConscript files ...
>>> scons: done reading SConscript files.
>>> scons: Building targets ...
>>> scons: `.' is up to date.
>>> E = exists
>>> R = exists in repository only
>>> b = implicit builder
>>> B = explicit builder
>>> S = side effect
>>> P = precious
>>> A = always build
>>> C = current
>>> N = no clean
>>> H = no cache
>>>
>>> [E b C ]+-.
>>> [E C ] +-SConstruct
>>> [E B C ] +-aaa
>>> [E C ] | +-permissions
>>> [E C ] | +-/bin/mkdir
>>> [E C ] | +-/bin/chmod
>>> [E C ] +-permissions
>>>
>>
>> Even if I change the decider I still don't get a rebuild:
>>
>> env = DefaultEnvironment()
>>>
>>> env.Decider('MD5-timestamp')
>>> env.Command(Dir('aaa'), File('permissions'), ['mkdir $TARGET', 'chmod
>>> `cat permissions` $TARGET'])
>>>
>>
>> Information:
>>
>> $ scons --version
>>> SCons by Steven Knight et al.:
>>> script: v2.5.1.rel_2.5.1:3735:9dc6cee5c168[MODIFIED], 2016/11/03
>>> 14:02:02, by bdbaddog on mongodog
>>> engine: v2.5.1.rel_2.5.1:3735:9dc6cee5c168[MODIFIED], 2016/11/03
>>> 14:02:02, by bdbaddog on mongodog
>>> engine path: ['/usr/lib/scons/SCons']
>>> Copyright (c) 2001 - 2016 The SCons Foundation
>>>
>>
>>>
>> $ python --version
>>>
>> Python 2.7.13
>>>
>>
>> SCons distribution per Debian Apt 2.5.1-1
>>
>> Thoughts?
>> William
>>
>>
>>
>>
>>
>>
>> On Tue, Mar 21, 2017 at 5:39 PM, William Blevins <wblevins001 at gmail.com>
>> wrote:
>>
>>> I see what you are saying. I need to look into it some. Did this work in
>>> a previous version of SCons? What version did it stop working in? What
>>> version of Python are you using?
>>>
>>> It's a bit strange. I needed to fix your example, but I got it to
>>> replicate. At a glance, the dependency tree looks correct, but nothing
>>> happens when the permissions file is updated.
>>>
>>> V/R,
>>> William
>>>
>>> On Tue, Mar 21, 2017 at 2:09 PM, Manish Vachharajani <
>>> manishv at unbounded.systems> wrote:
>>>
>>>> Let me try to get the discussion back to my main point. There is a bug
>>>> in scons and there should be a ticket for it and a fix (unless I've
>>>> misunderstood something, which from the discussion I don't think I have).
>>>> The nature of the bug is this:
>>>>
>>>> 1) The user tells scons that there is a target (foo in my example)
>>>> 2) The target depends on a source (foo-contents.txt in my example)
>>>> 3) When the target is out of date with respect to the source, rerun the
>>>> builder
>>>>
>>>> It isn't scons' job, in my opinion, to second guess me and decide that
>>>> it doesn't really need to run the builder, or that the target isn't the
>>>> precise output set of the builder etc. The point is that there is a target
>>>> which is out of date with respect to the source and scons is not running
>>>> the builder. Everything else, IMHO, is an aside. Does that make it
>>>> clear? I'm not trying to be rude but something is getting lost when I'm
>>>> trying to be indirect.
>>>>
>>>> I'm glad to discuss what I believe is the root cause based on my
>>>> investigations in the python debugger. 99% of the time this doesn't matter
>>>> because there is almost never a directory only set of targets with a
>>>> builder that isn't simply a mkdir builder or equivalent. Note, however, if
>>>> the builder were something like:
>>>>
>>>> env.Command(['/a/b/c'],
>>>> ['permissions.txt'],
>>>> [mkdir $TARGET,
>>>> chmod `permissions.txt` $TARGET])
>>>>
>>>> scons would fail to update the permissions on the directory when
>>>> permissions.txt changed if /a/b/c already existed. I hope that makes it
>>>> more clear the exact nature of the issue. My example is simple because it
>>>> is the smallest SConstruct I could create to demonstrate the issue.
>>>>
>>>> For my specific case, I have workarounds, and a custom builder that
>>>> uses them. A custom builder doesn't help without the workarounds because
>>>> the problem is in the up to date logic, not the scanner and action logic in
>>>> scons. Oh, and while package managers let you see content, it doesn't help
>>>> if the package is going to fetched from the network and I need to calculate
>>>> dependencies before the package is fetched.
>>>>
>>>> Manish
>>>>
>>>>
>>>>
>>>> On Tue, Mar 21, 2017 at 10:53 AM, William Blevins <
>>>> wblevins001 at gmail.com> wrote:
>>>>
>>>>> Manish,
>>>>>
>>>>> Ah, ok. I was confused when your original example seemed so simple.
>>>>> You should probably create a custom builder then:
>>>>> https://bitbucket.org/scons/scons/wiki/ToolsForFools
>>>>>
>>>>> It isn't a bug that SCons doesn't rebuild targets it doesn't have. The
>>>>> issue is that you need to tell SCons about said targets. Do this via a
>>>>> custom builder.
>>>>>
>>>>> I haven't used npm, but most package managers have a way to list the
>>>>> contents of a package, so just convert that list into a target set. I would
>>>>> need more information about the source -> target dependency chain in terms
>>>>> of actual vs expected in order to say much more.
>>>>>
>>>>> Sorry for the fix response and run, but normally I don't get time to
>>>>> answer emails this early in the day. Let us know if you think a custom
>>>>> builder will not solve your problem.
>>>>>
>>>>> V/R,
>>>>> William
>>>>>
>>>>> On Tue, Mar 21, 2017 at 12:22 PM, Manish Vachharajani <
>>>>> manishv at unbounded.systems> wrote:
>>>>>
>>>>>> If you look at my original message, the problem is that in my real
>>>>>> use case, I have no idea what the file targets will be. The only target I
>>>>>> know will be built by the builder is a set of directories. Your example
>>>>>> simply causes scons to have a file target in the second Command builder
>>>>>> which works around the bug as I describe in my original email. In real
>>>>>> life, my builder looks something like this;
>>>>>>
>>>>>> env.Command(['node_modules/minimist', 'node_modules/@types/node,
>>>>>> ...], #these are directories
>>>>>> ['package.json'],
>>>>>> ['npm install --global-style'],
>>>>>> chdir=Dir('.')) #chdir because npm doesn't have a way to
>>>>>> install elsewhere
>>>>>>
>>>>>> In this case, I have no idea what files npm install will create. The
>>>>>> targets array is actually created by reading package.json and looking at
>>>>>> the dependencies and devDependencies fields of the main object, so those
>>>>>> aren't even hard-coded in the SConstruct/SConscript.
>>>>>>
>>>>>> Again, if I fake out scons by have a 'node_modules/.built' File
>>>>>> target in the list and add a 'touch node_modules/.built' command to the
>>>>>> builder after 'npm install', all will work as expected because there is a
>>>>>> File target. In your example, things work because the second builder has a
>>>>>> file target and SCons will only run the first builder in your example when
>>>>>> foo does not exist, regardless of the state of foo-contents.txt in relation
>>>>>> to what is in .sconsign. That is the essence of the bug, if the target
>>>>>> directory exists and it is the only target, the builder will not run
>>>>>> regardless of the state of the sources.
>>>>>>
>>>>>> Manish
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 20, 2017 at 9:09 PM, William Blevins <
>>>>>> wblevins001 at gmail.com> wrote:
>>>>>>
>>>>>>> Manish,
>>>>>>>
>>>>>>> * Why does my example not work?
>>>>>>>
>>>>>>> SCons Comand Builder parameters are target(s), source(s),
>>>>>>> command(s): http://scons.org/doc/HTML/scon
>>>>>>> s-user.html#chap-builders-commands
>>>>>>>
>>>>>>> Foo is the target parameter, and foo-contents.txt is the source
>>>>>>> parameter. Targets explicitly depend on sources. In your example, directory
>>>>>>> foo depends on source file foo-contents.txt; target file copy does not
>>>>>>> depend on source file.
>>>>>>>
>>>>>>> * How do I fix this issue?
>>>>>>>
>>>>>>> 1. Call Command Builder correctly: Command('foo', 'foo-contents',
>>>>>>> 'mkdir $TARGET') which creates file dependency on directory, and then
>>>>>>> Command('foo/foo-contents.txt', 'foo-contents.txt', 'cp $SOURCE
>>>>>>> $TARGET') which creates file target dependency on file source.
>>>>>>>
>>>>>>> 2: [PREFERRED] Use the SCons FileSystem factory
>>>>>>> http://scons.org/doc/HTML/scons-user.html#chap-factories
>>>>>>>
>>>>>>> V/R,
>>>>>>> William
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Mar 20, 2017 at 1:47 PM, Manish Vachharajani <
>>>>>>> manishv at unbounded.systems> wrote:
>>>>>>>
>>>>>>>> I believe there is a bug in how up to date is computed for file
>>>>>>>> nodes and directory targets. Below is a SConstruct and run output
>>>>>>>> illustrating the bug.
>>>>>>>>
>>>>>>>> ##############
>>>>>>>> # SConstruct
>>>>>>>> env = Environment()
>>>>>>>> targets = env.Command('foo',
>>>>>>>> 'foo-contents.txt',
>>>>>>>> ['mkdir foo',
>>>>>>>> 'cp foo-contents.txt foo'])
>>>>>>>> env.Default(targets)
>>>>>>>>
>>>>>>>> ##############
>>>>>>>> # Shell log
>>>>>>>>
>>>>>>>> # Create the file that will be installed into foo/foo-contents.txt
>>>>>>>> $ echo "Hello, " >> foo-contents.txt
>>>>>>>> # All is well, scons runs the command builder as expected
>>>>>>>> $ scons
>>>>>>>>
>>>>>>>> scons: Reading SConscript files ...
>>>>>>>> scons: done reading SConscript files.
>>>>>>>> scons: Building targets ...
>>>>>>>> mkdir foo
>>>>>>>> cp foo-contents.txt foo
>>>>>>>> scons: done building targets.
>>>>>>>>
>>>>>>>> # All is well, the target is up to date as expected
>>>>>>>> $ scons
>>>>>>>>
>>>>>>>> scons: Reading SConscript files ...
>>>>>>>> scons: done reading SConscript files.
>>>>>>>> scons: Building targets ...
>>>>>>>> scons: `foo' is up to date.
>>>>>>>> scons: done building targets.
>>>>>>>>
>>>>>>>> # Uh-oh, source is updated but the target is considered up to date!
>>>>>>>> $ echo "World!" >> foo-contents.txt
>>>>>>>> $ scons
>>>>>>>>
>>>>>>>> scons: Reading SConscript files ...
>>>>>>>> scons: done reading SConscript files.
>>>>>>>> scons: Building targets ...
>>>>>>>> scons: `foo' is up to date.
>>>>>>>> scons: done building targets.
>>>>>>>>
>>>>>>>> For the bug to trigger, all targets for a builder must be
>>>>>>>> directories. The target directory must exist and the source file must
>>>>>>>> exist and be out of date due to a content update (resulting in a signature
>>>>>>>> mismatch).
>>>>>>>>
>>>>>>>> I think the problem is that is_up_to_date for File nodes only
>>>>>>>> checks the signatures of its children, which avoids this bug as long as one
>>>>>>>> of the builder targets is a File and not a Directory. However, I haven't
>>>>>>>> done enough analysis to be sure that this is the issue.
>>>>>>>>
>>>>>>>> A work around is to make sure that you have a file (or create a
>>>>>>>> dummy .built file) and include it as a target.
>>>>>>>>
>>>>>>>> This is showing up specifically for us when trying to calculate
>>>>>>>> dependencies for a package manager (npm) where we have no insight into the
>>>>>>>> contents of the resulting directory.
>>>>>>>>
>>>>>>>> Manish
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Scons-users mailing list
>>>>>>>> Scons-users at scons.org
>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>
>>
>
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20170321/d418d82d/attachment-0001.html>
More information about the Scons-dev
mailing list