[Scons-dev] scons daemon
Bill Deegan
bill at baddogconsulting.com
Thu Jan 14 19:41:05 EST 2016
Ben,
So the location of SConstruct/SConscript doesn't have to imply the location
of the source/target files at all.
(Yes I've seen/worked on such a system)
So any such assumption is not safe. Also keep in mind a single SConscript
can specify files in many locations.
Doubling memory.. that's gonna be a deal killer for big builds. (in some
cases scons can use 1G+ of RAM).
Which is the type of build which might benefit most from your type of
improvement.
Any documentation would go in the manpage and/or users guide whose source
is in the SCons repo/source tree and is written in DocBook.
This is a good starting place for anyone who wants to contribute code to
the core (and has sections on testing and documentation)
https://bitbucket.org/scons/scons/wiki/DeveloperGuide
Although your implementation is interesting, I'm not sure it's something we
can accept into the core. (for the reasons listed above).
Can I suggest you revisit a) running two processes and instead move the
logic into interactive mode (or some extension thereof), and b) relax your
concerns about files being changed while the build is happening and/or
perhaps the "paranoid mode" decider we've spoken about in other threads
(William Blevins has a patch for this, though how/when we'd pull such
functionality into the core is still in discussion)
Anyone other core developers care to add? I may be missing something and/or
have a misunderstanding which makes my commentary not definitive.
-Bill
On Thu, Jan 14, 2016 at 3:24 PM, Schleimer, Ben <bensch128 at yahoo.com> wrote:
> Hi Bill,
> I decided to have the daemon copy the source tree because I wanted to
> make sure that once a background build starts, it can complete or fail
> without additional source file changes.
>
> About copying the directory: I guess that if the scons scan can correctly
> locate every source file and dependency, then only the source and
> SConscripts/SConstruct files need to be copied to the temp directory.
>
> I did assume that the SConstruct file was in the top level of each source
> tree. It could be changed that the SConscript files are used to indicate
> the top level instead. I didn't consider that a usage case.
>
> About memory usage: It looks like the memory usage roughly doubles because
> the parent process retains the data from the scan and then the child
> process retains the memory during it's build.
>
> My memory stats:
> Num source_files=107, num target_files=21, num SConscripts=7
> 28.772MB res for parent process
> 24.5MB res for child process
>
> I will try to add some tests.
> Where do you suggest putting them?
>
> Can you give me a stub in the Wiki to add documentation?
>
> Sincerely,
> Ben
>
>
> On Thursday, January 14, 2016 1:55 PM, Bill Deegan <
> bill at baddogconsulting.com> wrote:
>
>
>
> A few thoughts..
>
> Seems like you are assuming that the SConstruct is in the top directory of
> a source tree.
> This is not a safe assumption. I've seen builds where the SConscripts are
> scattered over several filesystems (building a project from several shared
> source repositories).
> Can you add some tests?
> And documentation?
> Any comments on the memory footprint of running this on a larger source
> tree? (where SCons is using a fair amount of memory already)
>
> Is it really necessary to make a full copy of the source tree to a temp
> dir?
> How would this impact the use of VariantDir?
>
> -Bill
>
> On Thu, Jan 14, 2016 at 11:56 AM, Schleimer, Ben via Scons-dev <
> scons-dev at scons.org> wrote:
>
> Hi,
> I finished up the watch mode as best as I could and posted a pull
> request for it here
> https://bitbucket.org/scons/scons/pull-requests/289/watch-mode/diff
>
> I am hoping that people will try it out for their incremental builds and
> see it if is working for them or not and if it helps their iteration time.
> I tried to minimize changes in the core scons code. The only major change
> I had to make was to add
> SCons.Script._SConscript.global_list_of_processed_sconscipt_files
>
>
> This is used to track where the SConscript files are so that a
> commonprefix can be established for all of the SConstruct files.
>
> Sincerely
> Ben
>
>
> On Tuesday, December 29, 2015 8:23 PM, "Schleimer, Ben via Scons-dev" <
> scons-dev at scons.org> wrote:
>
>
> >
> >
> >Hi William,
> >
> > The non-interactive timings are not affected by the patch.
> >
> >
> >Besides, I'm giving up on that patch and writing a completely separate
> Watch mode. It'll be started by "scons --watch" and it'll be a background
> compilation triggered by file changes.
> >
> >
> >
> >Once I get something simple working, I'll post the pull request so people
> can criticize it and hopefully improve it so that it works in a general way.
> >
> >
> >Sincerely
> >Ben
> >
> >
> >
> >
> >On Tuesday, December 29, 2015 7:25 PM, William Blevins <
> wblevins001 at gmail.com> wrote:
> >
> >
> >>
> >>
> >>No, I just wanted the profile from before the patch and after the patch
> on the same code. Unless I misunderstood your output from last time, you
> only supplied the information regarding the patched version correct?
> >>
> >>
> >>V/R,
> >>William
> >>
> >>
> >>On Tue, Dec 29, 2015 at 11:25 PM, Schleimer, Ben via Scons-dev <
> scons-dev at scons.org> wrote:
> >>
> >>Hi,
> >>>
> >>>
> >>>
> >>>> Try running with --debug=time
> >>>
> >>>
> >>>Did you want to see all of the build output?
> >>>
> >>>
> >>>Ben
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>On Tuesday, December 29, 2015 2:57 PM, Bill Deegan <
> bill at baddogconsulting.com> wrote:
> >>>
> >>>
> >>>>
> >>>>
> >>>>Try running with --debug=time
> >>>>
> >>>>
> >>>>--debug=time
> >>>>Prints various time profiling information:
> the time spent executing each individual build command;
> the total build time (time SCons ran from beginning to end);
> the total time spent reading and executing SConscript files;
> the total time spent SCons itself spend running
> (that is, not counting reading and executing SConscript files);
> and both the total time spent executing all build commands
> and the elapsed wall-clock time spent executing those build commands.
> (When scons is executed without the -j option,
> the elapsed wall-clock time will typically
> be slightly longer than the total time spent
> executing all the build commands,
> due to the SCons processing that takes place
> in between executing each command.
> When scons is executed with the -j option,
> and your build configuration allows good parallelization,
> the elapsed wall-clock time should
> be significantly smaller than the
> total time spent executing all the build commands,
> since multiple build commands and
> intervening SCons processing
> should take place in parallel.)
> >>>>-Bill
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>On Tue, Dec 29, 2015 at 2:47 PM, William Blevins <
> wblevins001 at gmail.com> wrote:
> >>>>
> >>>>I guess my point here was that I wanted to know where the time savings
> was happening since SCons does split out some process timings.
> >>>>>
> >>>>>
> >>>>>I just wanted to see a regular build timing with and without your
> patch.
> >>>>>
> >>>>>
> >>>>>V/R,
> >>>>>William
> >>>>>
> >>>>>
> >>>>>On Tue, Dec 29, 2015 at 7:07 PM, Schleimer, Ben via Scons-dev <
> scons-dev at scons.org> wrote:
> >>>>>
> >>>>>Hi William,
> >>>>>>
> >>>>>>
> >>>>>>>Can you give the time saving using --debug=time for -j1 and -j4?
> >>>>>>>
> >>>>>>
> >>>>>> Sure,
> >>>>>>
> >>>>>>regular scons build (scons -jX --debug=time)
> >>>>>>a clean build with -j1 is:
> >>>>>>Total build time: 20.901390 seconds
> >>>>>>Total SConscript file execution time: 0.149543 seconds
> >>>>>>Total SCons execution time: 0.444850 seconds
> >>>>>>Total command execution time: 20.306997 seconds
> >>>>>>
> >>>>>>
> >>>>>>an incremental build with -j1 is:
> >>>>>>Total build time: 2.439754 seconds
> >>>>>>Total SConscript file execution time: 0.151794 seconds
> >>>>>>Total SCons execution time: 0.337350 seconds
> >>>>>>Total command execution time: 1.950610 seconds
> >>>>>>
> >>>>>>
> >>>>>>a clean build with -j4 is:
> >>>>>>Total build time: 7.158301 seconds
> >>>>>>Total SConscript file execution time: 0.153021 seconds
> >>>>>>Total SCons execution time: 0.107556 seconds
> >>>>>>Total command execution time: 6.897724 seconds
> >>>>>>
> >>>>>>
> >>>>>>an incremental build with -j4 is:
> >>>>>>Total build time: 2.368260 seconds
> >>>>>>Total SConscript file execution time: 0.151281 seconds
> >>>>>>Total SCons execution time: 0.183930 seconds
> >>>>>>Total command execution time: 2.033049 seconds
> >>>>>>
> >>>>>>
> >>>>>>interactive build (scons --interactive -jX --debug=time)
> >>>>>>a clean build with -j1:
> >>>>>>time to do a build = 19.983217001 sec
> >>>>>>
> >>>>>>an incremental build with -j1:
> >>>>>>time to do a build = 2.2111852169 sec
> >>>>>>
> >>>>>>a clean build with -j4:
> >>>>>>time to do a build = 6.93614792824 sec
> >>>>>>
> >>>>>>an incremental build with -j4:
> >>>>>>time to do a build = 2.17438697815 sec
> >>>>>>
> >>>>>>
> >>>>>>It's consistently 0.2 seconds faster with the interactive build.
> >>>>>>Not a huge amount but I'm not using that many SConscript files.
> >>>>>>(5 SConscript files and 2 SConstruct files)
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>Cheers
> >>>>>>
> >>>>>>
> >>>>>>Ben
> >>>>>>_______________________________________________
> >>>>>>Scons-dev mailing list
> >>>>>>Scons-dev at scons.org
> >>>>>>https://pairlist2.pair.net/mailman/listinfo/scons-dev
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>_______________________________________________
> >>>>>Scons-dev mailing list
> >>>>>Scons-dev at scons.org
> >>>>>https://pairlist2.pair.net/mailman/listinfo/scons-dev
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>_______________________________________________
> >>>Scons-dev mailing list
> >>>Scons-dev at scons.org
> >>>https://pairlist2.pair.net/mailman/listinfo/scons-dev
> >>>
> >>>
> >>
> >>
> >>
> >
> >_______________________________________________
> >Scons-dev mailing list
> >Scons-dev at scons.org
> >https://pairlist2.pair.net/mailman/listinfo/scons-dev
> >
> >
> >
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20160114/5fdc779d/attachment-0001.html>
More information about the Scons-dev
mailing list