[Scons-dev] multi-thread tasks

[Bill Deegan]

James,

Looks good.
Dirk has a query in the pull request.
If you can resolve his query we can probably merge this fairly quickly.
Kudos on getting both tests and adding to CHANGES.txt.. ;)

-Bill

On Thu, Sep 15, 2016 at 11:47 PM, James Corey <jc-scons at neniam.net> wrote:

> OK, I finished the code.  User controls behavior by specifying
> threads_needed attribute on Action.  Value is clipped at num_jobs.
> Tasks continue to be run in order--no attempt is made to re-order or
> schedule smaller jobs while waiting for available slots, since we
> don't know how long they would take.  It is fair, and we can't do
> better without complicated packing and prediction.  I think it won't
> be an issue in practice.
> Patch includes two tests and updated documentation and CHANGES.txt.
>
> Pull request is https://bitbucket.org/scons/scons/pull-requests/355
>
> Thanks for feedback.
>
> On Mon, Sep 12, 2016 at 9:22 PM, William Blevins <wblevins001 at gmail.com>
> wrote:
> > James,
> >
> > As a fellow user, I think SCons could use something like this. I have had
> > similar issues in the past, but I generally just let the O/S handle the
> > additional threads. To my knowledge, jobs and builder/action invocations
> are
> > 1:1.
> >
> > I would think that a simple property is appropriate, but there may be
> some
> > edge cases to consider.
> >
> > What happens if the action requires more threads than are available?
> Does it
> > never execute?
> > How will SCons optimize executing a task that requires less threads
> versus
> > waiting for X-threads to be available?
> >
> > V/R,
> > William
> >
> > On Mon, Sep 12, 2016 at 6:06 PM, James Corey <jc-scons at neniam.net>
> wrote:
> >>
> >> I have a need to allow some external process actions to be scheduled
> which
> >> themselves run with multiple threads, and I would like to adjust scons
> >> to take this into account when running jobs in parallel (e.g.
> num_jobs=4).
> >>
> >> Initial tests suggest that the following crude hack is effective at
> >> limiting
> >> the load:
> >>
> >> diff -r 6b54eedc08ac src/engine/SCons/Job.py
> >> --- a/src/engine/SCons/Job.py   Sun Mar 02 13:52:56 2014 -0500
> >> +++ b/src/engine/SCons/Job.py   Mon Sep 12 14:39:50 2016 -0700
> >> @@ -391,9 +391,14 @@
> >>                          task.postprocess()
> >>                      else:
> >>                          if task.needs_execute():
> >> +                            try:
> >> +                                nthreads = task.node.threads_needed
> >> +                            except (NameError, AttributeError):
> >> +                                nthreads = 1
> >> +                            task.threads_used = nthreads
> >>                              # dispatch task
> >>                              self.tp.put(task)
> >> -                            jobs = jobs + 1
> >> +                            jobs = jobs + task.threads_used
> >>                          else:
> >>                              task.executed()
> >>                              task.postprocess()
> >> @@ -404,7 +409,7 @@
> >>                  # back and put the next batch of tasks on the queue.
> >>                  while True:
> >>                      task, ok = self.tp.get()
> >> -                    jobs = jobs - 1
> >> +                    jobs = jobs - task.threads_used
> >>
> >>                      if ok:
> >>                          task.executed()
> >>
> >> I would like to clean this up, limit threads_used to the value of
> >> maxjobs, and optionally wait if necessary for the given number of job
> >> slots to become available, as well as establish a proper method for an
> >> action to set this value.  Naturally I'd base it on the tip and write
> >> tests, with the intention that the quality of the patch be sufficient
> >> for consideration into upstream, should there be interest.
> >>
> >> Since I am a beginner at scons, I ask the following specific
> >> questions:
> >>
> >> * Would this type of functionality be welcome?
> >> * Should I generalize it to allow for other resource metering (e.g.
> >> memory)?
> >> * Is a simple attribute appropriate, or should it be a callback?
> >> * Is it more proper to put such on the node or the task?
> >> * How do the tasks of Job.py relate to the Actions of the SConscript?
> 1:1?
> >> * How should a user override the default of 1 thread needed, from a
> >> SConscript?
> >> * Any other advice?
> >>
> >> Assuming there is any interest, should I put together a speculative
> >> patch to a fork and submit a pull request?  Or should I try to design it
> >> via discussion here first, and then submit a pull request?
> >>
> >> Thanks in advance.
> >> _______________________________________________
> >> Scons-dev mailing list
> >> Scons-dev at scons.org
> >> https://pairlist2.pair.net/mailman/listinfo/scons-dev
> >
> >
> >
> > _______________________________________________
> > Scons-dev mailing list
> > Scons-dev at scons.org
> > https://pairlist2.pair.net/mailman/listinfo/scons-dev
> >
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20160916/2aa00687/attachment.html>