[Scons-dev] multi-thread tasks

[William Blevins]

James,

As a fellow user, I think SCons could use something like this. I have had
similar issues in the past, but I generally just let the O/S handle the
additional threads. To my knowledge, jobs and builder/action invocations
are 1:1.

I would think that a simple property is appropriate, but there may be some
edge cases to consider.

What happens if the action requires more threads than are available? Does
it never execute?
How will SCons optimize executing a task that requires less threads versus
waiting for X-threads to be available?

V/R,
William

On Mon, Sep 12, 2016 at 6:06 PM, James Corey <jc-scons at neniam.net> wrote:

> I have a need to allow some external process actions to be scheduled which
> themselves run with multiple threads, and I would like to adjust scons
> to take this into account when running jobs in parallel (e.g. num_jobs=4).
>
> Initial tests suggest that the following crude hack is effective at
> limiting
> the load:
>
> diff -r 6b54eedc08ac src/engine/SCons/Job.py
> --- a/src/engine/SCons/Job.py   Sun Mar 02 13:52:56 2014 -0500
> +++ b/src/engine/SCons/Job.py   Mon Sep 12 14:39:50 2016 -0700
> @@ -391,9 +391,14 @@
>                          task.postprocess()
>                      else:
>                          if task.needs_execute():
> +                            try:
> +                                nthreads = task.node.threads_needed
> +                            except (NameError, AttributeError):
> +                                nthreads = 1
> +                            task.threads_used = nthreads
>                              # dispatch task
>                              self.tp.put(task)
> -                            jobs = jobs + 1
> +                            jobs = jobs + task.threads_used
>                          else:
>                              task.executed()
>                              task.postprocess()
> @@ -404,7 +409,7 @@
>                  # back and put the next batch of tasks on the queue.
>                  while True:
>                      task, ok = self.tp.get()
> -                    jobs = jobs - 1
> +                    jobs = jobs - task.threads_used
>
>                      if ok:
>                          task.executed()
>
> I would like to clean this up, limit threads_used to the value of
> maxjobs, and optionally wait if necessary for the given number of job
> slots to become available, as well as establish a proper method for an
> action to set this value.  Naturally I'd base it on the tip and write
> tests, with the intention that the quality of the patch be sufficient
> for consideration into upstream, should there be interest.
>
> Since I am a beginner at scons, I ask the following specific
> questions:
>
> * Would this type of functionality be welcome?
> * Should I generalize it to allow for other resource metering (e.g.
> memory)?
> * Is a simple attribute appropriate, or should it be a callback?
> * Is it more proper to put such on the node or the task?
> * How do the tasks of Job.py relate to the Actions of the SConscript? 1:1?
> * How should a user override the default of 1 thread needed, from a
> SConscript?
> * Any other advice?
>
> Assuming there is any interest, should I put together a speculative
> patch to a fork and submit a pull request?  Or should I try to design it
> via discussion here first, and then submit a pull request?
>
> Thanks in advance.
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20160913/11bda914/attachment-0001.html>