[Scons-dev] Multiprocessing instead of threads?

[Jason Kenny]

Ideally threads in python work great, given the code in the threaded code is a c-extension and or some subprocess or AIO like call. Pure thread python code does not scale because of the GIL.

Given my work At Intel on threaded/MPI systems. Multiprocessing is always slower then than threading. There is more overhead and work to be done to communicate a simple state change that from a CPU point of view is magnitudes slower than what would happen in a thread. Certain patterns scale well with multiprocessing library that allow a system to go across different machine. These pattern work well in python multiprocessing library as it allow for reduced communication and scaling of the CPU resources on a given box. SCons however has a global DAG state that has to be shared. The one way this could work is if the tasking system could use it. The issues is how we share state on what is building or finished, output text across different processes, dealing with pure python builders, etc. 

For me is not that this could not be useful… the issue is that it is not as simple as one would hope.

Jason



From: Russel Winder
Sent: Sunday, November 22, 2015 3:29 PM
To: SCons developer list
Cc: Sarah Mount
Subject: Re: [Scons-dev] Multiprocessing instead of threads?


On Fri, 2015-11-20 at 19:51 -0800, Bill Deegan wrote:
> Greetings,
> 
> Would there be value in using python's multiprocessing package
> instead of
> threads?

As any consultant will tell you: it depends. Threads in Python suck due
to the GIL. Conversely, threads work fine for concurrency due to the
GIL, assuming you have no interest whatsoever in parallelism.

Parallelism in Python currently requires use of multiprocessing. It
works but there is an overhead. And the synchronization infrastructure
of multiprocessing is really rather naïve and under-performant.

Sarah Mount has needed a far better low level process communication
synchronization and communication infrastructure for Python-CSP. I said
I would investigate. To date I have failed. Perhaps in 2016-02…

> Anyone have experience using this package?

Yes.

!

-- 
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder at ekiga.net
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel at winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20151123/d2c8a767/attachment.html>