[Scons-dev] Multiprocessing instead of threads?

[William Blevins]

On Sat, Nov 21, 2015 at 4:13 PM, Jason Kenny <dragon512 at live.com> wrote:

> I have thought about it myself
>
>
>
> It might have value if we can separate the task logic in some way to allow
> a way to reduce data syncing on the node. We have to remember that we have
> a big global data structure that we use to control the build state. As item
> get done we do more work in a safe way. In a thread model this works great
> as we are in the same process space. In multithreading we have to sync this
> data across processes. This is a magnitudes slower than the threaded case.
>

I would actually like to see evidence for this claim because it may not be
true at all depending on how much SCons spends in GIL contention. One issue
here is that SCons internal operations are not parallelized, so maybe
multithreading buys us little, but without actually conducting some sort of
test, I would be wary of just scrapping the idea.



> So the trick is can we use multiprocessing for us to speed up python based
> tasked ( vs the ones that are commandline only).  I don’t have a clear
> answer but have thought of two ideas..
>
>
>
> 1)      Can we provide a better interface to making builders or actions
> of pure python code so that we could make a temp script that we call as a
> separate python process. This would allow us to reduce contention on the
> GIL to logic related to build state, vs builder logic
>

This sounds like a good idea if internally we had a process API that could
use Threads or Subprocesses interchangeably.


> 2)      Can we do something to change the task logic to reduce
> communication to a master “process” that keeps state and delegates out
> actions to a sperate process
>
>
>
> If we would like to make an improvement I am very sure that if we made the
> build off of builders vs nodes this would greatly speed up the build logic.
> This was the basic premise of the Greg Noels TNG work. I can say more this
> in a different thread .. don’t want to side track this thread.
>
>
>
> I think we should also note make more processes will require more memory
> overall, vs threads. Howe much depends on details that are not completely
> clear to me at the moment.
>
>
>
> Jason
>
>
>
>
>
> Sent from Mail <http://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
>
>
>
> *From: *Bill Deegan
> *Sent: *Friday, November 20, 2015 9:52 PM
> *To: *SCons developer list
> *Subject: *[Scons-dev] Multiprocessing instead of threads?
>
>
>
>
>
> Greetings,
>
> Would there be value in using python's multiprocessing package instead of
> threads?
>
> Anyone have experience using this package?
>
> -Bill
>
>
>
>
>
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20151121/75aa3dc7/attachment.html>