[Scons-dev] Multiprocessing instead of threads?

[William Blevins]

On Mon, Nov 23, 2015 at 7:57 PM, Neal Becker <ndbecker2 at gmail.com> wrote:

> Jason Kenny wrote:
>
> > I have thought about it myself
> >
> > It might have value if we can separate the task logic in some way to
> allow
> > a way to reduce data syncing on the node. We have to remember that we
> have
> > a big global data structure that we use to control the build state. As
> > item get done we do more work in a safe way. In a thread model this works
> > great as we are in the same process space. In multithreading we have to
> > sync this data across processes. This is a magnitudes slower than the
> > threaded case. So the trick is can we use multiprocessing for us to speed
> > up python based tasked ( vs the ones that are commandline only).  I don’t
> > have a clear answer but have thought of two ideas..
>
> What about shared memory?
>

Threads automatically benefit here as they are shared memory by default,
but with extra work, we could also do shared memory with subprocesses.


>
> >
> > 1) Can we provide a better interface to making builders or actions of
> pure
> > python code so that we could make a temp script that we call as a
> separate
> > python process. This would allow us to reduce contention on the GIL to
> > logic related to build state, vs builder logic 2) Can we do something to
> > change the task logic to reduce communication to a master “process” that
> > keeps state and delegates out actions to a sperate process
> >
> > If we would like to make an improvement I am very sure that if we made
> the
> > build off of builders vs nodes this would greatly speed up the build
> > logic. This was the basic premise of the Greg Noels TNG work. I can say
> > more this in a different thread .. don’t want to side track this thread.
> >
> > I think we should also note make more processes will require more memory
> > overall, vs threads. Howe much depends on details that are not completely
> > clear to me at the moment.
> >
> > Jason
> >
> >
> > Sent from Mail for Windows 10
> >
> >
> >
> > From: Bill Deegan
> > Sent: Friday, November 20, 2015 9:52 PM
> > To: SCons developer list
> > Subject: [Scons-dev] Multiprocessing instead of threads?
> >
> >
> > Greetings,
> > Would there be value in using python's multiprocessing package instead of
> > threads? Anyone have experience using this package?
> > -Bill
>
>
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20151123/d2380c2b/attachment.html>