[Scons-dev] Multiprocessing instead of threads?

[Neal Becker]

Jason Kenny wrote:

> I have thought about it myself
> 
> It might have value if we can separate the task logic in some way to allow
> a way to reduce data syncing on the node. We have to remember that we have
> a big global data structure that we use to control the build state. As
> item get done we do more work in a safe way. In a thread model this works
> great as we are in the same process space. In multithreading we have to
> sync this data across processes. This is a magnitudes slower than the
> threaded case. So the trick is can we use multiprocessing for us to speed
> up python based tasked ( vs the ones that are commandline only).  I don’t
> have a clear answer but have thought of two ideas..

What about shared memory?

> 
> 1) Can we provide a better interface to making builders or actions of pure
> python code so that we could make a temp script that we call as a separate
> python process. This would allow us to reduce contention on the GIL to
> logic related to build state, vs builder logic 2) Can we do something to
> change the task logic to reduce communication to a master “process” that
> keeps state and delegates out actions to a sperate process
> 
> If we would like to make an improvement I am very sure that if we made the
> build off of builders vs nodes this would greatly speed up the build
> logic. This was the basic premise of the Greg Noels TNG work. I can say
> more this in a different thread .. don’t want to side track this thread.
> 
> I think we should also note make more processes will require more memory
> overall, vs threads. Howe much depends on details that are not completely
> clear to me at the moment.
> 
> Jason
> 
> 
> Sent from Mail for Windows 10
> 
> 
> 
> From: Bill Deegan
> Sent: Friday, November 20, 2015 9:52 PM
> To: SCons developer list
> Subject: [Scons-dev] Multiprocessing instead of threads?
> 
> 
> Greetings,
> Would there be value in using python's multiprocessing package instead of
> threads? Anyone have experience using this package?
> -Bill