[Scons-dev] Multiprocessing instead of threads?

[Jason Kenny]

I have thought about it myself

It might have value if we can separate the task logic in some way to allow a way to reduce data syncing on the node. We have to remember that we have a big global data structure that we use to control the build state. As item get done we do more work in a safe way. In a thread model this works great as we are in the same process space. In multithreading we have to sync this data across processes. This is a magnitudes slower than the threaded case. So the trick is can we use multiprocessing for us to speed up python based tasked ( vs the ones that are commandline only).  I don’t have a clear answer but have thought of two ideas..

1) Can we provide a better interface to making builders or actions of pure python code so that we could make a temp script that we call as a separate python process. This would allow us to reduce contention on the GIL to logic related to build state, vs builder logic
2) Can we do something to change the task logic to reduce communication to a master “process” that keeps state and delegates out actions to a sperate process

If we would like to make an improvement I am very sure that if we made the build off of builders vs nodes this would greatly speed up the build logic. This was the basic premise of the Greg Noels TNG work. I can say more this in a different thread .. don’t want to side track this thread. 

I think we should also note make more processes will require more memory overall, vs threads. Howe much depends on details that are not completely clear to me at the moment. 

Jason


Sent from Mail for Windows 10



From: Bill Deegan
Sent: Friday, November 20, 2015 9:52 PM
To: SCons developer list
Subject: [Scons-dev] Multiprocessing instead of threads?


Greetings,
Would there be value in using python's multiprocessing package instead of threads?
Anyone have experience using this package?
-Bill


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20151121/435f8a7d/attachment-0001.html>