[Scons-dev] SCons for distributed builds?

[Russel Winder]

On Thu, 2018-04-05 at 16:18 +0000, Akins, Christopher (GE Aviation, Non-GE,
US) wrote:
> 
[…]
> I'm trying to modernize the process, and SCons has come favorably to my
> attention. Due to the lengthy and memory-intensive simulations, though, it
> would be useful if we could use the same build system to assign tasks to
> different nodes in a cluster. I'm willing to sink some time into making
> the necessary mods myself - for fault-tolerance, retries, etc - but I
> thought I'd ask if this had come up in the past. If it hasn't, would there
> be interest in having a distributed feature, or would it be more of a
> private fork?

Off the top of my head:

I would keep the decision making all on a single master node. SCons is
controlled by an acyclic directed graph and whilst there is Hazelcast etc.
Python is a single-threaded, single CPU progrmming language at its heart:
keeping data structures in a single PVM is the best way forward.

Distributing discrete, self-contained tasks which are essentially pure
functions makes a great deal of sense for seriously big builds. I did some
Python and SCons training for a company in 2007 who had a not dissimilar
problem to the one I think is being described here, and it worked very well.
Though I suspect Parts would have helped a lot and saved a lot of hand
crafting.

For distribution, 0MQ and a dedicated application level communications
protocol may well help.

Traditionally, Python has not been the best language for handling
asynchronous activity, Go or Java (or Scala, Kotlin, Groovy, etc) were the
"go to" languages since they have threadpools and tasks so as to avoid
messing around with single threaded event loops and callbacks.  I haven't
plaued with the async/await in Python 3.5+ nor asyncio, but it is a single
threaded event loop, and I am a blocked task in a thread pool sort of person
as "callback hell" is not where I want to go. Now that SCons is a Python 3
system, this stuff should be tried.

I wouldn't start anything new in this area until after a conversation with
Jason about Parts. 

-- 
Russel.
=========================================
Dr Russel Winder     t:+44 20 7585 2200
41 Buckmaster Road   m:+44 7770 465 077
London SW11 1EN, UK  w: www.russel.org.uk
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20180405/816339fd/attachment.sig>