[Scons-dev] SCons with shared resource

[Brian Fitzgerald]

One answer is to use env.SideEffect with a dummy target.
	
For example, it's best to run spinning-platter-media linking serialized. We have this in our SConscript (enabled only for magnetic disks, not for SSDs, mind you):

noParallel = []

def NoParallelLink(env, disable = True):
  global noParallel

  if disable:
    env.SideEffect("#src/dummy_no_parallel", noParallel)
  noParallel = []

and then in builders that want something serialized, we add to the noParallel list.

  dll = env.SharedLibrary(target = dest, source = source + obj)
  noParallel.append(dll)

and then finally, at the end of the SConscript, we call the NoParallelLink function (which is attached to our environment for historical reasons, probably because the original author thought that was the best way to do it).

if GetOption('serialld'):
  env.BnetNoParallelLink()

Basically, the SideEffect function says "the commands in this list can't be run at the same time". We could have actually implemented it as serial calls to SideEffect, because it's cumulative, it's just the way we chose to do it.

-----Original Message-----
From: scons-dev-bounces at scons.org [mailto:scons-dev-bounces at scons.org] On Behalf Of Kenny, Jason L
Sent: Wednesday, November 20, 2013 3:34 PM
To: scons-dev at scons.org
Subject: Re: [Scons-dev] SCons with shared resource

Technically all tasks in SCons run in parallel, thanks to a GIL. What run in parallel are the subprocess calls to a command line ( as these turn into waits from the python point of view). The easy way to do what you suggest its to add locks around this shared data to force the exeution of one task at a time.

Jason

-----Original Message-----
From: scons-dev-bounces at scons.org [mailto:scons-dev-bounces at scons.org] On Behalf Of Alex.Burton at csiro.au
Sent: Wednesday, November 20, 2013 5:03 PM
To: scons-dev at scons.org
Subject: [Scons-dev] SCons with shared resource

Hi,

I am successfully using scons for batch processing of simulation work.

Unfortunately some of the tasks share resources that prevent them being executed in parallel.
Other tasks are can be run in parallel.

This means that I need to run scons in single processor mode, meaning some tasks are unnecessarily serialised.

I am considering modifying the scons code to allow specification of a shared resource/ mutex for certain tasks.

Does anyone have any advise as to how this is best implemented ?

Thanks,

Alex

Alex Burton
Research Engineer - Solar Thermal - Energy Technology - CSIRO
+61 2 4960 6110
+61 416 594 752


_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev
_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev