[Scons-dev] Partial fix for mixed output with -j

[TOM TANNER (BLOOMBERG/ LONDON)]

I think that's pretty much what I've done:

1) Change posixy.py env_spawn, fork_span and spawnvpe_spawn to call exec_popen3 and exec_piped_fork

2) There is a small change in Job.py. This is to ensure the entire output from one particular build job goes to one place.

This means that if you have a job that does
    Copy(file1, '$SOURCE')
    [ 'some shell command $TARGET file1' ]
    Delete(file1)

all the output gets put together. I think that is the correct way to divide the output. And given that, I can't see where else you could put the changes.

The basic way the code works is:

   redirect job output to file
   process job
   take a lock
   output file to stdout
   release lock

I can't help thinking if this is done somewhere else, you'll still get output from any particular job intermingled still, just differently.

Another reason for doing this in Job is that I know at that point that the build is multi-threaded, and I didn't want to change the behaviour any for non-threaded builds. At times there's an advantage to seeing the output as it happens rather than after it has finished. Especially if something hangs.

I could have the class which I assign to stdout/stderr defined somewhere else, and do that small piece of work in the scons startup, but it seemed better to keep the changes to one file.

My python sadly is not at the level where I'd know how to override some of the stuff I've done at the user level rather than changing the sources.

----- Original Message -----
From: scons-dev at scons.org
To: scons-dev at scons.org
At:  6/22 18:10:26

Sorry let me explain.

I have a add on called Parts that has the goal of moving a number of improvements to SCons. Now that we have the HG move done I can start pushing stuff again.

In this code I do something similar to what you propose to clean up the output for -j builds. The way I did this was to modify the Spawn functions. This allowed me to do two different things.

1) map the output to different logger, or colorizers
2) group text together to avoid the annoying mixing of text that happens in a -j build.

What I am suggesting is that if you tweak the default Spawn logic, vs the Job manager this will be a lot easier.

The other think you probably will want to do is change the start up the code to remap stdout -> mapping function -> _stdout_ so you can easily map all other output such as custom pure python builders.

You can find the Parts at parts.tigris.org still. There are sample codes you can run to see what it does. Ideally it should just work by importing part in to your Sconstruct file.

Jason

-----Original Message-----
From: scons-dev-bounces at scons.org [mailto:scons-dev-bounces at scons.org] On Behalf Of TOM TANNER (BLOOMBERG/ LONDON)
Sent: Friday, June 22, 2012 11:59 AM
To: scons-dev at scons.org
Subject: Re: [Scons-dev] Partial fix for mixed output with -j

Parts?

My version of the SCons source has nothing called Parts, neither file nor class.

----- Original Message -----
From: scons-dev at scons.org
To: scons-dev at scons.org
At:  6/22 17:51:26

I implemented this in a spawn function in Parts, as I found this was the easiest place to do it. Make the default spawn function call subprocess.popen and remap those streams. 

Jason 

-----Original Message-----
From: scons-dev-bounces at scons.org [mailto:scons-dev-bounces at scons.org] On Behalf Of TOM TANNER (BLOOMBERG/ LONDON)
Sent: Friday, June 22, 2012 11:30 AM
To: scons-dev at scons.org
Subject: Re: [Scons-dev] Partial fix for mixed output with -j

I've moved the stderr/stdout interception into Job.py because it's easiest to make it behave as currently for non-parallel builds.

However there's no way I can get hold of the environment at that point.

This basically means that the 'spawn' functions in posix.py have been replaced with wrappers round the 'pspawn' functions.

However, it now behaves unchanged for non-parallel builds and the resulting code looks a bit nicer.

I looked at the windows piped spawn/exec spawn. I don't have anything equivalent I can try on windows, but the change should be to do the same thing. What would be nice though is to save the copying from tempfile to the scons stdout but I don't know well enough how windows does redirections.

This page: http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/redirection.mspx
sugggests that >> would work

I can put the diffs here next week if anyone is interested (I can't use bitbucket from work :-( )
 


----- Original Message -----
From: scons-dev at scons.org
To: scons-dev at scons.org
At:  6/22 12:43:27

I'm also interested in this improvement. If there's anything I can help with, send me an email

Bye
Roberto



> -----Original Message-----

> From: scons-dev-bounces at scons.org [mailto:scons-dev- 

> bounces at scons.org] On Behalf Of TOM TANNER (BLOOMBERG/ LONDON)

> Sent: venerdì 22 giugno 2012 12:40

> To: scons-dev at scons.org

> Subject: Re: [Scons-dev] Partial fix for mixed output with -j

> 

> I'd love to but our systems are very locked down.

> 

> I'll see how far I can get.

> 

> (and as you can see our email client is top-reply)

> 

> ----- Original Message -----

> From: scons-dev at scons.org

> To: scons-dev at scons.org

> At:  6/21 17:02:49

> 

> On 20 June 2012, TOM TANNER (BLOOMBERG/ LONDON) said:

> > OK, this is just a 'possible' at the moment. I'm not sure it's in 

> > the right place (debug output still gets mixed up), it's possible 

> > this should be part of the Task object, and it happens even with no 

> > -j setting which is silly (but I don't know how to check for that), 

> > but it

works

> wonderfully for parallel builds.

> >

> > However, I'd like some comments.

> >

> > Here's the diff output (a bit split up and mangled because I was 

> > having problems with gmane)

> >

> > 109,112d109

> > < from threading import Lock, local

> > < import tempfile

> > < import sys

> > < import os

> > 114,145d110

> > < class Threaded_Output_Redirector:

> > <     """ The general idea of this class is to allow redirection of

output to a

> 

> Please send unified diffs! Easiest way: clone the scons repo from 

> bitbucket.org (https://bitbucket.org/scons/scons), modify the code, 

> and

use

> "hg diff" to generate your patch.

> 

> It would be great to see less confusing output from parallel builds.

> Thanks for starting this work!

> 

>        Greg

> _______________________________________________

> Scons-dev mailing list

> Scons-dev at scons.org

> http://two.pairlist.net/mailman/listinfo/scons-dev

> 

> _______________________________________________

> Scons-dev mailing list

> Scons-dev at scons.org

> http://two.pairlist.net/mailman/listinfo/scons-dev

> 





_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev

_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev
_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev

_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev
_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev