[Scons-dev] Fix async subprocess execution with proper handling of std* streams

[Kenny, Jason L]

Hi Anatoly,

Attached is a word document with the overview of the deign I have with thoughts on some good and bad of it. Hopefully I proofed the document enough, document writing is not my strong suit. Anyways let me know if you have any other questions, etc.. on the design.

Eugene who has been helping with Parts work inside thinks he fixed a Scons/Parts rebuild issues we have been dealing with. I testing it out, if it looks good I will push this drop in a few days.

Thanks for looking at this!
Jason

-----Original Message-----
From: scons-dev-bounces at scons.org [mailto:scons-dev-bounces at scons.org] On Behalf Of anatoly techtonik
Sent: Tuesday, February 18, 2014 10:35 AM
To: SCons developer list
Subject: Re: [Scons-dev] Fix async subprocess execution with proper handling of std* streams

Thanks. That would be great. ;)

On Tue, Feb 18, 2014 at 5:21 PM, Kenny, Jason L <jason.l.kenny at intel.com> wrote:

> I will get it pushed… life getting in the way.

>

>

>

> From: scons-dev-bounces at scons.org [mailto:scons-dev-bounces at scons.org] 

> On Behalf Of Eugene Leskinen

> Sent: Tuesday, February 18, 2014 7:03 AM

> To: SCons developer list

> Subject: Re: [Scons-dev] Fix async subprocess execution with proper 

> handling of std* streams

>

>

>

> Hi Anatoly,

>

>

>

> The code supporting I/O redirection in Parts is located in 

> parts/part_logger.py file. Look at 

> http://parts.tigris.org/source/browse/parts/trunk/parts/parts/part_log

> ger.py?revision=486&view=markup

>

>

>

> The code uses Python threading module to create threads for each 

> stream and then runs a loop in every thready like your code does:

>

>

>

>         line = ' '

>

>         try:

>

>             while line:

>

>                 line = self.pipein.readline()

>

>                 if line:

>

>                     self.output.WriteStream(self.taskId, 

> self.streamId,

> line)

>

>         except:

>

>             # There was an error... that shouldn't happen, but still it did.

> So we report it

>

>             # to the caller and close our pipe end so that spawned 

> program won't block

>

>             self.error = traceback.format_exc()

>

>             self.pipein.close()

>

>             self.pipein = None

>

>

>

> Please note that public repo is a bit out-of-date. I hope Jason will 

> find time to pull up-to-date version from Intel internal repository.

>

>

>

> 2014-02-18 16:30 GMT+04:00 anatoly techtonik <techtonik at gmail.com>:

>

> On Wed, Feb 12, 2014 at 6:25 PM, Kenny, Jason L 

> <jason.l.kenny at intel.com>

> wrote:

>>> Fix async subprocess execution with proper handling of std* streams.

>>

>> So I fixed the issue with output in Parts. I not saying it could not 

>> be clean up some more. However it solves the output issues, and adds 

>> coloring ( which seems to need some fixing in certain cases.. ie mac 

>> mostly) and logging support. It might not be too hard for someone to 

>> move the code over to SCons and integrated its usage into SCons.

>

> Do you have, by any chance a description of output issues?

> It seems that I've lost my list during the last Windows partition crash.

>

> Also, can you pinpoint the parts in Parts that are relevant to the issues?

> I may have time to port them over to SCons.

>

> My approach

> https://bitbucket.org/techtonik/absub

> to make subprocess asynchronous by moving processing of input/output 

> streams into separate threads. I don't remember why, but the work 

> stalled. I remember there was some kind of error that I was not able 

> to debug due to distributed nature of the process. I needed to write 

> more tests for that, but it is either I run out of time, multithreaded 

> tests were complicated or I just had an annual Windows filesystem 

> crash.

>

> Now I am thinking about replacing concept of input/output streams with 

> the concept of channel, which is a queue of strings with limited size. 

> The queue can be rotated to become unlimited. Can be synchronous to 

> wait (block) until messages are read if the queue is full. This may 

> help to clear confusion around all that input output streams handling, 

> pipes and descriptor inheritances.

>

> --

> anatoly t.

> _______________________________________________

> Scons-dev mailing list

> Scons-dev at scons.org

> http://two.pairlist.net/mailman/listinfo/scons-dev

>

>

>

>

>

> --

> Regards,

> Eugene Leskinen

>

>

> _______________________________________________

> Scons-dev mailing list

> Scons-dev at scons.org

> http://two.pairlist.net/mailman/listinfo/scons-dev

>




--
anatoly t.
_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Part Standard Streams design.docx
Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document
Size: 59888 bytes
Desc: Part Standard Streams design.docx
Url : <http://two.pairlist.net/pipermail/scons-dev/attachments/20140224/b260c9f3/attachment-0001.bin>