hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Owen O'Malley <o...@yahoo-inc.com>
Subject Re: [jira] Commented: (HADOOP-76) Implement speculative re-execution of reduces
Date Tue, 07 Nov 2006 21:39:46 GMT

On Nov 7, 2006, at 8:40 AM, Arkady Borkovsky wrote:

> Relationship with streaming?
>
> when defining the implementation for speculative re-execution,  
> please take into account how Streaming is implemented.
> Specifically, streaming tasks are supposed run in their own working  
> directories and read and write (temporary) (Unix) files there.
> There may be some details that are important for speculative re- 
> execution implementation.

For the immediate term, Streaming probably won't support speculative  
execution. I suspect there are some unexplored corner cases that will  
make it hard. The "normal" Streaming data path via stdout should work  
since it will use the framework's normal collector from the reduce.

-- Owen

Mime
View raw message