hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arkady Borkovsky <ark...@yahoo-inc.com>
Subject Re: [jira] Commented: (HADOOP-76) Implement speculative re-execution of reduces
Date Tue, 07 Nov 2006 16:40:51 GMT
Relationship with streaming?

when defining the implementation for speculative re-execution, please  
take into account how Streaming is implemented.
Specifically, streaming tasks are supposed run in their own working  
directories and read and write (temporary) (Unix) files there.
There may be some details that are important for speculative  
re-execution implementation.

On Nov 7, 2006, at 7:24 AM, Devaraj Das (JIRA) wrote:

>     [  
> http://issues.apache.org/jira/browse/HADOOP-76? 
> page=comments#action_12447830 ]
>
> Devaraj Das commented on HADOOP-76:
> -----------------------------------
>
> One more comment - need to document somewhere how exactly (what config  
> params) a user of the PhasedFileSystem (a map method) can access the  
> JobId, TaskId and TIPId when he wants to create an instance of  
> PhasedFileSystem. Better yet, add a new constructor in  
> PhasedFileSystem that takes a JobConf object. Inside that constructor,  
> you can get the JobId, TaskId and TIPId values and proceed. The user  
> doesn't have to bother about details in this case.
>
>> Implement speculative re-execution of reduces
>> ---------------------------------------------
>>
>>                 Key: HADOOP-76
>>                 URL: http://issues.apache.org/jira/browse/HADOOP-76
>>             Project: Hadoop
>>          Issue Type: Improvement
>>          Components: mapred
>>    Affects Versions: 0.1.0
>>            Reporter: Doug Cutting
>>         Assigned To: Sanjay Dahiya
>>            Priority: Minor
>>         Attachments: Hadoop-76.patch, Hadoop-76_1.patch,  
>> spec_reducev.patch
>>
>>
>> As a first step, reduce task outputs should go to temporary files  
>> which are renamed when the task completes.
>
> -- 
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the  
> administrators:  
> http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see:  
> http://www.atlassian.com/software/jira
>
>


Mime
View raw message