hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Dahiya (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-76) Implement speculative re-execution of reduces
Date Thu, 02 Nov 2006 11:47:21 GMT
     [ http://issues.apache.org/jira/browse/HADOOP-76?page=all ]

Sanjay Dahiya updated HADOOP-76:

    Attachment: Hadoop-76_1.patch

Here is update patch for review

changes in this patch 
 - Moved temp files to (mapred.system.dir)/<jobid> 
-  2 new variables in TIP - 
        -    runningTasks ( to track currently running instances of attempts for same task,
earlier this was done using a boolean) 
        -    hasCompletedTask ( for reduces if a attempts succeeds then all subsequent failures
for same TIP should be ignored, existing setup tries to schedule another task for that.) 
- Added a phaseFileSystem, which takes a jobid, tipid, taskid and creates all files in (mapred.system.dir)/<jobid>/<tipid>/<taskid>,
these are moved to their final location on commit or deleted. 
- Task Constructor needs tip id now, its passed in RPC as well. 
- Task.localizeConfiguration adds tip id to conf. 
- Minor change in FSDirectory to add exception message in log if rename fails. 

+Planning to add an example for using PhasedFileSystem - changing RandomWriter to use PhasedFileSystem,
as Devaraj suggested in a separate patch on the same issue. 

thanks owen for review. 

> Implement speculative re-execution of reduces
> ---------------------------------------------
>                 Key: HADOOP-76
>                 URL: http://issues.apache.org/jira/browse/HADOOP-76
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.1.0
>            Reporter: Doug Cutting
>         Assigned To: Sanjay Dahiya
>            Priority: Minor
>         Attachments: Hadoop-76.patch, Hadoop-76_1.patch, spec_reducev.patch
> As a first step, reduce task outputs should go to temporary files which are renamed when
the task completes.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message