hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-76) Implement speculative re-execution of reduces
Date Thu, 12 Oct 2006 20:53:46 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-76?page=comments#action_12441868 ] 
            
Owen O'Malley commented on HADOOP-76:
-------------------------------------

I think it is better for now to just use JobConf.setSpeculativeExecution for both maps and
reduces.

machinesWhereFailed is a list of machines where the task has failed. To find where it is currently
running, you need to use recentTasks. Currently, the recentTasks is a set of task ids that
are running. You should probably make it a map from task id to task tracker id. 

Don't block speculative reduces based on other reduces running. That would make a perpetually
busy cluster never run speculative reduces.

*** Please create a library that lets you create files off to the side and when you call commit
moves them into place. Speculative reduces need it, but that functionality is useful other
places, such as side effect-based maps. The class should also have an abort method that cleans
up.

The defaults for the map speculative execution don't look too unreasonable, so just use them
for now.

> Implement speculative re-execution of reduces
> ---------------------------------------------
>
>                 Key: HADOOP-76
>                 URL: http://issues.apache.org/jira/browse/HADOOP-76
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.1.0
>            Reporter: Doug Cutting
>         Assigned To: Sanjay Dahiya
>            Priority: Minor
>         Attachments: spec_reducev.patch
>
>
> As a first step, reduce task outputs should go to temporary files which are renamed when
the task completes.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message