hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-2874) Optimize finding of speculative tasks
Date Fri, 22 Feb 2008 11:18:21 GMT
Optimize finding of speculative tasks

                 Key: HADOOP-2874
                 URL: https://issues.apache.org/jira/browse/HADOOP-2874
             Project: Hadoop Core
          Issue Type: Improvement
          Components: mapred
            Reporter: Amar Kamat

Assuming HADOOP-2119 provides better data structures for handling running TIPs, finding new
speculative tasks can be further optimized. Two of which could be 
1) {{conf.getMapSpeculativeExecution()}} and {{conf.getReduceSpeculativeExecution()}} should
be moved to {{JobInProgress}}. A simple check for this boolean can prove useful before checking
for speculative tasks. This will be useful for jobs with large maps and reducers where scanning
all the TIPs  can be costly. 
2) Since the progress of a TIP changes only when {{TaskInProgress.recomputeProgress()}} is
invoked, it makes more sense to check for speculation in {{JobInProgress.updateTaskStatus()}}
and move the TIPs that can be speculated to the front of the running queue.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message