hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-3840) Support pluggable speculative execution
Date Sun, 27 Jul 2008 01:42:31 GMT
Support pluggable speculative execution

                 Key: HADOOP-3840
                 URL: https://issues.apache.org/jira/browse/HADOOP-3840
             Project: Hadoop Core
          Issue Type: Improvement
          Components: mapred
            Reporter: Matei Zaharia
            Priority: Minor

HADOOP-3412 introduced an way to plug in a job scheduler for MapReduce. However, the job schedulers
all use JobInProgress.obtainNewMapTask or obtainNewReduceTask to select tasks to run from
each job, which uses a threshold-based speculative execution algorithm that has several shortcomings
(see JIRAs about the scheduler not speculating tasks that freeze after having 80% progress
for example). As a first step towards supporting better speculative execution policies while
not breaking backwards compatibility, it makes sense to make the speculative execution policy
pluggable. Luckily this is easy - we just need an interface around obtainNewMapTask and obtainNewReduceTask.
This JIRA suggests adding a TaskSelector abstract class which, given a TaskTracker and a JobInProgress,
chooses a task to run on the tracker. A default implementation that uses the current methods
in JobInProgress is provided. Both TaskSchedulers in trunk are changed to use TaskSelector.

In addition, there are methods to count how many speculative tasks a job needs, since TaskInProgress.hasSpeculative()
may not work if we change the algorithm for selecting speculative tasks. This count is needed
for some schedulers, such as a fair scheduler.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message