Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <255464340.1217122951693.JavaMail.jira@brutus>
Date: Sat, 26 Jul 2008 18:42:31 -0700 (PDT)
From: "Matei Zaharia (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Created: (HADOOP-3840) Support pluggable speculative
 execution
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Support pluggable speculative execution
---------------------------------------

                 Key: HADOOP-3840
                 URL: https://issues.apache.org/jira/browse/HADOOP-3840
             Project: Hadoop Core
          Issue Type: Improvement
          Components: mapred
            Reporter: Matei Zaharia
            Priority: Minor


HADOOP-3412 introduced an way to plug in a job scheduler for MapReduce. However, the job schedulers all use JobInProgress.obtainNewMapTask or obtainNewReduceTask to select tasks to run from each job, which uses a threshold-based speculative execution algorithm that has several shortcomings (see JIRAs about the scheduler not speculating tasks that freeze after having 80% progress for example). As a first step towards supporting better speculative execution policies while not breaking backwards compatibility, it makes sense to make the speculative execution policy pluggable. Luckily this is easy - we just need an interface around obtainNewMapTask and obtainNewReduceTask. This JIRA suggests adding a TaskSelector abstract class which, given a TaskTracker and a JobInProgress, chooses a task to run on the tracker. A default implementation that uses the current methods in JobInProgress is provided. Both TaskSchedulers in trunk are changed to use TaskSelector.

In addition, there are methods to count how many speculative tasks a job needs, since TaskInProgress.hasSpeculative() may not work if we change the algorithm for selecting speculative tasks. This count is needed for some schedulers, such as a fair scheduler.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.