Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Message-ID: <840246381.1247244315060.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 09:45:15 -0700 (PDT)
From: "rahul k singh (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Subject: [jira] Commented: (MAPREDUCE-725)
 CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
In-Reply-To: <944828554.1246985654837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729713#action_12729713 ] 

rahul k singh commented on MAPREDUCE-725:
-----------------------------------------

Looked at the JobInProgress's findSpeculativeTask , it doesn't change any state of TaskInProgress. We can refactor this method in order to find out if jip has any speculative tasks.

We are using hasSpeculative to block a slot on the task tracker. As hasSpeculative is very dynamic in nature , there are chances that by the time tasktracker comes back for the task , we dont have any speculativeTasks to assign, this would mean we would waste some heartbeat and also would lead to under utilization. 
This problem is already there with the capacity-scheduler though.

One way of reducing this is to have MIN_SPEC_CAP(in JobInProgress) kind of variable in capacity scheduler.

Any thoughts?

> CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-725
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-725
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/capacity-sched
>            Reporter: Arun C Murthy
>            Priority: Critical
>
> CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask has a duplicate of the old speculation code (pre HADOOP-2141) which needs to be fixed in-order for speculation to work correctly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.