Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <909032955.1245311047731.JavaMail.jira@brutus>
Date: Thu, 18 Jun 2009 00:44:07 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5964) Fix the 'cluster drain' problem in
 the Capacity Scheduler wrt High RAM Jobs
In-Reply-To: <1711507105.1243989610977.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721114#action_12721114 ] 

Arun C Murthy commented on HADOOP-5964:
---------------------------------------

Some notes about this patch:

# I've introduced a new org.apache.hadoop.mapred.server.jobtracker.TaskTracker class to track all information about a given TaskTracker at the JobTracker such as TaskTrackerStatus, reservations for high-ram jobs etc. I chose the new package based on the proposals at HADOOP-398.
# I've changed the TaskScheduler.assignTasks api to use the newly introduced rather than the tepid TaskTrackerStatus. Clearly other schedulers (ala CapacityTaskScheduler) can start to take advantage of this as I've fixed them appropriately in this patch.
# I've had to make some classes public (JobInProgress, TaskTrackerStatus) for org.apache.hadoop.mapred.server.jobtracker.TaskTracker to work with appropriate caveats in the javadocs etc.

> Fix the 'cluster drain' problem in the Capacity Scheduler wrt High RAM Jobs
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-5964
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5964
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/capacity-sched
>    Affects Versions: 0.20.0
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5964_0_20090602.patch, HADOOP-5964_1_20090608.patch, HADOOP-5964_2_20090609.patch, HADOOP-5964_4_20090615.patch, HADOOP-5964_6_20090617.patch, HADOOP-5964_7_20090618.patch
>
>
> When a HighRAMJob turns up at the head of the queue, the current implementation of support for HighRAMJobs in the Capacity Scheduler has problem in that the scheduler stops assigning tasks to all TaskTrackers in the cluster until a HighRAMJob finds a suitable TaskTrackers for all its tasks.
> This causes a severe utilization problem since effectively no new tasks are allowed to run until the HighRAMJob (at the head of the queue) gets slots.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.