Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <253885232.1226580104434.JavaMail.jira@brutus>
Date: Thu, 13 Nov 2008 04:41:44 -0800 (PST)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4558) Scheduler fails to reclaim capacity
 if Jobs are submitted to queue one after the other
In-Reply-To: <499010363.1225464224236.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-4558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647273#action_12647273 ] 

Amar Kamat commented on HADOOP-4558:
------------------------------------

Note that once a task is assigned, capacity scheduler is not updated to reflect the change. It waits for a heartbeat to update itself via {{updateQSIObjects()}}. I feel its better we update the scheduler after the tasks are assigned so that the scheduler is up to date. This needs benchmarking and discussion. Following is the use case
- job1 is added
- job1 takes up one slot more than guaranteed 
- job2 is added
- ideally the reclaim thread should detect the capacity violation and kill that one extra task but will not do as the count is stale and one less.
- upon next heartbeat the scheduler will detect that job1 has violated and hence will start the reclaim process. 

> Scheduler fails to reclaim capacity if Jobs are submitted to queue one after the other
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4558
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4558
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/capacity-sched
>    Affects Versions: 0.19.0
>         Environment: Cluster Capacity Maps=Reduces =210 each
> Two Queues: 
> Q1:  default, GC (%) =40, GC=84 (Maps and Reduces each). Reclaim time = 3 mins.
> Q2: test_q1, GC (%) =60, GC=126 (Maps and Reduces each) Reclaim time = 2 mins
>            Reporter: Karam Singh
>            Assignee: Amar Kamat
>         Attachments: 4558.1.patch, HADOOP-4558-v1.4.patch
>
>
> Scheduler fails to reclaim capacity if Jobs are submitted to queue one after the other.
> First job submitted with tasks equal to cluster's M/R Capacity
> Second is submitted to different queue when all tasks of First Job are running, scheduler fails to reclaim capacity for second job.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.