Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Wed, 15 Oct 2014 00:21:33 +0000 (UTC)
From: "Wangda Tan (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12738747.1409734526000.267360.1413332493922@Atlassian.JIRA>
In-Reply-To: <JIRA.12738747.1409734526000@Atlassian.JIRA>
References: <JIRA.12738747.1409734526000@Atlassian.JIRA>
 <JIRA.12738747.1409734526952@arcas>
Subject: [jira] [Updated] (YARN-2496) Changes for capacity scheduler to
 support allocate resource respect labels
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/YARN-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wangda Tan updated YARN-2496:
-----------------------------
    Attachment: YARN-2496-20141014-1.patch

[~vinodkv], thanks for your comments. They are all make sense to me and addressed in the new patch, except a few,

bq. SchedulerUtils.checkAndThrowIfLabelNotIncluded -> checkIfLabelInTheCollection()
Changed to checkIfLabelInClusterNodeLabels

bq. Seems like removing accessible labels from a queue doesn't work given the changes in AbstractCSQueue.
Removing accessible labels from a queue should be done by first modifying capacity scheduler configuration, then call refreshQueue by RMAdminCLI

bq. I think reservations are still not handled per accessible node-labels in the patch. We can fix it separately though.
Reservation is partially works. Resource will be reserved only when the node satisfies queue accessible-label and the node satisfies node-label-expression of the resource request. 
The reservation will not work with the changes of YARN-1769 (continuous reservation looking). Which is required to count reserved resource by node label, we haven't counted reserved resource by node label.

> Changes for capacity scheduler to support allocate resource respect labels
> --------------------------------------------------------------------------
>
>                 Key: YARN-2496
>                 URL: https://issues.apache.org/jira/browse/YARN-2496
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-2496-20141009-1.patch, YARN-2496-20141014-1.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch
>
>
> This JIRA Includes:
> - Add/parse labels option to {{capacity-scheduler.xml}} similar to other options of queue like capacity/maximum-capacity, etc.
> - Include a "default-label-expression" option in queue config, if an app doesn't specify label-expression, "default-label-expression" of queue will be used.
> - Check if labels can be accessed by the queue when submit an app with labels-expression to queue or update ResourceRequest with label-expression
> - Check labels on NM when trying to allocate ResourceRequest on the NM with label-expression
> - Respect  labels when calculate headroom/user-limit


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)