hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4634) Scheduler UI/Metrics need to consider cases like non-queue label mappings
Date Thu, 18 Feb 2016 18:36:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15152800#comment-15152800
] 

Sunil G commented on YARN-4634:
-------------------------------

Hi [~leftnoteasy]
Yes, I understood the scenario which you have mentioned. And this was one of my doubt while
I was fixing this pblm.

We can have some queues as orphan ones (queues that are not part of any labels). If we assume
these queues can be categorized under DEFULT_PARTITION, in  a way its fine i think. Logically
its somewhat fine if we show capacity as 0 under other labels.

bq.To me, we can ignore label only when it is a completely "orphan" label, which is not assigned
to any node and queue.
I agree to this. Due to this reason, I have placed {{this.setLabelToQueueMappingAvailable(true);}}
in {{reinitializeQueueLabels}}.

{code}
      for (Entry<String, Set<String>> entry : queueToLabels.entrySet()) {
        String queue = entry.getKey();
        Queue q = new Queue();
        this.queueCollections.put(queue, q);

        Set<String> labels = entry.getValue();
        if (labels.contains(ANY)) {
          continue;
        }
        this.setLabelToQueueMappingAvailable(true);

        q.accessibleNodeLabels.addAll(labels);
{code}

For any queue, if labels are not configured, then {{accessableLabels}} is "*" or "ANY". Hence
if there is one queue which has some real labels, then I set this variable to true and think
that there is a queue which is under a label. Now eventhough other queues are not with any
labels, still i can consider them as under "DEFAULT_PARTITION"
As  you mentioned, if none of the queues has any label, then this code wont hit. 

So i have now 2 informations
1. isLabelToQueueMappingAvailable() ==> Gives information whether there is any label-queue
is available.
2. ( nodeLabelManager.pullRMNodeLabelsInfo() == null)  or only ANY is available

Second one give information that whether any labels are in cluster. So with these 2 entities,
we can decide whether all queues are orphan OR no labels are in cluster.

Is this ok?

> Scheduler UI/Metrics need to consider cases like non-queue label mappings
> -------------------------------------------------------------------------
>
>                 Key: YARN-4634
>                 URL: https://issues.apache.org/jira/browse/YARN-4634
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>    Affects Versions: 2.7.1
>            Reporter: Sunil G
>            Assignee: Sunil G
>         Attachments: 0001-YARN-4634.patch
>
>
> Currently when label-queue mappings are not available, there are few assumptions taken
in UI and in metrics.
> In above case where labels are enabled and available in cluster but without any queue
mappings, UI displays queues under labels. This is not correct.
> Currently  labels enabled check and availability of labels are considered to render scheduler
UI. Henceforth we also need to check whether 
> - queue-mappings are available
> - nodes are mapped with labels with proper exclusivity flags on
> This ticket also will try to see the default configurations in queue when labels are
not mapped. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message