hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4751) In 2.7, Labeled queue usage not shown properly in capacity scheduler UI
Date Mon, 28 Mar 2016 22:34:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-4751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215015#comment-15215015

Eric Payne commented on YARN-4751:

Thank you very much, [~leftnoteasy]!
1) The ultimate solution seems to be YARN-3362. Have you evaluated how hard to back port it?
My intent with this JIRA was simply to provide accurate visual feedback on the Capacity Scheduler
Application Queues page for resources used by labeled queues. I will look deeper into what
it would take to backport YARN-3362. My concern is that it would include more risk and features
than we really want to backport into 2.7.
2) If you don't want to backport YARN-3362. IIUC, the computation of total-used-capacity-considers-all-labels
seems wrong:
In your patch it is Σ(queue.label.used_capacity), actually it should be Σ(queue.label.used_resource)
/ S(root.label.total_resource)
I think you are talking about {{getAllUsedCapacityConsideringLabels}}. IIUC, in this patch,
used capacity is calculated correctly by {{CSQueueUtils#updateQueueStatistics}} when it is
called by {{LeafQueue}} to allocate resources. {{updateQueueStatistics}} calls {{AbstractCSQueue#getAllUsedResourcesConsideringLabels}}
to retrieve all resources consumed by the queue, including labeled resources. It then divides
the consumed resources by the queue's limit:
  public static void updateQueueStatistics(
    Resource usedResources = childQueue.getAllUsedResourcesConsideringLabels();
      usedCapacity = 
          Resources.equals(queueLimit, Resources.none()) ? 0 :
          Resources.divide(calculator, clusterResource, 
              usedResources, queueLimit);
Even still, this patch is, perhaps, not ideal. When used capacity, absolute used capacity,
and available resources are set, these metrics are associated with {{NO_LABEL}}. So, although
the metrics reflect all used resources for a queue (even at parent levels), they are not set
per label but just gathered in with {{NO_LABEL}}. In order to associate the resources with
a label, I think the interface to {{CSQueueUtils#updateQueueStatistics}} may need to change.

Let me get a more precise view of what it would take to backport YARN-3362.
Thanks again.

> In 2.7, Labeled queue usage not shown properly in capacity scheduler UI
> -----------------------------------------------------------------------
>                 Key: YARN-4751
>                 URL: https://issues.apache.org/jira/browse/YARN-4751
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler, yarn
>    Affects Versions: 2.7.3
>            Reporter: Eric Payne
>            Assignee: Eric Payne
>         Attachments: 2.7 CS UI No BarGraph.jpg, YARH-4752-branch-2.7.001.patch, YARH-4752-branch-2.7.002.patch
> In 2.6 and 2.7, the capacity scheduler UI does not have the queue graphs separated by
partition. When applications are running on a labeled queue, no color is shown in the bar
graph, and several of the "Used" metrics are zero.

This message was sent by Atlassian JIRA

View raw message