hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4751) In 2.7, Labeled queue usage not shown properly in capacity scheduler UI
Date Mon, 28 Mar 2016 22:34:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-4751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215015#comment-15215015
] 

Eric Payne commented on YARN-4751:
----------------------------------

Thank you very much, [~leftnoteasy]!
{quote}
1) The ultimate solution seems to be YARN-3362. Have you evaluated how hard to back port it?
{quote}
My intent with this JIRA was simply to provide accurate visual feedback on the Capacity Scheduler
Application Queues page for resources used by labeled queues. I will look deeper into what
it would take to backport YARN-3362. My concern is that it would include more risk and features
than we really want to backport into 2.7.
{quote}
2) If you don't want to backport YARN-3362. IIUC, the computation of total-used-capacity-considers-all-labels
seems wrong:
In your patch it is Σ(queue.label.used_capacity), actually it should be Σ(queue.label.used_resource)
/ S(root.label.total_resource)
{quote}
I think you are talking about {{getAllUsedCapacityConsideringLabels}}. IIUC, in this patch,
used capacity is calculated correctly by {{CSQueueUtils#updateQueueStatistics}} when it is
called by {{LeafQueue}} to allocate resources. {{updateQueueStatistics}} calls {{AbstractCSQueue#getAllUsedResourcesConsideringLabels}}
to retrieve all resources consumed by the queue, including labeled resources. It then divides
the consumed resources by the queue's limit:
{code}
  public static void updateQueueStatistics(
...
    Resource usedResources = childQueue.getAllUsedResourcesConsideringLabels();
...
      usedCapacity = 
          Resources.equals(queueLimit, Resources.none()) ? 0 :
          Resources.divide(calculator, clusterResource, 
              usedResources, queueLimit);
...
    childQueue.setUsedCapacity(usedCapacity);
{code}
Even still, this patch is, perhaps, not ideal. When used capacity, absolute used capacity,
and available resources are set, these metrics are associated with {{NO_LABEL}}. So, although
the metrics reflect all used resources for a queue (even at parent levels), they are not set
per label but just gathered in with {{NO_LABEL}}. In order to associate the resources with
a label, I think the interface to {{CSQueueUtils#updateQueueStatistics}} may need to change.

Let me get a more precise view of what it would take to backport YARN-3362.
Thanks again.


> In 2.7, Labeled queue usage not shown properly in capacity scheduler UI
> -----------------------------------------------------------------------
>
>                 Key: YARN-4751
>                 URL: https://issues.apache.org/jira/browse/YARN-4751
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler, yarn
>    Affects Versions: 2.7.3
>            Reporter: Eric Payne
>            Assignee: Eric Payne
>         Attachments: 2.7 CS UI No BarGraph.jpg, YARH-4752-branch-2.7.001.patch, YARH-4752-branch-2.7.002.patch
>
>
> In 2.6 and 2.7, the capacity scheduler UI does not have the queue graphs separated by
partition. When applications are running on a labeled queue, no color is shown in the bar
graph, and several of the "Used" metrics are zero.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message