hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4304) AM max resource configuration per partition to be displayed/updated correctly in UI and in various partition related metrics
Date Fri, 18 Dec 2015 22:22:46 GMT

    [ https://issues.apache.org/jira/browse/YARN-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064874#comment-15064874
] 

Wangda Tan commented on YARN-4304:
----------------------------------

Hi [~sunilg],

1) Changes for the max-available-to-a-queue may be separated to a separated patch. The major
concern is 
- performance: For every allocated container, we need to iterate all labels to get a total
resources.
- I think the longer term fix should be, add a by-partition info to queue metrics, including
max/guaranteed/available/used, etc. I can help to review proposal/patches.

2) There're several methods are using:
{code}
  public synchronized Resource getAMResourceLimitPerPartition(
      String nodePartition)
{code} 

I think after we have YARN-4418, we don't need to calculate AMResourceLimitPerPartition everytime.
So I will suggest to split the method to calculate-and-get and read-only methods. How about
call them calculateAndGetAMResourceLimitPerPartition and getAMResourceLimitPerPartition? {{getPendingAppDiagnosticMessage}}/REST-API
will use read-only interface.


Agree? 

3) Could you upload screenshots/REST api responses? 

To your queston:
bq. I am giving AM Resource limit for user AM limit, This is not really correct. But to get
this, we need to have some round about way. I gave this implementation based on below comment.
Is this what you also expect?
Is it possible to use {{org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.CapacitySchedulerLeafQueueInfo#users}}
to get AM Resource Limit instead? Is there any concern of you for this approach? I think using
same queue's limit for queue and user may not be correct to me.

> AM max resource configuration per partition to be displayed/updated correctly in UI and
in various partition related metrics
> ----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-4304
>                 URL: https://issues.apache.org/jira/browse/YARN-4304
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: webapp
>    Affects Versions: 2.7.1
>            Reporter: Sunil G
>            Assignee: Sunil G
>         Attachments: 0001-YARN-4304.patch, 0002-YARN-4304.patch, 0003-YARN-4304.patch,
0004-YARN-4304.patch, 0005-YARN-4304.patch, 0005-YARN-4304.patch, REST_and_UI.zip
>
>
> As we are supporting per-partition level max AM resource percentage configuration, UI
and various metrics also need to display correct configurations related to same. 
> For eg: Current UI still shows am-resource percentage per queue level. This is to be
updated correctly when label config is used.
> - Display max-am-percentage per-partition in Scheduler UI (label also) and in ClusterMetrics
page
> - Update queue/partition related metrics w.r.t per-partition am-resource-percentage



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message