hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "yunjiong zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6285) Add option to set max limit on ResourceManager for ApplicationClientProtocol.getApplications
Date Tue, 14 Mar 2017 01:21:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15923364#comment-15923364

yunjiong zhao commented on YARN-6285:

 convertFromProtoFormat is called once for every app
This is not true.
There are multiple place will hit convertFromProtoFormat. For example:
Inside RMAppImpl.getLogAggregationStatusForAppReport():
      for (Entry<NodeId, LogAggregationReport> report : reports.entrySet()) {
        switch (report.getValue().getLogAggregationStatus()) { // will call convertFromProtoFormat
Inside RMAppImpl.getLogAggregationReportsForApp
for (Entry<NodeId, LogAggregationReport> output : outputs.entrySet()) {
          if (!output.getValue().getLogAggregationStatus()
And our cluster which have more than 3000 nodes and running applications some times more than
500, from above two places getApplications may call convertFromProtoFormat 3,000,000 times.

I'm not saying it will completely solve the problem.
But definitely can approve the situation.

> Add option to set max limit on ResourceManager for ApplicationClientProtocol.getApplications
> --------------------------------------------------------------------------------------------
>                 Key: YARN-6285
>                 URL: https://issues.apache.org/jira/browse/YARN-6285
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: yunjiong zhao
>            Assignee: yunjiong zhao
>         Attachments: YARN-6285.001.patch, YARN-6285.002.patch, YARN-6285.003.patch
> When users called ApplicationClientProtocol.getApplications, it will return lots of data,
and generate lots of garbage on ResourceManager which caused long time GC.
> For example, on one of our RM, when called rest API " http://<rm http address:port>/ws/v1/cluster/apps"
it can return 150MB data which have 944 applications.
> getApplications have limit parameter, but some user might not set it, and then the limit
will be Long.MAX_VALUE.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message