hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6285) Add option to set max limit on ResourceManager for ApplicationClientProtocol.getApplications
Date Mon, 06 Mar 2017 23:02:33 GMT

    [ https://issues.apache.org/jira/browse/YARN-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898331#comment-15898331

Wangda Tan commented on YARN-6285:

[~zhaoyunjiong], from your previous comment: 

bq. 1. Slowness in getApplications, below stack trace files shows it spend at least 2.25 seconds
in getApplications.

Does it include ResourceRequest? From our experiences, the there're lots of information in
ResourceRequest/getLogAggregationReportsForApp and it should contribute to most of the data/size
of get_applications responses. If it is possible, could you do a test to see, how much time
will be spent if we don't include ResourceRequest/getLogAggregationReportsForApp in the get_applications
response? This will be very helpful for us to make decisions. 

To solve the problem, instead of adding the limit of apps in the server side, I would prefer
to optimize following items:
1) Add parameter to indicate if we should include ResourceRequest/getLogAggregationReportsForApp
in the response, default is true to make it compatible. (Can be done if above experimental
shows it really helps).
2) If required, use cache to store finished apps since their report will not change. (Can
be done separately)

In addition, following items can be optimized in client side: 
3) Properly use filter to include only app with expected states (RUNNING/ACCEPT) to avoid
including all apps.
4) Set limit when requesting app reports from client.


> Add option to set max limit on ResourceManager for ApplicationClientProtocol.getApplications
> --------------------------------------------------------------------------------------------
>                 Key: YARN-6285
>                 URL: https://issues.apache.org/jira/browse/YARN-6285
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: yunjiong zhao
>            Assignee: yunjiong zhao
>         Attachments: YARN-6285.001.patch, YARN-6285.002.patch, YARN-6285.003.patch
> When users called ApplicationClientProtocol.getApplications, it will return lots of data,
and generate lots of garbage on ResourceManager which caused long time GC.
> For example, on one of our RM, when called rest API " http://<rm http address:port>/ws/v1/cluster/apps"
it can return 150MB data which have 944 applications.
> getApplications have limit parameter, but some user might not set it, and then the limit
will be Long.MAX_VALUE.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message