hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6285) Add option to set max limit on ResourceManager for ApplicationClientProtocol.getApplications
Date Sat, 04 Mar 2017 13:55:45 GMT

    [ https://issues.apache.org/jira/browse/YARN-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895687#comment-15895687
] 

Rohith Sharma K S commented on YARN-6285:
-----------------------------------------

Sorry for pitching late, I have one doubt/concerns on the issue raised.
bq. on one of our RM, when called rest API " http://<rm http address:port>/ws/v1/cluster/apps"
it can return 150MB data which have 944 applications.
I am surprised that retrieval of 1K applications downloading 150MB data. I remember that retrieval
of 100K applications were taken hardly less than 50MB. There is something wrong in retrieval
of applications. Would you tell which Hadoop version? Are these applications are RUNNING or
FINISHED? If RUNNIG, how many applications are there in cluster?
# If there is any issue with creating a ApplicationReport, then reducing the limit do not
solve size of the downloading data. 
# Note that getApplications API do not return the applications in any order. It is basically
a random random list of applications based on hash code. See YARN-4988. 
# Currently, getApplications return ALL *submitted*/*running*/*finished* applications in randomized
order i.e based on the hash code. But after the patch, if admin configures *max-limit-get-applications*
then user may not get RUNNING applications in default API. This will effect for *ALL* the
user.



> Add option to set max limit on ResourceManager for ApplicationClientProtocol.getApplications
> --------------------------------------------------------------------------------------------
>
>                 Key: YARN-6285
>                 URL: https://issues.apache.org/jira/browse/YARN-6285
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: yunjiong zhao
>            Assignee: yunjiong zhao
>         Attachments: YARN-6285.001.patch, YARN-6285.002.patch, YARN-6285.003.patch
>
>
> When users called ApplicationClientProtocol.getApplications, it will return lots of data,
and generate lots of garbage on ResourceManager which caused long time GC.
> For example, on one of our RM, when called rest API " http://<rm http address:port>/ws/v1/cluster/apps"
it can return 150MB data which have 944 applications.
> getApplications have limit parameter, but some user might not set it, and then the limit
will be Long.MAX_VALUE.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message