hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-6523) RM requires large memory in sending out security tokens as part of Node Heartbeat in large cluster
Date Wed, 26 Apr 2017 14:17:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984853#comment-15984853
] 

Sunil G edited comment on YARN-6523 at 4/26/17 2:16 PM:
--------------------------------------------------------

Yes. Its ideal to keep a copy of apps tokens on each RMNodeImpl itself. I think RMNodeImpl
could pull renewed tokens from {{DelegationTokenRenewer}} at regular intervals if there is
a concern over increased number of events to be fired to RMNodeImpl regarding token renewal.
I still feel an event publishing model will be more concrete.


was (Author: sunilg):
Yes. Its ideal to keep a copy on each RMNodeImpl itself. I think RMNodeImpl could pull for
renewed tokens from {{DelegationTokenRenewer]} at regular intervals if there is a concern
over increased number of events to be fired to RMNodeImpl regarding token renewal. I still
feel an event publishing model will be more concrete.

> RM requires large memory in sending out security tokens as part of Node Heartbeat in
large cluster
> --------------------------------------------------------------------------------------------------
>
>                 Key: YARN-6523
>                 URL: https://issues.apache.org/jira/browse/YARN-6523
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: RM
>    Affects Versions: 2.8.0, 2.7.3
>            Reporter: Naganarasimha G R
>            Assignee: Naganarasimha G R
>            Priority: Critical
>
> Currently as part of heartbeat response RM sets all application's tokens though all applications
might not be active on the node. On top of it NodeHeartbeatResponsePBImpl converts tokens
for each app into SystemCredentialsForAppsProto. Hence for each node and each heartbeat too
many SystemCredentialsForAppsProto objects were getting created.
> We hit a OOM while testing for 2000 concurrent apps on 500 nodes cluster with 8GB RAM
configured for RM



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message