hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-5353) ResourceManager can leak delegation tokens when they are shared across apps
Date Mon, 11 Jul 2016 20:27:11 GMT

     [ https://issues.apache.org/jira/browse/YARN-5353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jason Lowe updated YARN-5353:
-----------------------------
    Attachment: YARN-5353.001.patch

Seems to me that we need to make sure that the appTokens map always has the application removed
when the application is marked as finished.  It's our one chance to clean up the app entry,
and currently the code can conditionally decide to leave the app's entry in the map.

Attaching a patch that always removes the appTokens entry corresponding to an app when the
app finished event is received.  Any tokens that are shared with other apps will continue
to exist in the allTokens map, so I think we'll still be good as far as token-sharing goes.

> ResourceManager can leak delegation tokens when they are shared across apps
> ---------------------------------------------------------------------------
>
>                 Key: YARN-5353
>                 URL: https://issues.apache.org/jira/browse/YARN-5353
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.7.0, 2.6.1
>            Reporter: Jason Lowe
>            Priority: Critical
>         Attachments: YARN-5353.001.patch
>
>
> Recently saw a ResourceManager go into heavy GC.  Heap dump showed that there were millions
of delegation tokens on the heap.  It looks like most of them belonged to the appTokens map
in DelegationTokenRenewer.  When an app completes and tokens are removed for it, I noticed
that the appTokens entry for the app is not cleaned up if tokens were shared with other active
apps.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message