hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14907) Memory leak in FileSystem cache
Date Tue, 26 Sep 2017 16:17:01 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181044#comment-16181044

Daryn Sharp commented on HADOOP-14907:

You have a leak, but the screenshot isn't showing the leak.  The highlighted strings are both
in the same UGI instance.

Screenshot-1 shows a {{HashMap.Node}} from the {{FileSystem.Cache}}.  The {{HashMap.Node.key}}
field above it is a {{FileSystem.Cache.Key}} which references a {{UserGroupInformation}}.
 The {{HashMap.Node.value}} field shown is a {{DistributedFileSystem}} instance, which references
the same {{FileSystem.Cache.Key}}.  You can see from the hashcodes that the ugi  is identical.

The problem is you have ~20k {{Subject}} instances.  Are you repeatedly invoking {{UserGroupInformation.createRemoteUser}}?

> Memory leak in FileSystem cache
> -------------------------------
>                 Key: HADOOP-14907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: hdfs-client
>    Affects Versions: 2.7.4
>            Reporter: cen yuhai
>         Attachments: screenshot-1.png, screenshot-2.png
> There is a memory leak in FileSystem cache. It will take a lot of memory.I think the
root cause is that the equals function in class Key is not right. You can see in the screenshot-1.png,
the same user etl is in different key... And also FileSystem cache should be a LRU cache

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message