hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei-Chiu Chuang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-13971) Fix memory leak in FileSystem.Cache.Key class
Date Tue, 10 Jan 2017 22:36:58 GMT
Wei-Chiu Chuang created HADOOP-13971:
----------------------------------------

             Summary: Fix memory leak in FileSystem.Cache.Key class
                 Key: HADOOP-13971
                 URL: https://issues.apache.org/jira/browse/HADOOP-13971
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs, security
            Reporter: Wei-Chiu Chuang


Calling {{FileSystem#get(final URI uri, final Configuration conf, final String user)}} multiple
times can result in memory leak because of the hash method implementation of UserGroupInformation.
FileSystem always instantiates a new FileSystem object despite using the same user name/same
URI.

In the past, other downstream projects work around this bug by either disabling cache (set
{{fs.%s.impl.disable.cache}} to {{true}}) or call {{FileSystem.closeAllForUGI()}} to release
resource on demand. (See for instance HIVE-3098, YARN-58, TEZ-1585)

However, neither approach is desirable. The first workaround loses performance because it
disables cache. This bug was discussed extensively in  HADOOP-12707, but the proposed workaround
{{FileSystem.closeAllForUGI()}} is insufficient, because it won't purge the objects from cache
due to the same hash method implementation bug.

I would like to file a new jira, knowing that current workarounds do not work, and invite
more discussion. An ideal approach is to change UGI hash method, but it may break many downstream
applications, so setting target version as 3.0.0-beta



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message