hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei-Chiu Chuang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-13971) Fix memory leak in FileSystem.Cache.Key class
Date Tue, 10 Jan 2017 22:36:58 GMT
Wei-Chiu Chuang created HADOOP-13971:

             Summary: Fix memory leak in FileSystem.Cache.Key class
                 Key: HADOOP-13971
                 URL: https://issues.apache.org/jira/browse/HADOOP-13971
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs, security
            Reporter: Wei-Chiu Chuang

Calling {{FileSystem#get(final URI uri, final Configuration conf, final String user)}} multiple
times can result in memory leak because of the hash method implementation of UserGroupInformation.
FileSystem always instantiates a new FileSystem object despite using the same user name/same

In the past, other downstream projects work around this bug by either disabling cache (set
{{fs.%s.impl.disable.cache}} to {{true}}) or call {{FileSystem.closeAllForUGI()}} to release
resource on demand. (See for instance HIVE-3098, YARN-58, TEZ-1585)

However, neither approach is desirable. The first workaround loses performance because it
disables cache. This bug was discussed extensively in  HADOOP-12707, but the proposed workaround
{{FileSystem.closeAllForUGI()}} is insufficient, because it won't purge the objects from cache
due to the same hash method implementation bug.

I would like to file a new jira, knowing that current workarounds do not work, and invite
more discussion. An ideal approach is to change UGI hash method, but it may break many downstream
applications, so setting target version as 3.0.0-beta

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message