hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3336) FileSystem memory leak in DelegationTokenRenewer
Date Thu, 12 Mar 2015 06:14:38 GMT

    [ https://issues.apache.org/jira/browse/YARN-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358146#comment-14358146
] 

Chris Nauroth commented on YARN-3336:
-------------------------------------

Hi [~zxu].  Nice catch.

I think the current version of the patch would change the owner of all obtained delegation
tokens from the application submitter to the user running the ResourceManager daemon (i.e.
the "yarn" user).  Instead, can we simply call {{close}} on the {{FileSystem}} after {{addDelegationTokens}}?
 Closing a {{FileSystem}} also has the effect of removing it from the cache.  Since we already
know that a new instance is getting created every time through this code path, I don't think
closing the instance can impact any other threads.

> FileSystem memory leak in DelegationTokenRenewer
> ------------------------------------------------
>
>                 Key: YARN-3336
>                 URL: https://issues.apache.org/jira/browse/YARN-3336
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>            Reporter: zhihai xu
>            Assignee: zhihai xu
>            Priority: Critical
>         Attachments: YARN-3336.000.patch
>
>
> FileSystem memory leak in DelegationTokenRenewer.
> Every time DelegationTokenRenewer#obtainSystemTokensForUser is called, a new FileSystem
entry will be added to  FileSystem#CACHE which will never be garbage collected.
> This is the implementation of obtainSystemTokensForUser:
> {code}
>   protected Token<?>[] obtainSystemTokensForUser(String user,
>       final Credentials credentials) throws IOException, InterruptedException {
>     // Get new hdfs tokens on behalf of this user
>     UserGroupInformation proxyUser =
>         UserGroupInformation.createProxyUser(user,
>           UserGroupInformation.getLoginUser());
>     Token<?>[] newTokens =
>         proxyUser.doAs(new PrivilegedExceptionAction<Token<?>[]>() {
>           @Override
>           public Token<?>[] run() throws Exception {
>             return FileSystem.get(getConfig()).addDelegationTokens(
>               UserGroupInformation.getLoginUser().getUserName(), credentials);
>           }
>         });
>     return newTokens;
>   }
> {code}
> The memory leak happened when FileSystem.get(getConfig()) is called with a new proxy
user.
> Because createProxyUser will always create a new Subject.
> The calling sequence is 
> FileSystem.get(getConfig())=>FileSystem.get(getDefaultUri(conf), conf)=>FileSystem.CACHE.get(uri,
conf)=>FileSystem.CACHE.getInternal(uri, conf, key)=>FileSystem.CACHE.map.get(key)=>createFileSystem(uri,
conf)
> {code}
> public static UserGroupInformation createProxyUser(String user,
>       UserGroupInformation realUser) {
>     if (user == null || user.isEmpty()) {
>       throw new IllegalArgumentException("Null user");
>     }
>     if (realUser == null) {
>       throw new IllegalArgumentException("Null real user");
>     }
>     Subject subject = new Subject();
>     Set<Principal> principals = subject.getPrincipals();
>     principals.add(new User(user));
>     principals.add(new RealUser(realUser));
>     UserGroupInformation result =new UserGroupInformation(subject);
>     result.setAuthenticationMethod(AuthenticationMethod.PROXY);
>     return result;
>   }
> {code}
> FileSystem#Cache#Key.equals will compare the ugi
> {code}
>       Key(URI uri, Configuration conf, long unique) throws IOException {
>         scheme = uri.getScheme()==null?"":uri.getScheme().toLowerCase();
>         authority = uri.getAuthority()==null?"":uri.getAuthority().toLowerCase();
>         this.unique = unique;
>         this.ugi = UserGroupInformation.getCurrentUser();
>       }
>       public boolean equals(Object obj) {
>         if (obj == this) {
>           return true;
>         }
>         if (obj != null && obj instanceof Key) {
>           Key that = (Key)obj;
>           return isEqual(this.scheme, that.scheme)
>                  && isEqual(this.authority, that.authority)
>                  && isEqual(this.ugi, that.ugi)
>                  && (this.unique == that.unique);
>         }
>         return false;        
>       }
> {code}
> UserGroupInformation.equals will compare subject by reference.
> {code}
>   public boolean equals(Object o) {
>     if (o == this) {
>       return true;
>     } else if (o == null || getClass() != o.getClass()) {
>       return false;
>     } else {
>       return subject == ((UserGroupInformation) o).subject;
>     }
>   }
> {code}
> So in this case, every time createProxyUser and FileSystem.get(getConfig()) are called,
a new FileSystem will be created and a new entry will be added to FileSystem.CACHE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message