hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-5010) Reduce the frequency of getCurrentUser() calls from namenode
Date Thu, 18 Jul 2013 22:02:48 GMT
Kihwal Lee created HDFS-5010:

             Summary: Reduce the frequency of getCurrentUser() calls from namenode
                 Key: HDFS-5010
                 URL: https://issues.apache.org/jira/browse/HDFS-5010
             Project: Hadoop HDFS
          Issue Type: Improvement
            Reporter: Kihwal Lee

According to the "worst" jstack of a busy namenode I took, there were 29 BLOCKED handler threads.
All of them were blocked on the UGI class lock. Here is the breakdown:
-2 ensureInitialized() - from non static synchronized methods. This Jira will unblock these.
-27 getCurrentUser()

Among the 27 threads that were blocked at getCurrentUser(),
- 18 FSPermissionChecker() - from FSNamesystem#getPermissionChecker() in most namenode RPC
- 8 BlockTokenSecretManager#generateToken() - getBlockLocations()
- 1 NameNodeRpcServer.mkdirs

I think FSPermissionChecker can be modified to be created with a passed in UGI. FSNamesystem
can the one already stored in RPC server by calling getRemoteUser(). This will eliminate a
bulk of getCurrentUser() calls from namenode RPC handlers. A similar change can be made to
mkdirs. Block token generation is not as straightforward. Even without it we can eliminate
majority of the calls.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message