hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5010) Reduce the frequency of getCurrentUser() calls from namenode
Date Fri, 19 Jul 2013 16:58:53 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13713840#comment-13713840

Daryn Sharp commented on HDFS-5010:

Need to look at it more, but feels backwards for the {{NameNodeRpcServer}} to ask {{FSNamesystem}}
for the user which in turn asks the RPC server.  Although I guess webhdfs user lookup complicates
> Reduce the frequency of getCurrentUser() calls from namenode
> ------------------------------------------------------------
>                 Key: HDFS-5010
>                 URL: https://issues.apache.org/jira/browse/HDFS-5010
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode, performance
>    Affects Versions: 2.1.0-beta, 0.23.9
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>         Attachments: HDFS-5010.b023.patch, HDFS-5010.patch, HDFS-5010.patch
> According to the "worst" jstack of a busy namenode I took, there were 29 BLOCKED handler
threads. All of them were blocked on the UGI class lock. Here is the breakdown:
> - 2 ensureInitialized() - from non static synchronized methods. HADOOP-9748 will unblock
> - 27 getCurrentUser()
> Among the 27 threads that were blocked at getCurrentUser(),
> - 18 FSPermissionChecker() - from FSNamesystem#getPermissionChecker() in most namenode
RPC methods
> - 8 BlockTokenSecretManager#generateToken() - getBlockLocations()
> - 1 NameNodeRpcServer.mkdirs
> I think FSPermissionChecker can be modified to be created with a passed in UGI. FSNamesystem
can the one already stored in RPC server by calling getRemoteUser(). This will eliminate a
bulk of getCurrentUser() calls from namenode RPC handlers. A similar change can be made to
mkdirs. Block token generation is not as straightforward. Even without it we can eliminate
majority of the calls.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message