hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5010) Reduce the frequency of getCurrentUser() calls from namenode
Date Fri, 19 Jul 2013 22:24:48 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714163#comment-13714163

Suresh Srinivas commented on HDFS-5010:

+1 for the patch.
> Reduce the frequency of getCurrentUser() calls from namenode
> ------------------------------------------------------------
>                 Key: HDFS-5010
>                 URL: https://issues.apache.org/jira/browse/HDFS-5010
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode, performance
>    Affects Versions: 2.1.0-beta, 0.23.9
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>         Attachments: HDFS-5010.b023.patch, HDFS-5010.b023.patch, HDFS-5010.patch, HDFS-5010.patch,
> According to the "worst" jstack of a busy namenode I took, there were 29 BLOCKED handler
threads. All of them were blocked on the UGI class lock. Here is the breakdown:
> - 2 ensureInitialized() - from non static synchronized methods. HADOOP-9748 will unblock
> - 27 getCurrentUser()
> Among the 27 threads that were blocked at getCurrentUser(),
> - 18 FSPermissionChecker() - from FSNamesystem#getPermissionChecker() in most namenode
RPC methods
> - 8 BlockTokenSecretManager#generateToken() - getBlockLocations()
> - 1 NameNodeRpcServer.mkdirs
> I think FSPermissionChecker can be modified to be created with a passed in UGI. FSNamesystem
can the one already stored in RPC server by calling getRemoteUser(). This will eliminate a
bulk of getCurrentUser() calls from namenode RPC handlers. A similar change can be made to
mkdirs. Block token generation is not as straightforward. Even without it we can eliminate
majority of the calls.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message