hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9748) Reduce blocking on UGI.ensureInitialized
Date Thu, 18 Jul 2013 22:00:50 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13712934#comment-13712934
] 

Kihwal Lee commented on HADOOP-9748:
------------------------------------

ensureInitialized() forced many frequently called methods to unconditionally acquire the class
lock. This patch certainly reduces the lock contention, but the highly contended getCurrentUser()
can still block many threads. I understand this is being addressed in HADOOP-9749.

There may be something else we can do to improve this for namenode.

According to the "worst" jstack of a busy namenode I took, there were 29 BLOCKED handler threads.
All of them were blocked on the UGI class lock. Here is the breakdown:
-  2  ensureInitialized() - from non static synchronized methods. This Jira will unblock these.
-  27 getCurrentUser() 

Among the 27 threads that were blocked at getCurrentUser(),
-  18 FSPermissionChecker() - from FSNamesystem#getPermissionChecker() in most namenode RPC
methods
-   8 BlockTokenSecretManager#generateToken() - getBlockLocations() 
-   1 NameNodeRpcServer.mkdirs

I think FSPermissionChecker can be modified to be created with a passed in UGI. FSNamesystem
can the one already stored in RPC server by calling getRemoteUser(). This will eliminate a
bulk of getCurrentUser() calls from namenode RPC handlers. A similar change can be made to
mkdirs. Block token generation is not as straightforward. Even without it we can eliminate
majority of the calls.  We could potentially do the same for other RPC servers.

I will file a HDFS jira for this. HADOOP-9749 is still needed since getCurrentUser() is used
everywhere beyond namenode RPC server.

+1 for this patch.
                
> Reduce blocking on UGI.ensureInitialized
> ----------------------------------------
>
>                 Key: HADOOP-9748
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9748
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>    Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Critical
>         Attachments: HADOOP-9748.patch, HADOOP-9748.patch
>
>
> EnsureInitialized is always sync'ed on the class, when it should only sync if it actually
has to initialize.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message