Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Fri, 19 Jul 2013 21:44:50 +0000 (UTC)
From: "Kihwal Lee (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12658648.1374184871250.84044.1374270290510@arcas>
In-Reply-To: <JIRA.12658648.1374184871250@arcas>
References: <JIRA.12658648.1374184871250@arcas>
Subject: [jira] [Updated] (HDFS-5010) Reduce the frequency of
 getCurrentUser() calls from namenode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HDFS-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kihwal Lee updated HDFS-5010:
-----------------------------

    Attachment: HDFS-5010.patch
                HDFS-5010.b023.patch

The new set of patches removes the dependency on FSNamesystem from BlockManager.
                
> Reduce the frequency of getCurrentUser() calls from namenode
> ------------------------------------------------------------
>
>                 Key: HDFS-5010
>                 URL: https://issues.apache.org/jira/browse/HDFS-5010
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode, performance
>    Affects Versions: 2.1.0-beta, 0.23.9
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>         Attachments: HDFS-5010.b023.patch, HDFS-5010.b023.patch, HDFS-5010.patch, HDFS-5010.patch, HDFS-5010.patch
>
>
> According to the "worst" jstack of a busy namenode I took, there were 29 BLOCKED handler threads. All of them were blocked on the UGI class lock. Here is the breakdown:
> - 2 ensureInitialized() - from non static synchronized methods. HADOOP-9748 will unblock these.
> - 27 getCurrentUser()
> Among the 27 threads that were blocked at getCurrentUser(),
> - 18 FSPermissionChecker() - from FSNamesystem#getPermissionChecker() in most namenode RPC methods
> - 8 BlockTokenSecretManager#generateToken() - getBlockLocations()
> - 1 NameNodeRpcServer.mkdirs
> I think FSPermissionChecker can be modified to be created with a passed in UGI. FSNamesystem can the one already stored in RPC server by calling getRemoteUser(). This will eliminate a bulk of getCurrentUser() calls from namenode RPC handlers. A similar change can be made to mkdirs. Block token generation is not as straightforward. Even without it we can eliminate majority of the calls.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira