hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4995) Make getContentSummary() less expensive
Date Mon, 15 Jul 2013 21:30:49 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13709012#comment-13709012

Kihwal Lee commented on HDFS-4995:

Here is my first cut of idea.

If the result of these commands don't need to be strictly consistent, we can probably avoid
potentially long lock region.
- Lock directory when obtaining a copy of children list (ReadOnlyList).
- Lock namesystem when computing in INodeFile. 
- Handle stale children in the depth first traversing.

Recursion may need to be done at FSDirectory-level. This also makes it easy to ensure the
namespace lock is not acquired while the directory lock is held. If we do this at the finest
granularity, lock contention will increase and at minimum getContentSummary() will take considerably
longer.  We may internally batch (increased lock granularity) to reduce the impact. In this
case, processing of a batch needs to be in the same namespace lock. Locking may look less
tangled if the recursion is in FSNamesystem.
> Make getContentSummary() less expensive
> ---------------------------------------
>                 Key: HDFS-4995
>                 URL: https://issues.apache.org/jira/browse/HDFS-4995
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 0.23.9, 2.3.0
>            Reporter: Kihwal Lee
> When users call du or count DFS command, getContentSummary() method is called against
namenode. If the directory has many directories and files, it could hold the namesystem lock
for a long time. We've seen it taking over 20 seconds. Namenode should not allow regular users
to cause extended locking.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message