hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-4995) Make getContentSummary() less expensive
Date Thu, 17 Oct 2013 19:52:42 GMT

     [ https://issues.apache.org/jira/browse/HDFS-4995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Kihwal Lee updated HDFS-4995:

    Attachment: HDFS-4995.trunk1.patch

Here is a candidate patch, which adds a runtime context object to keep track of progress at
various inodes. When it goes over the configured limit, locks are relinquished and reacquired
after a delay.  If this happens, all affected inodes should check for safety.  

With this change, the consistency is not guaranteed. But it is no different from what is expected
from "du" command against normal file systems. If a tree is actively being modified, the result
may not be consistent, but still useful. If the tree is not changing, it will give a consistent
result.  Even without this patch, the result of getContentSummary() will get stale if the
tree is constantly being modified. So I think the strict consistency is not required.

By setting the limit to something like 750,000, the max blocking time becomes about 15-25ms
on todays typical Intel servers.  This is set to 0 by default, which makes the getContentSummary()
calls blocking, which is the behavior before this patch.

> Make getContentSummary() less expensive
> ---------------------------------------
>                 Key: HDFS-4995
>                 URL: https://issues.apache.org/jira/browse/HDFS-4995
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 0.23.9, 2.3.0
>            Reporter: Kihwal Lee
>         Attachments: HDFS-4995.trunk1.patch
> When users call du or count DFS command, getContentSummary() method is called against
namenode. If the directory has many directories and files, it could hold the namesystem lock
for a long time. We've seen it taking over 20 seconds. Namenode should not allow regular users
to cause extended locking.

This message was sent by Atlassian JIRA

View raw message