hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8046) Allow better control of getContentSummary
Date Fri, 03 Apr 2015 13:47:53 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14394439#comment-14394439
] 

Kihwal Lee commented on HDFS-8046:
----------------------------------

Test failures are not related. No new test case added since it only changes the sleep time
to be configurable. It's hard to add a meaningful test for this.
- TestRetryCacheWithHA - HDFS-7524
- TestFailureToReadEdits#testFailureToReadEditsOnTransitionToActive\[0\] - After the test
case timed out on a new checkpoint, the mini dfs cluster shutdown. But the standby was actually
done with checkpointing and uploading the image at that point. The SBN said "Exception during
image upload..." on shutdown. So it looks like a timing issue in the test.

> Allow better control of getContentSummary
> -----------------------------------------
>
>                 Key: HDFS-8046
>                 URL: https://issues.apache.org/jira/browse/HDFS-8046
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>         Attachments: HDFS-8046.v1.patch
>
>
> On busy clusters, users performing quota checks against a big directory structure can
affect the namenode performance. It has become a lot better after HDFS-4995, but as clusters
get bigger and busier, it is apparent that we need finer grain control to avoid long read
lock causing throughput drop.
> Even with unfair namesystem lock setting, a long read lock (10s of milliseconds) can
starve many readers and especially writers. So the locking duration should be reduced, which
can be done by imposing a lower count-per-iteration limit in the existing implementation.
 But HDFS-4995 came with a fixed amount of sleep between locks. This needs to be made configurable,
so that {{getContentSummary()}} doesn't get exceedingly slow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message