hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7217) Better batching of IBRs
Date Thu, 09 Oct 2014 03:00:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164632#comment-14164632

Kihwal Lee commented on HDFS-7217:

Precommit is not working correctly.
bq. -1 javadoc
Actually there is no new javadoc warnings. Diff is zero.

bq.  -1 core tests. The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:
According to the precommit build log, all test cases in TestHDFSServerPorts actually passed.
 There was no test failure.

> Better batching of IBRs
> -----------------------
>                 Key: HDFS-7217
>                 URL: https://issues.apache.org/jira/browse/HDFS-7217
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Kihwal Lee
>         Attachments: HDFS-7217.patch
> After HDFS-2691 (pipeline recovery in HA), the number of IBR(incremental block report)s
have doubled.  Since processing of IBR requires exclusive FSNamesystem write lock, this can
be a source of significant overhead on clusters with high write load.  
> On one of the busy clusters, we have observed 60 to 70 percent of available handlers
being constantly occupied by IBRs. This degrades throughput greatly when compared to 0.23.

This message was sent by Atlassian JIRA

View raw message