hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-7217) Better batching of IBRs
Date Wed, 08 Oct 2014 23:30:34 GMT
Kihwal Lee created HDFS-7217:

             Summary: Better batching of IBRs
                 Key: HDFS-7217
                 URL: https://issues.apache.org/jira/browse/HDFS-7217
             Project: Hadoop HDFS
          Issue Type: Improvement
            Reporter: Kihwal Lee

After HDFS-2691 (pipeline recovery in HA), the number of IBR(incremental block report)s have
doubled.  Since processing of IBR requires exclusive FSNamesystem write lock, this can be
a source of significant overhead on clusters with high write load.  

On one of the busy clusters, we have observed 60 to 70 percent of available handlers being
constantly occupied by IBRs. This degrades throughput greatly when compared to 0.23.

This message was sent by Atlassian JIRA

View raw message