hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6425) Large postponedMisreplicatedBlocks has impact on blockReport latency
Date Wed, 17 Dec 2014 12:24:32 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249737#comment-14249737
] 

Hudson commented on HDFS-6425:
------------------------------

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #44 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/44/])
HDFS-6425. Large postponedMisreplicatedBlocks has impact on blockReport latency. Contributed
by Ming Ma. (kihwal: rev b7923a356e9f111619375b94d12749d634069347)
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
* hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
* hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencingWithReplication.java


> Large postponedMisreplicatedBlocks has impact on blockReport latency
> --------------------------------------------------------------------
>
>                 Key: HDFS-6425
>                 URL: https://issues.apache.org/jira/browse/HDFS-6425
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Ming Ma
>            Assignee: Ming Ma
>             Fix For: 2.7.0
>
>         Attachments: HDFS-6425-2.patch, HDFS-6425-3.patch, HDFS-6425-Test-Case.pdf, HDFS-6425.patch
>
>
> Sometimes we have large number of over replicates when NN fails over. When the new active
NN took over, over replicated blocks will be put to postponedMisreplicatedBlocks until all
DNs for that block aren't stale anymore.
> We have a case where NNs flip flop. Before postponedMisreplicatedBlocks became empty,
NN fail over again and again. So postponedMisreplicatedBlocks just kept increasing until the
cluster is stable. 
> In addition, large postponedMisreplicatedBlocks could make rescanPostponedMisreplicatedBlocks
slow. rescanPostponedMisreplicatedBlocks takes write lock. So it could slow down the block
report processing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message