Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Tue, 29 Sep 2015 03:47:04 +0000 (UTC)
From: "Masatake Iwasaki (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12465184.1274509229000.98905.1443498424644@Atlassian.JIRA>
In-Reply-To: <JIRA.12465184.1274509229000@Atlassian.JIRA>
References: <JIRA.12465184.1274509229000@Atlassian.JIRA>
 <JIRA.12465184.1274509229369@arcas>
Subject: [jira] [Updated] (HDFS-1172) Blocks in newly completed files are
 considered under-replicated too quickly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Masatake Iwasaki updated HDFS-1172:
-----------------------------------
    Attachment: HDFS-1172.010.patch

bq. Also if a block's effective replica number (including pending replica number) is >= than its replication factor, the block should not be in neededReplication. 

I rethinked about this and fixed {{checkReplication}} accordingly.

I also fixed to address checkstyle warnings. Warning about file length of BlockManager.java is not introduced here. The failure of {{TestBlockManager.testBlocksAreNotUnderreplicatedInSingleRack}} seems not to be related to the patch and I could not reproduce it in my environment.


> Blocks in newly completed files are considered under-replicated too quickly
> ---------------------------------------------------------------------------
>
>                 Key: HDFS-1172
>                 URL: https://issues.apache.org/jira/browse/HDFS-1172
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>         Attachments: HDFS-1172-150907.patch, HDFS-1172.008.patch, HDFS-1172.009.patch, HDFS-1172.010.patch, HDFS-1172.patch, hdfs-1172.txt, hdfs-1172.txt, replicateBlocksFUC.patch, replicateBlocksFUC1.patch, replicateBlocksFUC1.patch
>
>
> I've seen this for a long time, and imagine it's a known issue, but couldn't find an existing JIRA. It often happens that we see the NN schedule replication on the last block of files very quickly after they're completed, before the other DNs in the pipeline have a chance to report the new block. This results in a lot of extra replication work on the cluster, as we replicate the block and then end up with multiple excess replicas which are very quickly deleted.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)