Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Thu, 8 Oct 2015 16:27:28 +0000 (UTC)
From: "Masatake Iwasaki (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12465184.1274509229000.58151.1444321648132@Atlassian.JIRA>
In-Reply-To: <JIRA.12465184.1274509229000@Atlassian.JIRA>
References: <JIRA.12465184.1274509229000@Atlassian.JIRA>
 <JIRA.12465184.1274509229369@arcas>
Subject: [jira] [Updated] (HDFS-1172) Blocks in newly completed files are
 considered under-replicated too quickly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Masatake Iwasaki updated HDFS-1172:
-----------------------------------
    Attachment: HDFS-1172.012.patch

I update the patch.
* addressed the failure of TestRecoverStripedFile: fixed to avoid updating pendingReplications if file is striped.
* added calling to {{DataNodeTestUtils#triggerHeartbeat}} in order to make sure {{TestReplication#testNoExtraReplicationWhenBlockReceivedIsLate}} fails without the fix of BlockManager.
* fixed checkstyle warning except for file length.
* fixed whitespace error.
* release audit is not related to the fix.
* failure of TestBlockReport and TestCheckpoint is not related to the code path of the patch. I could not reproduce the failure on my env.

> Blocks in newly completed files are considered under-replicated too quickly
> ---------------------------------------------------------------------------
>
>                 Key: HDFS-1172
>                 URL: https://issues.apache.org/jira/browse/HDFS-1172
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Masatake Iwasaki
>         Attachments: HDFS-1172-150907.patch, HDFS-1172.008.patch, HDFS-1172.009.patch, HDFS-1172.010.patch, HDFS-1172.011.patch, HDFS-1172.012.patch, HDFS-1172.patch, hdfs-1172.txt, hdfs-1172.txt, replicateBlocksFUC.patch, replicateBlocksFUC1.patch, replicateBlocksFUC1.patch
>
>
> I've seen this for a long time, and imagine it's a known issue, but couldn't find an existing JIRA. It often happens that we see the NN schedule replication on the last block of files very quickly after they're completed, before the other DNs in the pipeline have a chance to report the new block. This results in a lot of extra replication work on the cluster, as we replicate the block and then end up with multiple excess replicas which are very quickly deleted.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)