Return-Path: 
 <hdfs-issues-return-123054-apmail-hadoop-hdfs-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DE8B3176C0
	for <apmail-hadoop-hdfs-issues-archive@minotaur.apache.org>;
 Mon, 18 May 2015 05:33:00 +0000 (UTC)
Received: (qmail 57759 invoked by uid 500); 18 May 2015 05:33:00 -0000
Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org
Received: (qmail 57710 invoked by uid 500); 18 May 2015 05:33:00 -0000
Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hdfs-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:hdfs-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:hdfs-issues@hadoop.apache.org>
List-Id: <hdfs-issues.hadoop.apache.org>
Reply-To: hdfs-issues@hadoop.apache.org
Delivered-To: mailing list hdfs-issues@hadoop.apache.org
Received: (qmail 57693 invoked by uid 99); 18 May 2015 05:33:00 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 May 2015 05:33:00 +0000
Date: Mon, 18 May 2015 05:33:00 +0000 (UTC)
From: "Yi Liu (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12830576.1431926576000.140080.1431927180385@Atlassian.JIRA>
In-Reply-To: <JIRA.12830576.1431926576000@Atlassian.JIRA>
References: <JIRA.12830576.1431926576000@Atlassian.JIRA>
 <JIRA.12830576.1431926576811@arcas>
Subject: [jira] [Commented] (HDFS-8418) Fix the isNeededReplication
 calculation for Striped block in NN
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HDFS-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14547551#comment-14547551 ] 

Yi Liu commented on HDFS-8418:
------------------------------

It will cause some unexpected behavior, for example: schedule unnecessary striped block reconstruction work, I can see the exception from the log:
{code}
2015-05-18 13:25:57,955 WARN  datanode.DataNode (ErasureCodingWorker.java:processErasureCodingTasks(158)) - Failed to recover striped block blk_-9223372036854775792_1001
java.lang.IllegalArgumentException: No enough live striped blocks.
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
	at org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker$ReconstructAndTransferBlock.<init>(ErasureCodingWorker.java:278)
	at org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker.processErasureCodingTasks(ErasureCodingWorker.java:156)
	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:730)
	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:617)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:854)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:671)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:820)
	at java.lang.Thread.run(Thread.java:744)
{code}


> Fix the isNeededReplication calculation for Striped block in NN
> ---------------------------------------------------------------
>
>                 Key: HDFS-8418
>                 URL: https://issues.apache.org/jira/browse/HDFS-8418
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Yi Liu
>            Assignee: Yi Liu
>            Priority: Critical
>
> Currently when calculating {{isNeededReplication}} for striped block, we use BlockCollection#getPreferredBlockReplication to get expected replica number for striped block. See an example:
> {code}
> public void checkReplication(BlockCollection bc) {
>     final short expected = bc.getPreferredBlockReplication();
>     for (BlockInfo block : bc.getBlocks()) {
>       final NumberReplicas n = countNodes(block);
>       if (isNeededReplication(block, expected, n.liveReplicas())) { 
>         neededReplications.add(block, n.liveReplicas(),
>             n.decommissionedAndDecommissioning(), expected);
>       } else if (n.liveReplicas() > expected) {
>         processOverReplicatedBlock(block, expected, null, null);
>       }
>     }
>   }
> {code}
> But actually it's not correct, for example, if the length of striped file is less than a cell, then the expected replica of the block should be {{1 + parityBlkNum}} instead of {{dataBlkNum + parityBlkNum}}. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)