Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Thu, 30 Oct 2014 14:01:35 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12751132.1414513727000.372301.1414677695622@Atlassian.JIRA>
In-Reply-To: <JIRA.12751132.1414513727000@Atlassian.JIRA>
References: <JIRA.12751132.1414513727000@Atlassian.JIRA>
 <JIRA.12751132.1414513727007@arcas>
Subject: [jira] [Commented] (HDFS-7300) The getMaxNodesPerRack() method in
 BlockPlacementPolicyDefault is flawed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-7300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190077#comment-14190077 ] 

Hudson commented on HDFS-7300:
------------------------------

FAILURE: Integrated in Hadoop-Hdfs-trunk #1917 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1917/])
HDFS-7300.     HDFS-7300. The getMaxNodesPerRack() method in (kihwal: rev 3ae84e1ba8928879b3eda90e79667ba5a45d60f8)
* hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
* hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppendRestart.java
* hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> The getMaxNodesPerRack() method in BlockPlacementPolicyDefault is flawed
> ------------------------------------------------------------------------
>
>                 Key: HDFS-7300
>                 URL: https://issues.apache.org/jira/browse/HDFS-7300
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>             Fix For: 2.6.0
>
>         Attachments: HDFS-7300.patch, HDFS-7300.v2.patch
>
>
> The {{getMaxNodesPerRack()}} can produce an undesirable result in some cases.
> - Three replicas on two racks. The max is 3, so everything can go to one rack.
> - Two replicas on two or more racks. The max is 2, both replicas can end up in the same rack.
> {{BlockManager#isNeededReplication()}} fixes this after block/file is closed because {{blockHasEnoughRacks()}} will return fail.  This is not only extra work, but also can break the favored nodes feature.
> When there are two racks and two favored nodes are specified in the same rack, NN may allocate the third replica on a node in the same rack, because {{maxNodesPerRack}} is 3. When closing the file, NN moves a block to the other rack. There is 66% chance that a favored node is moved.  If {{maxNodesPerRack}} was 2, this would not happen.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)