Return-Path: X-Original-To: apmail-hadoop-common-commits-archive@www.apache.org Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5EFBB1816A for ; Fri, 5 Jun 2015 20:39:53 +0000 (UTC) Received: (qmail 66191 invoked by uid 500); 5 Jun 2015 20:39:53 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 66115 invoked by uid 500); 5 Jun 2015 20:39:53 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 66106 invoked by uid 99); 5 Jun 2015 20:39:53 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Jun 2015 20:39:53 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 05940E07E9; Fri, 5 Jun 2015 20:39:53 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: zhz@apache.org To: common-commits@hadoop.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: hadoop git commit: HDFS-8460. Erasure Coding: stateful read result doesn't match data occasionally because of flawed test. Contributed by Walter Su. Date: Fri, 5 Jun 2015 20:39:53 +0000 (UTC) Repository: hadoop Updated Branches: refs/heads/HDFS-7285 c0929ab3c -> 2eee19cd1 HDFS-8460. Erasure Coding: stateful read result doesn't match data occasionally because of flawed test. Contributed by Walter Su. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/2eee19cd Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/2eee19cd Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/2eee19cd Branch: refs/heads/HDFS-7285 Commit: 2eee19cd1d8e16f0631d78474e387ca4c526f47e Parents: c0929ab Author: Zhe Zhang Authored: Fri Jun 5 13:39:37 2015 -0700 Committer: Zhe Zhang Committed: Fri Jun 5 13:39:37 2015 -0700 ---------------------------------------------------------------------- hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt | 3 +++ .../src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java | 2 ++ .../java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java | 5 +++++ 3 files changed, 10 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/2eee19cd/hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt index 9b05d2b..61d7a76 100755 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt +++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt @@ -286,3 +286,6 @@ HDFS-8319. Erasure Coding: support decoding for stateful read. (Jing Zhao via zhz) + + HDFS-8460. Erasure Coding: stateful read result doesn't match data + occasionally because of flawed test. (Walter Su via zhz) http://git-wip-us.apache.org/repos/asf/hadoop/blob/2eee19cd/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java index db230e3..b9ded80 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java @@ -1859,6 +1859,7 @@ public class DFSTestUtil { /** * Creates the metadata of a file in striped layout. This method only * manipulates the NameNode state without injecting data to DataNode. + * You should disable periodical heartbeat before use this. * @param file Path of the file to create * @param dir Parent path of the file * @param numBlocks Number of striped block groups to add to the file @@ -1908,6 +1909,7 @@ public class DFSTestUtil { * Adds a striped block group to a file. This method only manipulates NameNode * states of the file and the block without injecting data to DataNode. * It does mimic block reports. + * You should disable periodical heartbeat before use this. * @param dataNodes List DataNodes to host the striped block group * @param previous Previous block in the file * @param numStripes Number of stripes in each block group http://git-wip-us.apache.org/repos/asf/hadoop/blob/2eee19cd/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java index de43441..b53983b 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java @@ -32,6 +32,8 @@ import static org.junit.Assert.assertArrayEquals; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; +import org.apache.hadoop.hdfs.server.datanode.DataNode; +import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils; import org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset; import org.apache.hadoop.hdfs.server.namenode.ErasureCodingSchemaManager; import org.apache.hadoop.hdfs.util.StripedBlockUtil; @@ -71,6 +73,9 @@ public class TestDFSStripedInputStream { cluster = new MiniDFSCluster.Builder(conf).numDataNodes( DATA_BLK_NUM + PARITY_BLK_NUM).build(); cluster.waitActive(); + for (DataNode dn : cluster.getDataNodes()) { + DataNodeTestUtils.setHeartbeatsDisabledForTests(dn, true); + } fs = cluster.getFileSystem(); fs.mkdirs(dirPath); fs.getClient().createErasureCodingZone(dirPath.toString(), null, CELLSIZE);