Return-Path: Delivered-To: apmail-hadoop-hdfs-commits-archive@minotaur.apache.org Received: (qmail 76645 invoked from network); 5 Apr 2011 21:27:36 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 5 Apr 2011 21:27:36 -0000 Received: (qmail 53693 invoked by uid 500); 5 Apr 2011 21:27:36 -0000 Delivered-To: apmail-hadoop-hdfs-commits-archive@hadoop.apache.org Received: (qmail 53657 invoked by uid 500); 5 Apr 2011 21:27:36 -0000 Mailing-List: contact hdfs-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-dev@hadoop.apache.org Delivered-To: mailing list hdfs-commits@hadoop.apache.org Received: (qmail 53649 invoked by uid 99); 5 Apr 2011 21:27:36 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Apr 2011 21:27:36 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Apr 2011 21:27:31 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id EF3B523889ED; Tue, 5 Apr 2011 21:27:08 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1089234 - in /hadoop/hdfs/branches/HDFS-1052: ./ src/c++/libhdfs/ src/contrib/hdfsproxy/ src/java/ src/java/org/apache/hadoop/hdfs/ src/java/org/apache/hadoop/hdfs/server/datanode/ src/test/hdfs/ src/test/hdfs/org/apache/hadoop/hdfs/ src/w... Date: Tue, 05 Apr 2011 21:27:08 -0000 To: hdfs-commits@hadoop.apache.org From: suresh@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20110405212708.EF3B523889ED@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: suresh Date: Tue Apr 5 21:27:08 2011 New Revision: 1089234 URL: http://svn.apache.org/viewvc?rev=1089234&view=rev Log: Merging change r1036767 from trunk to federation Added: hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/org/apache/hadoop/hdfs/BlockReaderTestUtil.java - copied, changed from r1036767, hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/BlockReaderTestUtil.java Modified: hadoop/hdfs/branches/HDFS-1052/ (props changed) hadoop/hdfs/branches/HDFS-1052/CHANGES.txt hadoop/hdfs/branches/HDFS-1052/build.xml (props changed) hadoop/hdfs/branches/HDFS-1052/src/c++/libhdfs/ (props changed) hadoop/hdfs/branches/HDFS-1052/src/contrib/hdfsproxy/ (props changed) hadoop/hdfs/branches/HDFS-1052/src/java/ (props changed) hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/BlockReader.java hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java (props changed) hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/ (props changed) hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/org/apache/hadoop/hdfs/TestClientBlockVerification.java hadoop/hdfs/branches/HDFS-1052/src/webapps/datanode/ (props changed) hadoop/hdfs/branches/HDFS-1052/src/webapps/hdfs/ (props changed) hadoop/hdfs/branches/HDFS-1052/src/webapps/secondary/ (props changed) Propchange: hadoop/hdfs/branches/HDFS-1052/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Apr 5 21:27:08 2011 @@ -1,4 +1,4 @@ /hadoop/core/branches/branch-0.19/hdfs:713112 /hadoop/hdfs/branches/HDFS-265:796829-820463 /hadoop/hdfs/branches/branch-0.21:820487 -/hadoop/hdfs/trunk:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036738,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 +/hadoop/hdfs/trunk:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036738,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 Modified: hadoop/hdfs/branches/HDFS-1052/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/hdfs/branches/HDFS-1052/CHANGES.txt?rev=1089234&r1=1089233&r2=1089234&view=diff ============================================================================== --- hadoop/hdfs/branches/HDFS-1052/CHANGES.txt (original) +++ hadoop/hdfs/branches/HDFS-1052/CHANGES.txt Tue Apr 5 21:27:08 2011 @@ -814,6 +814,9 @@ Release 0.22.0 - Unreleased HDFS-981. test-contrib fails due to test-cactus failure (cos) + HDFS-1001. DataXceiver and BlockReader disagree on when to send/recv + CHECKSUM_OK. (bc Wong via eli) + Release 0.21.1 - Unreleased HDFS-1411. Correct backup node startup command in hdfs user guide. Propchange: hadoop/hdfs/branches/HDFS-1052/build.xml ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Apr 5 21:27:08 2011 @@ -2,4 +2,4 @@ /hadoop/core/trunk/build.xml:779102 /hadoop/hdfs/branches/HDFS-265/build.xml:796829-820463 /hadoop/hdfs/branches/branch-0.21/build.xml:820487 -/hadoop/hdfs/trunk/build.xml:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 +/hadoop/hdfs/trunk/build.xml:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 Propchange: hadoop/hdfs/branches/HDFS-1052/src/c++/libhdfs/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Apr 5 21:27:08 2011 @@ -1,3 +1,3 @@ /hadoop/core/branches/branch-0.19/mapred/src/c++/libhdfs:713112 /hadoop/core/trunk/src/c++/libhdfs:776175-784663 -/hadoop/hdfs/trunk/src/c++/libhdfs:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 +/hadoop/hdfs/trunk/src/c++/libhdfs:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 Propchange: hadoop/hdfs/branches/HDFS-1052/src/contrib/hdfsproxy/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Apr 5 21:27:08 2011 @@ -2,4 +2,4 @@ /hadoop/core/trunk/src/contrib/hdfsproxy:776175-784663 /hadoop/hdfs/branches/HDFS-265/src/contrib/hdfsproxy:796829-820463 /hadoop/hdfs/branches/branch-0.21/src/contrib/hdfsproxy:820487 -/hadoop/hdfs/trunk/src/contrib/hdfsproxy:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 +/hadoop/hdfs/trunk/src/contrib/hdfsproxy:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 Propchange: hadoop/hdfs/branches/HDFS-1052/src/java/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Apr 5 21:27:08 2011 @@ -2,4 +2,4 @@ /hadoop/core/trunk/src/hdfs:776175-785643,785929-786278 /hadoop/hdfs/branches/HDFS-265/src/java:796829-820463 /hadoop/hdfs/branches/branch-0.21/src/java:820487 -/hadoop/hdfs/trunk/src/java:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 +/hadoop/hdfs/trunk/src/java:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 Modified: hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/BlockReader.java URL: http://svn.apache.org/viewvc/hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/BlockReader.java?rev=1089234&r1=1089233&r2=1089234&view=diff ============================================================================== --- hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/BlockReader.java (original) +++ hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/BlockReader.java Tue Apr 5 21:27:08 2011 @@ -433,7 +433,7 @@ public class BlockReader extends FSInput return readFully(this, buf, offset, len); } - /* When the reader reaches end of a block and there are no checksum + /* When the reader reaches end of the read and there are no checksum * errors, we send OP_STATUS_CHECKSUM_OK to datanode to inform that * checksum was verified and there was no error. */ Modified: hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java URL: http://svn.apache.org/viewvc/hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java?rev=1089234&r1=1089233&r2=1089234&view=diff ============================================================================== --- hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java (original) +++ hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java Tue Apr 5 21:27:08 2011 @@ -1850,10 +1850,12 @@ public class DataNode extends Configured "LastPacketInBlock" set to true or with a zero length. If there is no checksum error, it replies to DataNode with OP_STATUS_CHECKSUM_OK: - Client optional response at the end of data transmission : + Client optional response at the end of data transmission of any length: +------------------------------+ | 2 byte OP_STATUS_CHECKSUM_OK | +------------------------------+ + The DataNode always checks OP_STATUS_CHECKSUM_OK. It will close the + client connection if it is absent. PACKET : Contains a packet header, checksum and data. Amount of data ======== carried is set by BUFFER_SIZE. Propchange: hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Apr 5 21:27:08 2011 @@ -4,4 +4,4 @@ /hadoop/core/trunk/src/hdfs/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java:776175-785643,785929-786278 /hadoop/hdfs/branches/HDFS-265/src/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java:796829-820463 /hadoop/hdfs/branches/branch-0.21/src/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java:820487 -/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java:1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 +/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java:1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 Propchange: hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Apr 5 21:27:08 2011 @@ -2,4 +2,4 @@ /hadoop/core/trunk/src/test/hdfs:776175-785643 /hadoop/hdfs/branches/HDFS-265/src/test/hdfs:796829-820463 /hadoop/hdfs/branches/branch-0.21/src/test/hdfs:820487 -/hadoop/hdfs/trunk/src/test/hdfs:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 +/hadoop/hdfs/trunk/src/test/hdfs:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 Copied: hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/org/apache/hadoop/hdfs/BlockReaderTestUtil.java (from r1036767, hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/BlockReaderTestUtil.java) URL: http://svn.apache.org/viewvc/hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/org/apache/hadoop/hdfs/BlockReaderTestUtil.java?p2=hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/org/apache/hadoop/hdfs/BlockReaderTestUtil.java&p1=hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/BlockReaderTestUtil.java&r1=1036767&r2=1089234&rev=1089234&view=diff ============================================================================== --- hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/BlockReaderTestUtil.java (original) +++ hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/org/apache/hadoop/hdfs/BlockReaderTestUtil.java Tue Apr 5 21:27:08 2011 @@ -25,8 +25,8 @@ import java.util.Random; import java.util.List; import java.io.IOException; -import org.apache.hadoop.hdfs.protocol.Block; import org.apache.hadoop.hdfs.protocol.DatanodeInfo; +import org.apache.hadoop.hdfs.protocol.ExtendedBlock; import org.apache.hadoop.hdfs.protocol.LocatedBlock; import org.apache.hadoop.hdfs.server.common.HdfsConstants; import org.apache.hadoop.hdfs.server.datanode.DataNode; @@ -125,7 +125,7 @@ public class BlockReaderTestUtil { throws IOException { InetSocketAddress targetAddr = null; Socket sock = null; - Block block = testBlock.getBlock(); + ExtendedBlock block = testBlock.getBlock(); DatanodeInfo[] nodes = testBlock.getLocations(); targetAddr = NetUtils.createSocketAddr(nodes[0].getName()); sock = new Socket(); Modified: hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/org/apache/hadoop/hdfs/TestClientBlockVerification.java URL: http://svn.apache.org/viewvc/hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/org/apache/hadoop/hdfs/TestClientBlockVerification.java?rev=1089234&r1=1089233&r2=1089234&view=diff ============================================================================== --- hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/org/apache/hadoop/hdfs/TestClientBlockVerification.java (original) +++ hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/org/apache/hadoop/hdfs/TestClientBlockVerification.java Tue Apr 5 21:27:08 2011 @@ -18,21 +18,11 @@ package org.apache.hadoop.hdfs; -import java.net.Socket; -import java.net.InetSocketAddress; -import java.io.DataOutputStream; -import java.util.Random; import java.util.List; -import java.io.IOException; -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hdfs.protocol.DatanodeInfo; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; import org.apache.hadoop.hdfs.protocol.LocatedBlock; -import org.apache.hadoop.hdfs.server.common.HdfsConstants; import org.apache.hadoop.fs.Path; -import org.apache.hadoop.fs.FileSystem; -import org.apache.hadoop.net.NetUtils; import org.junit.Test; import org.junit.AfterClass; @@ -44,57 +34,18 @@ import static org.mockito.Mockito.never; import static org.junit.Assert.*; public class TestClientBlockVerification { - static MiniDFSCluster cluster = null; - static Configuration conf = null; - static FileSystem fs = null; + + static BlockReaderTestUtil util = null; static final Path TEST_FILE = new Path("/test.file"); static final int FILE_SIZE_K = 256; static LocatedBlock testBlock = null; @BeforeClass public static void setupCluster() throws Exception { - conf = new HdfsConfiguration(); - int numDataNodes = 1; - conf.setInt("dfs.replication", numDataNodes); - cluster = new MiniDFSCluster.Builder(conf).numDataNodes(numDataNodes).build(); - cluster.waitActive(); - fs = cluster.getFileSystem(); - - // Write a file with 256K of data - DataOutputStream os = fs.create(TEST_FILE); - byte data[] = new byte[1024]; - new Random().nextBytes(data); - for (int i = 0; i < FILE_SIZE_K; i++) { - os.write(data); - } - os.close(); - - // Locate the block we just wrote - DFSClient dfsclient = new DFSClient( - new InetSocketAddress("localhost", - cluster.getNameNodePort()), conf); - List locatedBlocks = dfsclient.getNamenode().getBlockLocations( - TEST_FILE.toString(), 0, FILE_SIZE_K * 1024).getLocatedBlocks(); - testBlock = locatedBlocks.get(0); // first block - } - - private BlockReader getBlockReader( - int offset, int lenToRead) throws IOException { - InetSocketAddress targetAddr = null; - Socket s = null; - ExtendedBlock block = testBlock.getBlock(); - DatanodeInfo[] nodes = testBlock.getLocations(); - targetAddr = NetUtils.createSocketAddr(nodes[0].getName()); - s = new Socket(); - s.connect(targetAddr, HdfsConstants.READ_TIMEOUT); - s.setSoTimeout(HdfsConstants.READ_TIMEOUT); - - String file = BlockReader.getFileName(targetAddr, - "test-blockpoolid", - block.getBlockId()); - return BlockReader.newBlockReader(s, file, block, - testBlock.getBlockToken(), offset, lenToRead, conf.getInt( - "io.file.buffer.size", 4096)); + final int REPLICATION_FACTOR = 1; + util = new BlockReaderTestUtil(REPLICATION_FACTOR); + List blkList = util.writeFile(TEST_FILE, FILE_SIZE_K); + testBlock = blkList.get(0); // Use the first block to test } /** @@ -102,8 +53,8 @@ public class TestClientBlockVerification */ @Test public void testBlockVerification() throws Exception { - BlockReader reader = spy(getBlockReader(0, FILE_SIZE_K * 1024)); - slurpReader(reader, FILE_SIZE_K * 1024, true); + BlockReader reader = spy(util.getBlockReader(testBlock, 0, FILE_SIZE_K * 1024)); + util.readAndCheckEOS(reader, FILE_SIZE_K * 1024, true); verify(reader).checksumOk(reader.dnSock); reader.close(); } @@ -113,8 +64,8 @@ public class TestClientBlockVerification */ @Test public void testIncompleteRead() throws Exception { - BlockReader reader = spy(getBlockReader(0, FILE_SIZE_K * 1024)); - slurpReader(reader, FILE_SIZE_K / 2 * 1024, false); + BlockReader reader = spy(util.getBlockReader(testBlock, 0, FILE_SIZE_K * 1024)); + util.readAndCheckEOS(reader, FILE_SIZE_K / 2 * 1024, false); // We asked the blockreader for the whole file, and only read // half of it, so no checksumOk @@ -130,9 +81,9 @@ public class TestClientBlockVerification @Test public void testCompletePartialRead() throws Exception { // Ask for half the file - BlockReader reader = spy(getBlockReader(0, FILE_SIZE_K * 1024 / 2)); + BlockReader reader = spy(util.getBlockReader(testBlock, 0, FILE_SIZE_K * 1024 / 2)); // And read half the file - slurpReader(reader, FILE_SIZE_K * 1024 / 2, true); + util.readAndCheckEOS(reader, FILE_SIZE_K * 1024 / 2, true); verify(reader).checksumOk(reader.dnSock); reader.close(); } @@ -149,8 +100,8 @@ public class TestClientBlockVerification for (int length : lengths) { DFSClient.LOG.info("Testing startOffset = " + startOffset + " and " + " len=" + length); - BlockReader reader = spy(getBlockReader(startOffset, length)); - slurpReader(reader, length, true); + BlockReader reader = spy(util.getBlockReader(testBlock, startOffset, length)); + util.readAndCheckEOS(reader, length, true); verify(reader).checksumOk(reader.dnSock); reader.close(); } @@ -158,32 +109,9 @@ public class TestClientBlockVerification } - /** - * Read the given length from the given block reader. - * - * @param expectEOF if true, will expect an eof response when done - */ - private void slurpReader(BlockReader reader, int length, boolean expectEof) - throws IOException { - byte buf[] = new byte[1024]; - int nRead = 0; - while (nRead < length) { - DFSClient.LOG.info("So far read " + nRead + " - going to read more."); - int n = reader.read(buf, 0, buf.length); - assertTrue(n > 0); - nRead += n; - } - DFSClient.LOG.info("Done reading, expect EOF for next read."); - if (expectEof) { - assertEquals(-1, reader.read(buf, 0, buf.length)); - } - } - @AfterClass public static void teardownCluster() throws Exception { - if (cluster != null) { - cluster.shutdown(); - } + util.shutdown(); } } Propchange: hadoop/hdfs/branches/HDFS-1052/src/webapps/datanode/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Apr 5 21:27:08 2011 @@ -2,4 +2,4 @@ /hadoop/core/trunk/src/webapps/datanode:776175-784663 /hadoop/hdfs/branches/HDFS-265/src/webapps/datanode:796829-820463 /hadoop/hdfs/branches/branch-0.21/src/webapps/datanode:820487 -/hadoop/hdfs/trunk/src/webapps/datanode:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 +/hadoop/hdfs/trunk/src/webapps/datanode:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 Propchange: hadoop/hdfs/branches/HDFS-1052/src/webapps/hdfs/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Apr 5 21:27:08 2011 @@ -2,4 +2,4 @@ /hadoop/core/trunk/src/webapps/hdfs:776175-784663 /hadoop/hdfs/branches/HDFS-265/src/webapps/hdfs:796829-820463 /hadoop/hdfs/branches/branch-0.21/src/webapps/hdfs:820487 -/hadoop/hdfs/trunk/src/webapps/hdfs:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 +/hadoop/hdfs/trunk/src/webapps/hdfs:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 Propchange: hadoop/hdfs/branches/HDFS-1052/src/webapps/secondary/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Apr 5 21:27:08 2011 @@ -2,4 +2,4 @@ /hadoop/core/trunk/src/webapps/secondary:776175-784663 /hadoop/hdfs/branches/HDFS-265/src/webapps/secondary:796829-820463 /hadoop/hdfs/branches/branch-0.21/src/webapps/secondary:820487 -/hadoop/hdfs/trunk/src/webapps/secondary:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021 +/hadoop/hdfs/trunk/src/webapps/secondary:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021