Return-Path: Delivered-To: apmail-hadoop-hdfs-commits-archive@minotaur.apache.org Received: (qmail 75217 invoked from network); 7 Apr 2011 00:25:44 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 7 Apr 2011 00:25:44 -0000 Received: (qmail 71341 invoked by uid 500); 7 Apr 2011 00:25:44 -0000 Delivered-To: apmail-hadoop-hdfs-commits-archive@hadoop.apache.org Received: (qmail 71304 invoked by uid 500); 7 Apr 2011 00:25:44 -0000 Mailing-List: contact hdfs-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-dev@hadoop.apache.org Delivered-To: mailing list hdfs-commits@hadoop.apache.org Received: (qmail 71296 invoked by uid 99); 7 Apr 2011 00:25:44 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Apr 2011 00:25:44 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Apr 2011 00:25:41 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 5907E238890A; Thu, 7 Apr 2011 00:25:20 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1089696 - in /hadoop/hdfs/branches/HDFS-1052: ./ bin/ src/c++/libhdfs/ src/contrib/hdfsproxy/ src/java/ src/java/org/apache/hadoop/hdfs/ src/java/org/apache/hadoop/hdfs/server/datanode/ src/java/org/apache/hadoop/hdfs/server/namenode/ src/... Date: Thu, 07 Apr 2011 00:25:19 -0000 To: hdfs-commits@hadoop.apache.org From: suresh@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20110407002520.5907E238890A@eris.apache.org> Author: suresh Date: Thu Apr 7 00:25:19 2011 New Revision: 1089696 URL: http://svn.apache.org/viewvc?rev=1089696&view=rev Log: Merging changes r1086461:r1087160 from trunk to federation Modified: hadoop/hdfs/branches/HDFS-1052/ (props changed) hadoop/hdfs/branches/HDFS-1052/CHANGES.txt hadoop/hdfs/branches/HDFS-1052/bin/hdfs hadoop/hdfs/branches/HDFS-1052/build.xml (props changed) hadoop/hdfs/branches/HDFS-1052/src/c++/libhdfs/ (props changed) hadoop/hdfs/branches/HDFS-1052/src/contrib/hdfsproxy/ (props changed) hadoop/hdfs/branches/HDFS-1052/src/java/ (props changed) hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/DFSClient.java hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/DFSOutputStream.java hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java (props changed) hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/ (props changed) hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml hadoop/hdfs/branches/HDFS-1052/src/webapps/datanode/ (props changed) hadoop/hdfs/branches/HDFS-1052/src/webapps/hdfs/ (props changed) hadoop/hdfs/branches/HDFS-1052/src/webapps/secondary/ (props changed) Propchange: hadoop/hdfs/branches/HDFS-1052/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Thu Apr 7 00:25:19 2011 @@ -1,4 +1,4 @@ /hadoop/core/branches/branch-0.19/hdfs:713112 /hadoop/hdfs/branches/HDFS-265:796829-820463 /hadoop/hdfs/branches/branch-0.21:820487 -/hadoop/hdfs/trunk:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036738,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461 +/hadoop/hdfs/trunk:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036738,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461,1086479,1086654,1086693,1086820,1087080,1087096,1087115,1087160 Modified: hadoop/hdfs/branches/HDFS-1052/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/hdfs/branches/HDFS-1052/CHANGES.txt?rev=1089696&r1=1089695&r2=1089696&view=diff ============================================================================== --- hadoop/hdfs/branches/HDFS-1052/CHANGES.txt (original) +++ hadoop/hdfs/branches/HDFS-1052/CHANGES.txt Thu Apr 7 00:25:19 2011 @@ -251,10 +251,6 @@ Trunk (unreleased changes) HDFS-1791. Federation: Add command to delete block pool directories from a datanode. (jitendra) - HDFS-1785. In BlockReceiver and DataXceiver, clientName.length() is used - multiple times for determining whether the source is a client or a - datanode. (szetszwo) - IMPROVEMENTS HDFS-1510. Added test-patch.properties required by test-patch.sh (nigel) @@ -300,9 +296,6 @@ Trunk (unreleased changes) HDFS-1736. Remove the dependency from DatanodeJspHelper to FsShell. (Daryn Sharp via szetszwo) - HDFS-1731. Amend previous commit for this JIRA to fix build on Cygwin. - (todd) - HDFS-780. Revive TestFuseDFS. (eli) HDFS-1445. Batch the calls in DataStorage to FileUtil.createHardLink(). @@ -317,6 +310,16 @@ Trunk (unreleased changes) HDFS-1120. Make DataNode's block-to-device placement policy pluggable (Harsh J Chouraria via todd) + HDFS-1785. In BlockReceiver and DataXceiver, clientName.length() is used + multiple times for determining whether the source is a client or a + datanode. (szetszwo) + + HDFS-1789. Refactor frequently used codes from DFSOutputStream and + DataXceiver. (szetszwo) + + HDFS-1767. Namenode ignores non-initial block report from datanodes + when in safemode during startup. (Matt Foley via suresh) + OPTIMIZATIONS HDFS-1458. Improve checkpoint performance by avoiding unnecessary image @@ -379,6 +382,11 @@ Trunk (unreleased changes) HDFS-1770. TestFiRename fails due to invalid block size. (eli) + HDFS-1797. Fix new findbugs warning introduced by HDFS-1120 (todd) + + HDFS-1611. Fix up some log messages in DFSClient and MBean registration + (Uma Maheswara Rao G via todd) + Release 0.22.0 - Unreleased NEW FEATURES @@ -832,6 +840,11 @@ Release 0.22.0 - Unreleased HDFS-1001. DataXceiver and BlockReader disagree on when to send/recv CHECKSUM_OK. (bc Wong via eli) + HDFS-1781. Fix the path for jsvc in bin/hdfs. (John George via szetszwo) + + HDFS-1782. Fix an NPE in RFSNamesystem.startFileInternal(..). + (John George via szetszwo) + Release 0.21.1 - Unreleased HDFS-1411. Correct backup node startup command in hdfs user guide. @@ -885,6 +898,9 @@ Release 0.21.1 - Unreleased HDFS-1596. Replace fs.checkpoint.* with dfs.namenode.checkpoint.* in documentations. (Harsh J Chouraria via szetszwo) + HDFS-1786. Some cli test cases expect a "null" message + (Uma Maheswara Rao G via todd) + Release 0.21.0 - 2010-08-13 INCOMPATIBLE CHANGES Modified: hadoop/hdfs/branches/HDFS-1052/bin/hdfs URL: http://svn.apache.org/viewvc/hadoop/hdfs/branches/HDFS-1052/bin/hdfs?rev=1089696&r1=1089695&r2=1089696&view=diff ============================================================================== --- hadoop/hdfs/branches/HDFS-1052/bin/hdfs (original) +++ hadoop/hdfs/branches/HDFS-1052/bin/hdfs Thu Apr 7 00:25:19 2011 @@ -143,7 +143,7 @@ if [ "$starting_secure_dn" = "true" ]; t HADOOP_SECURE_DN_PID="$HADOOP_PID_DIR/hadoop_secure_dn.pid" fi - exec "$HADOOP_HOME/bin/jsvc" \ + exec "$HADOOP_HDFS_HOME/bin/jsvc" \ -Dproc_$COMMAND -outfile "$HADOOP_LOG_DIR/jsvc.out" \ -errfile "$HADOOP_LOG_DIR/jsvc.err" \ -pidfile "$HADOOP_SECURE_DN_PID" \ Propchange: hadoop/hdfs/branches/HDFS-1052/build.xml ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Thu Apr 7 00:25:19 2011 @@ -2,4 +2,4 @@ /hadoop/core/trunk/build.xml:779102 /hadoop/hdfs/branches/HDFS-265/build.xml:796829-820463 /hadoop/hdfs/branches/branch-0.21/build.xml:820487 -/hadoop/hdfs/trunk/build.xml:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461 +/hadoop/hdfs/trunk/build.xml:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461,1086479,1086654,1086693,1086820,1087080,1087096,1087115,1087160 Propchange: hadoop/hdfs/branches/HDFS-1052/src/c++/libhdfs/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Thu Apr 7 00:25:19 2011 @@ -1,3 +1,3 @@ /hadoop/core/branches/branch-0.19/mapred/src/c++/libhdfs:713112 /hadoop/core/trunk/src/c++/libhdfs:776175-784663 -/hadoop/hdfs/trunk/src/c++/libhdfs:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461 +/hadoop/hdfs/trunk/src/c++/libhdfs:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461,1086479,1086654,1086693,1086820,1087080,1087096,1087115,1087160 Propchange: hadoop/hdfs/branches/HDFS-1052/src/contrib/hdfsproxy/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Thu Apr 7 00:25:19 2011 @@ -2,4 +2,4 @@ /hadoop/core/trunk/src/contrib/hdfsproxy:776175-784663 /hadoop/hdfs/branches/HDFS-265/src/contrib/hdfsproxy:796829-820463 /hadoop/hdfs/branches/branch-0.21/src/contrib/hdfsproxy:820487 -/hadoop/hdfs/trunk/src/contrib/hdfsproxy:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461 +/hadoop/hdfs/trunk/src/contrib/hdfsproxy:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461,1086479,1086654,1086693,1086820,1087080,1087096,1087115,1087160 Propchange: hadoop/hdfs/branches/HDFS-1052/src/java/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Thu Apr 7 00:25:19 2011 @@ -2,4 +2,4 @@ /hadoop/core/trunk/src/hdfs:776175-785643,785929-786278 /hadoop/hdfs/branches/HDFS-265/src/java:796829-820463 /hadoop/hdfs/branches/branch-0.21/src/java:820487 -/hadoop/hdfs/trunk/src/java:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461 +/hadoop/hdfs/trunk/src/java:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461,1086479,1086654,1086693,1086820,1087080,1087096,1087115,1087160 Modified: hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/DFSClient.java URL: http://svn.apache.org/viewvc/hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/DFSClient.java?rev=1089696&r1=1089695&r2=1089696&view=diff ============================================================================== --- hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/DFSClient.java (original) +++ hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/DFSClient.java Thu Apr 7 00:25:19 2011 @@ -192,7 +192,7 @@ public class DFSClient implements FSCons InetSocketAddress addr = NetUtils.createSocketAddr( datanodeid.getHost() + ":" + datanodeid.getIpcPort()); if (ClientDatanodeProtocol.LOG.isDebugEnabled()) { - ClientDatanodeProtocol.LOG.info("ClientDatanodeProtocol addr=" + addr); + ClientDatanodeProtocol.LOG.debug("ClientDatanodeProtocol addr=" + addr); } UserGroupInformation ticket = UserGroupInformation .createRemoteUser(locatedBlock.getBlock().getLocalBlock().toString()); Modified: hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/DFSOutputStream.java URL: http://svn.apache.org/viewvc/hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/DFSOutputStream.java?rev=1089696&r1=1089695&r2=1089696&view=diff ============================================================================== --- hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/DFSOutputStream.java (original) +++ hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/DFSOutputStream.java Thu Apr 7 00:25:19 2011 @@ -24,8 +24,8 @@ import java.io.BufferedOutputStream; import java.io.DataInputStream; import java.io.DataOutputStream; import java.io.FileNotFoundException; -import java.io.InterruptedIOException; import java.io.IOException; +import java.io.InterruptedIOException; import java.net.InetSocketAddress; import java.net.Socket; import java.nio.BufferOverflowException; @@ -46,21 +46,23 @@ import org.apache.hadoop.fs.permission.F import org.apache.hadoop.hdfs.protocol.ClientProtocol; import org.apache.hadoop.hdfs.protocol.DSQuotaExceededException; import org.apache.hadoop.hdfs.protocol.DataTransferProtocol; +import org.apache.hadoop.hdfs.protocol.DataTransferProtocol.BlockConstructionStage; +import org.apache.hadoop.hdfs.protocol.DataTransferProtocol.PacketHeader; +import org.apache.hadoop.hdfs.protocol.DataTransferProtocol.PipelineAck; import org.apache.hadoop.hdfs.protocol.DatanodeInfo; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; import org.apache.hadoop.hdfs.protocol.HdfsFileStatus; import org.apache.hadoop.hdfs.protocol.LocatedBlock; import org.apache.hadoop.hdfs.protocol.NSQuotaExceededException; import org.apache.hadoop.hdfs.protocol.UnresolvedPathException; -import org.apache.hadoop.hdfs.protocol.DataTransferProtocol.BlockConstructionStage; -import org.apache.hadoop.hdfs.protocol.DataTransferProtocol.PacketHeader; -import org.apache.hadoop.hdfs.protocol.DataTransferProtocol.PipelineAck; +import org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier; +import org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException; import org.apache.hadoop.hdfs.server.datanode.DataNode; import org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException; import org.apache.hadoop.hdfs.server.namenode.SafeModeException; import org.apache.hadoop.io.EnumSetWritable; -import org.apache.hadoop.io.Text; import org.apache.hadoop.io.IOUtils; +import org.apache.hadoop.io.Text; import org.apache.hadoop.ipc.RemoteException; import org.apache.hadoop.net.NetUtils; import org.apache.hadoop.security.AccessControlException; @@ -70,8 +72,6 @@ import org.apache.hadoop.util.DataChecks import org.apache.hadoop.util.Progressable; import org.apache.hadoop.util.PureJavaCrc32; import org.apache.hadoop.util.StringUtils; -import org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException; -import org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier; /**************************************************************** * DFSOutputStream creates files from a stream of bytes. @@ -888,18 +888,7 @@ class DFSOutputStream extends FSOutputSu boolean result = false; try { - if(DFSClient.LOG.isDebugEnabled()) { - DFSClient.LOG.debug("Connecting to " + nodes[0].getName()); - } - InetSocketAddress target = NetUtils.createSocketAddr(nodes[0].getName()); - s = dfsClient.socketFactory.createSocket(); - int timeoutValue = dfsClient.getDatanodeReadTimeout(nodes.length); - NetUtils.connect(s, target, timeoutValue); - s.setSoTimeout(timeoutValue); - s.setSendBufferSize(DFSClient.DEFAULT_DATA_SOCKET_SIZE); - if(DFSClient.LOG.isDebugEnabled()) { - DFSClient.LOG.debug("Send buf size " + s.getSendBufferSize()); - } + s = createSocketForPipeline(nodes, dfsClient); long writeTimeout = dfsClient.getDatanodeWriteTimeout(nodes.length); // @@ -1035,6 +1024,29 @@ class DFSOutputStream extends FSOutputSu } } + /** + * Create a socket for a write pipeline + * @param datanodes the datanodes on the pipeline + * @param client + * @return the socket connected to the first datanode + */ + static Socket createSocketForPipeline(final DatanodeInfo[] datanodes, + final DFSClient client) throws IOException { + if(DFSClient.LOG.isDebugEnabled()) { + DFSClient.LOG.debug("Connecting to datanode " + datanodes[0].getName()); + } + final InetSocketAddress isa = NetUtils.createSocketAddr(datanodes[0].getName()); + final Socket sock = client.socketFactory.createSocket(); + final int timeout = client.getDatanodeReadTimeout(datanodes.length); + NetUtils.connect(sock, isa, timeout); + sock.setSoTimeout(timeout); + sock.setSendBufferSize(DFSClient.DEFAULT_DATA_SOCKET_SIZE); + if(DFSClient.LOG.isDebugEnabled()) { + DFSClient.LOG.debug("Send buf size " + sock.getSendBufferSize()); + } + return sock; + } + private void isClosed() throws IOException { if (closed) { IOException e = lastException; Modified: hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java URL: http://svn.apache.org/viewvc/hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java?rev=1089696&r1=1089695&r2=1089696&view=diff ============================================================================== --- hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java (original) +++ hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java Thu Apr 7 00:25:19 2011 @@ -153,24 +153,9 @@ class DataXceiver extends DataTransferPr datanode.socketWriteTimeout); DataOutputStream out = new DataOutputStream( new BufferedOutputStream(baseStream, SMALL_BUFFER_SIZE)); - - if (datanode.isBlockTokenEnabled) { - try { - datanode.blockPoolTokenSecretManager.checkAccess(blockToken, null, block, - BlockTokenSecretManager.AccessMode.READ); - } catch (InvalidToken e) { - try { - ERROR_ACCESS_TOKEN.write(out); - out.flush(); - LOG.warn("Block token verification failed, for client " - + remoteAddress + " for OP_READ_BLOCK for block " + block + " : " - + e.getLocalizedMessage()); - throw e; - } finally { - IOUtils.closeStream(out); - } - } - } + checkAccess(out, block, blockToken, + DataTransferProtocol.Op.READ_BLOCK, + BlockTokenSecretManager.AccessMode.READ); // send the block BlockSender blockSender = null; @@ -250,32 +235,14 @@ class DataXceiver extends DataTransferPr " src: " + remoteAddress + " dest: " + localAddress); - DataOutputStream replyOut = null; // stream to prev target - replyOut = new DataOutputStream(new BufferedOutputStream( - NetUtils.getOutputStream(s, datanode.socketWriteTimeout), - SMALL_BUFFER_SIZE)); - DatanodeRegistration dnR = - datanode.getDNRegistrationForBP(block.getBlockPoolId()); - if (datanode.isBlockTokenEnabled) { - try { - datanode.blockPoolTokenSecretManager.checkAccess(blockToken, null, block, - BlockTokenSecretManager.AccessMode.WRITE); - } catch (InvalidToken e) { - try { - if (isClient) { - ERROR_ACCESS_TOKEN.write(replyOut); - Text.writeString(replyOut, dnR.getName()); - replyOut.flush(); - } - LOG.warn("Block token verification failed, for client " - + remoteAddress + " for OP_WRITE_BLOCK for block " + block - + " : " + e.getLocalizedMessage()); - throw e; - } finally { - IOUtils.closeStream(replyOut); - } - } - } + // reply to upstream datanode or client + final DataOutputStream replyOut = new DataOutputStream( + new BufferedOutputStream( + NetUtils.getOutputStream(s, datanode.socketWriteTimeout), + SMALL_BUFFER_SIZE)); + checkAccess(isClient? replyOut: null, block, blockToken, + DataTransferProtocol.Op.WRITE_BLOCK, + BlockTokenSecretManager.AccessMode.WRITE); DataOutputStream mirrorOut = null; // stream to next target DataInputStream mirrorIn = null; // reply from next target @@ -298,8 +265,7 @@ class DataXceiver extends DataTransferPr } // - // Open network conn to backup machine, if - // appropriate + // Connect to downstream machine, if appropriate // if (targets.length > 0) { InetSocketAddress mirrorTarget = null; @@ -321,7 +287,6 @@ class DataXceiver extends DataTransferPr SMALL_BUFFER_SIZE)); mirrorIn = new DataInputStream(NetUtils.getInputStream(mirrorSock)); - // Write header: Copied from DFSClient.java! DataTransferProtocol.Sender.opWriteBlock(mirrorOut, originalBlock, pipelineSize, stage, newGs, minBytesRcvd, maxBytesRcvd, clientname, srcDataNode, targets, blockToken); @@ -358,7 +323,7 @@ class DataXceiver extends DataTransferPr if (isClient) { throw e; } else { - LOG.info(dnR + ":Exception transfering block " + + LOG.info(datanode + ":Exception transfering block " + block + " to mirror " + mirrorNode + ". continuing without the mirror.\n" + StringUtils.stringifyException(e)); @@ -430,26 +395,11 @@ class DataXceiver extends DataTransferPr @Override protected void opBlockChecksum(DataInputStream in, ExtendedBlock block, Token blockToken) throws IOException { - DataOutputStream out = new DataOutputStream(NetUtils.getOutputStream(s, - datanode.socketWriteTimeout)); - if (datanode.isBlockTokenEnabled) { - try { - datanode.blockPoolTokenSecretManager.checkAccess(blockToken, null, - block, BlockTokenSecretManager.AccessMode.READ); - } catch (InvalidToken e) { - try { - ERROR_ACCESS_TOKEN.write(out); - out.flush(); - LOG.warn("Block token verification failed, for client " - + remoteAddress + " for OP_BLOCK_CHECKSUM for block " + block - + " : " + e.getLocalizedMessage()); - throw e; - } finally { - IOUtils.closeStream(out); - } - } - } - + final DataOutputStream out = new DataOutputStream( + NetUtils.getOutputStream(s, datanode.socketWriteTimeout)); + checkAccess(out, block, blockToken, + DataTransferProtocol.Op.BLOCK_CHECKSUM, + BlockTokenSecretManager.AccessMode.READ); updateCurrentThreadName("Reading metadata for block " + block); final MetaDataInputStream metadataIn = datanode.data.getMetaDataInputStream(block); @@ -703,4 +653,34 @@ class DataXceiver extends DataTransferPr IOUtils.closeStream(reply); } } + + private void checkAccess(final DataOutputStream out, + final ExtendedBlock blk, + final Token t, + final DataTransferProtocol.Op op, + final BlockTokenSecretManager.AccessMode mode) throws IOException { + if (datanode.isBlockTokenEnabled) { + try { + datanode.blockPoolTokenSecretManager.checkAccess(t, null, blk, mode); + } catch(InvalidToken e) { + try { + if (out != null) { + ERROR_ACCESS_TOKEN.write(out); + if (mode == BlockTokenSecretManager.AccessMode.WRITE) { + DatanodeRegistration dnR = + datanode.getDNRegistrationForBP(blk.getBlockPoolId()); + Text.writeString(out, dnR.getName()); + } + out.flush(); + } + LOG.warn("Block token verification failed: op=" + op + + ", remoteAddress=" + remoteAddress + + ", message=" + e.getLocalizedMessage()); + throw e; + } finally { + IOUtils.closeStream(out); + } + } + } + } } Propchange: hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Thu Apr 7 00:25:19 2011 @@ -4,4 +4,4 @@ /hadoop/core/trunk/src/hdfs/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java:776175-785643,785929-786278 /hadoop/hdfs/branches/HDFS-265/src/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java:796829-820463 /hadoop/hdfs/branches/branch-0.21/src/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java:820487 -/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java:1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461 +/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java:1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461,1086479,1086654,1086693,1086820,1087080,1087096,1087115,1087160 Modified: hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java URL: http://svn.apache.org/viewvc/hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java?rev=1089696&r1=1089695&r2=1089696&view=diff ============================================================================== --- hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java (original) +++ hadoop/hdfs/branches/HDFS-1052/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java Thu Apr 7 00:25:19 2011 @@ -1410,7 +1410,8 @@ public class FSNamesystem implements FSC ". Lease recovery is in progress. Try again later."); } else { - if(pendingFile.getLastBlock().getBlockUCState() == + BlockInfoUnderConstruction lastBlock=pendingFile.getLastBlock(); + if(lastBlock != null && lastBlock.getBlockUCState() == BlockUCState.UNDER_RECOVERY) { throw new RecoveryInProgressException( "Recovery in progress, file [" + src + "], " + @@ -3144,6 +3145,14 @@ public class FSNamesystem implements FSC throw new IOException("ProcessReport from dead or unregisterted node: " + nodeID.getName()); } + // To minimize startup time, we discard any second (or later) block reports + // that we receive while still in startup phase. + if (isInStartupSafeMode() && node.numBlocks() > 0) { + NameNode.stateChangeLog.info("BLOCK* NameSystem.processReport: " + + "discarded non-initial block report from " + nodeID.getName() + + " because namenode still in startup phase"); + return; + } blockManager.processReport(node, newReport); NameNode.getNameNodeMetrics().blockReport.inc((int) (now() - startTime)); @@ -4221,6 +4230,15 @@ public class FSNamesystem implements FSC return false; return safeMode.isOn(); } + + /** + * Check whether the name node is in startup mode. + */ + synchronized boolean isInStartupSafeMode() { + if (safeMode == null) + return false; + return safeMode.isOn() && !safeMode.isManual(); + } /** * Check whether replication queues are populated. @@ -4594,7 +4612,8 @@ public class FSNamesystem implements FSC bean = new StandardMBean(this,FSNamesystemMBean.class); mbeanName = MBeanUtil.registerMBean("NameNode", "FSNamesystemState", bean); } catch (NotCompliantMBeanException e) { - e.printStackTrace(); + LOG.warn("Exception in initializing StandardMBean as FSNamesystemMBean", + e); } LOG.info("Registered FSNamesystemStatusMBean"); Propchange: hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Thu Apr 7 00:25:19 2011 @@ -2,4 +2,4 @@ /hadoop/core/trunk/src/test/hdfs:776175-785643 /hadoop/hdfs/branches/HDFS-265/src/test/hdfs:796829-820463 /hadoop/hdfs/branches/branch-0.21/src/test/hdfs:820487 -/hadoop/hdfs/trunk/src/test/hdfs:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461 +/hadoop/hdfs/trunk/src/test/hdfs:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461,1086479,1086654,1086693,1086820,1087080,1087096,1087115,1087160 Modified: hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml URL: http://svn.apache.org/viewvc/hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml?rev=1089696&r1=1089695&r2=1089696&view=diff ============================================================================== --- hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml (original) +++ hadoop/hdfs/branches/HDFS-1052/src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml Thu Apr 7 00:25:19 2011 @@ -5024,8 +5024,8 @@ - TokenComparator - get: null + RegexpComparator + get: .*: No such file or directory @@ -5040,8 +5040,8 @@ - TokenComparator - get: null + RegexpComparator + get: .*: No such file or directory @@ -5056,8 +5056,8 @@ - TokenComparator - get: null + RegexpComparator + get: .*: No such file or directory @@ -5072,8 +5072,8 @@ - TokenComparator - get: null + RegexpComparator + get: .*: No such file or directory @@ -5459,8 +5459,8 @@ - TokenComparator - copyToLocal: null + RegexpComparator + copyToLocal: .*: No such file or directory @@ -5475,8 +5475,8 @@ - TokenComparator - copyToLocal: null + RegexpComparator + copyToLocal: .*: No such file or directory @@ -5491,8 +5491,8 @@ - TokenComparator - copyToLocal: null + RegexpComparator + copyToLocal: .*: No such file or directory @@ -5507,8 +5507,8 @@ - TokenComparator - copyToLocal: null + RegexpComparator + copyToLocal: .*: No such file or directory @@ -15593,11 +15593,15 @@ RegexpComparator - ^-touchz <path>: Write a timestamp in yyyy-MM-dd HH:mm:ss format( )* + ^-touchz <path>: Creates a file of zero length( )* + + + RegexpComparator + ^( |\t)*at <path> with current time as the timestamp of that <path>.( )* RegexpComparator - ^( |\t)*in a file at <path>. An error is returned if the file exists with non-zero length( )* + ^( |\t)* An error is returned if the file exists with non-zero length( )* @@ -16927,7 +16931,7 @@ -fs NAMENODE -mkdir /test -fs NAMENODE -touchz /test/file1 -fs NAMENODE -safemode enter - -fs NAMENODE -chown root /test/file1 + -fs NAMENODE -chown newgroup /test/file1 -fs NAMENODE -safemode leave Propchange: hadoop/hdfs/branches/HDFS-1052/src/webapps/datanode/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Thu Apr 7 00:25:19 2011 @@ -2,4 +2,4 @@ /hadoop/core/trunk/src/webapps/datanode:776175-784663 /hadoop/hdfs/branches/HDFS-265/src/webapps/datanode:796829-820463 /hadoop/hdfs/branches/branch-0.21/src/webapps/datanode:820487 -/hadoop/hdfs/trunk/src/webapps/datanode:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461 +/hadoop/hdfs/trunk/src/webapps/datanode:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461,1086479,1086654,1086693,1086820,1087080,1087096,1087115,1087160 Propchange: hadoop/hdfs/branches/HDFS-1052/src/webapps/hdfs/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Thu Apr 7 00:25:19 2011 @@ -2,4 +2,4 @@ /hadoop/core/trunk/src/webapps/hdfs:776175-784663 /hadoop/hdfs/branches/HDFS-265/src/webapps/hdfs:796829-820463 /hadoop/hdfs/branches/branch-0.21/src/webapps/hdfs:820487 -/hadoop/hdfs/trunk/src/webapps/hdfs:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461 +/hadoop/hdfs/trunk/src/webapps/hdfs:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461,1086479,1086654,1086693,1086820,1087080,1087096,1087115,1087160 Propchange: hadoop/hdfs/branches/HDFS-1052/src/webapps/secondary/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Thu Apr 7 00:25:19 2011 @@ -2,4 +2,4 @@ /hadoop/core/trunk/src/webapps/secondary:776175-784663 /hadoop/hdfs/branches/HDFS-265/src/webapps/secondary:796829-820463 /hadoop/hdfs/branches/branch-0.21/src/webapps/secondary:820487 -/hadoop/hdfs/trunk/src/webapps/secondary:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461 +/hadoop/hdfs/trunk/src/webapps/secondary:987665-1004788,1026178-1028906,1032470-1033639,1034073,1034082-1034181,1034501-1034544,1034932,1035141,1035143,1035145,1035163,1035386,1035410,1035508,1035515,1035552,1035718,1035795,1035841-1035842,1035890,1035920,1035924,1036132,1036213,1036303,1036310,1036631,1036692,1036767,1038222-1038859,1039957,1040005,1040027-1060619,1061067,1062011-1065960,1066305,1066970,1066986,1067079-1076024,1076696,1078925,1078940,1079069-1080836,1081580-1083021,1083043,1083084,1083234,1083902,1083951,1083958,1085460,1085509,1086454-1086461,1086479,1086654,1086693,1086820,1087080,1087096,1087115,1087160