Return-Path: X-Original-To: apmail-hadoop-common-commits-archive@www.apache.org Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C7A4E9E2D for ; Tue, 13 Mar 2012 20:40:42 +0000 (UTC) Received: (qmail 72925 invoked by uid 500); 13 Mar 2012 20:40:42 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 72864 invoked by uid 500); 13 Mar 2012 20:40:42 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 72854 invoked by uid 99); 13 Mar 2012 20:40:42 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Mar 2012 20:40:42 +0000 X-ASF-Spam-Status: No, hits=-1998.0 required=5.0 tests=ALL_TRUSTED,FB_GET_MEDS X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Mar 2012 20:40:38 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id D2EDE2388A56 for ; Tue, 13 Mar 2012 20:40:16 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1300347 - in /hadoop/common/branches/branch-1.0: ./ src/hdfs/org/apache/hadoop/hdfs/server/namenode/ src/test/org/apache/hadoop/hdfs/server/namenode/ Date: Tue, 13 Mar 2012 20:40:16 -0000 To: common-commits@hadoop.apache.org From: szetszwo@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20120313204016.D2EDE2388A56@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: szetszwo Date: Tue Mar 13 20:40:16 2012 New Revision: 1300347 URL: http://svn.apache.org/viewvc?rev=1300347&view=rev Log: svn merge -c 1293716 from branch-1 for HDFS-2978. Modified: hadoop/common/branches/branch-1.0/ (props changed) hadoop/common/branches/branch-1.0/CHANGES.txt (contents, props changed) hadoop/common/branches/branch-1.0/src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop/common/branches/branch-1.0/src/hdfs/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java hadoop/common/branches/branch-1.0/src/test/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java Propchange: hadoop/common/branches/branch-1.0/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 13 20:40:16 2012 @@ -3,6 +3,6 @@ /hadoop/common/branches/branch-0.20-security:1170042,1170087,1170997,1171137,1171380,1171613,1171891,1171905,1172184,1172188,1172190,1172192,1173470,1174471,1175114,1176179,1176720,1177907,1179036,1179171,1179519,1179857,1183086,1183101,1183176,1183556,1186509,1187141,1189029,1189361,1190079,1190089,1190100,1195737,1197330,1198906,1199401,1199571,1200733,1205148,1205655,1206180 /hadoop/common/branches/branch-0.20-security-203:1096071,1097011,1097249,1097269,1097281,1097966,1098816,1098819,1098823,1098827,1098832,1098839,1098854,1098863,1099088,1099191,1099324,1099330,1099333,1102071,1128115 /hadoop/common/branches/branch-0.20-security-204:1128390,1147228,1148069,1149316,1154413 -/hadoop/common/branches/branch-1:1207579,1208143,1235107,1293012,1293490 +/hadoop/common/branches/branch-1:1207579,1208143,1235107,1293012,1293490,1293716 /hadoop/core/branches/branch-0.19:713112 /hadoop/core/trunk:727001,727117,727191,727212,727217,727228,727255,727869,728187,729052,729987,732385,732572,732613,732777,732838,732869,733887,734870,734916,736426,738328,738697,740077,740157,741703,741762,743745,743816,743892,744894,745180,746010,746206,746227,746233,746274,746338,746902-746903,746925,746944,746968,746970,747279,747289,747802,748084,748090,748783,749262,749318,749863,750533,752073,752609,752834,752836,752913,752932,753112-753113,753346,754645,754847,754927,755035,755226,755348,755370,755418,755426,755790,755905,755938,755960,755986,755998,756352,757448,757624,757849,758156,758180,759398,759932,760502,760783,761046,761482,761632,762216,762879,763107,763502,764967,765016,765809,765951,771607,771661,772844,772876,772884,772920,773889,776638,778962,778966,779893,781720,784661,785046,785569 Modified: hadoop/common/branches/branch-1.0/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-1.0/CHANGES.txt?rev=1300347&r1=1300346&r2=1300347&view=diff ============================================================================== --- hadoop/common/branches/branch-1.0/CHANGES.txt (original) +++ hadoop/common/branches/branch-1.0/CHANGES.txt Tue Mar 13 20:40:16 2012 @@ -7,6 +7,8 @@ Release 1.0.2 - unreleased HADOOP-7206. Support Snappy compression. (Issei Yoshida and Alejandro Abdelnur via vinodkv). + HDFS-2978. The NameNode should expose name dir statuses via JMX. (atm) + IMPROVEMENTS MAPREDUCE-3773. Add queue metrics with buckets for job run times. (omalley Propchange: hadoop/common/branches/branch-1.0/CHANGES.txt ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 13 20:40:16 2012 @@ -3,7 +3,7 @@ /hadoop/common/branches/branch-0.20-security/CHANGES.txt:1170042,1170087,1170997,1171137,1171181,1171380,1171613,1171891,1171905,1172184,1172188,1172190,1172192,1173470,1173843,1174326,1174471,1174476,1174482,1175114,1176179,1176182,1176270,1176276,1176675,1176720,1177031,1177036,1177098,1177101,1177907,1178074,1179036,1179171,1179471,1179519,1179713,1179722,1179857,1179919,1183086,1183101,1183176,1183556,1186509,1187141,1189029,1189361,1190079,1190089,1190100,1195737,1197330,1198906,1199401,1199571,1200733,1205148,1205655,1206180 /hadoop/common/branches/branch-0.20-security-203/CHANGES.txt:1096071,1097011,1097249,1097269,1097281,1097966,1098816,1098819,1098823,1098827,1098832,1098839,1098854,1098863,1099088,1099191,1099324,1099330,1099333,1102071,1128115 /hadoop/common/branches/branch-0.20-security-204/CHANGES.txt:1128390,1147228,1148069,1149316,1152887,1154413,1159730,1161741 -/hadoop/common/branches/branch-1/CHANGES.txt:1207579,1208143,1208936,1235107,1239393,1293012,1293490 +/hadoop/common/branches/branch-1/CHANGES.txt:1207579,1208143,1208936,1235107,1239393,1293012,1293490,1293716 /hadoop/core/branches/branch-0.18/CHANGES.txt:727226 /hadoop/core/branches/branch-0.19/CHANGES.txt:713112 /hadoop/core/trunk/CHANGES.txt:727001,727117,727191,727212,727228,727255,727869,728187,729052,729987,732385,732572,732613,732777,732838,732869,733887,734870,734916,735082,736426,738602,738697,739416,740077,740157,741703,741762,743296,743745,743816,743892,744894,745180,745268,746010,746193,746206,746227,746233,746274,746902-746903,746925,746944,746968,746970,747279,747289,747802,748084,748090,748783,749262,749318,749863,750533,752073,752514,752555,752590,752609,752834,752836,752913,752932,753112-753113,753346,754645,754847,754927,755035,755226,755348,755370,755418,755426,755790,755905,755938,755986,755998,756352,757448,757624,757849,758156,758180,759398,759932,760502,760783,761046,761482,761632,762216,762879,763107,763502,764967,765016,765809,765951,771607,772844,772876,772884,772920,773889,776638,778962,778966,779893,781720,784661,785046,785569 Modified: hadoop/common/branches/branch-1.0/src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-1.0/src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java?rev=1300347&r1=1300346&r2=1300347&view=diff ============================================================================== --- hadoop/common/branches/branch-1.0/src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java (original) +++ hadoop/common/branches/branch-1.0/src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java Tue Mar 13 20:40:16 2012 @@ -83,6 +83,8 @@ import org.apache.hadoop.hdfs.security.t import org.apache.hadoop.hdfs.server.common.GenerationStamp; import org.apache.hadoop.hdfs.server.common.HdfsConstants; import org.apache.hadoop.hdfs.server.common.HdfsConstants.StartupOption; +import org.apache.hadoop.hdfs.server.common.Storage.StorageDirType; +import org.apache.hadoop.hdfs.server.common.Storage.StorageDirectory; import org.apache.hadoop.hdfs.server.common.Storage; import org.apache.hadoop.hdfs.server.common.UpgradeStatusReport; import org.apache.hadoop.hdfs.server.namenode.BlocksMap.BlockInfo; @@ -5794,6 +5796,30 @@ public class FSNamesystem implements FSC return JSON.toString(info); } + @Override // NameNodeMXBean + public String getNameDirStatuses() { + Map> statusMap = + new HashMap>(); + + Map activeDirs = new HashMap(); + for (Iterator it + = getFSImage().dirIterator(); it.hasNext();) { + StorageDirectory st = it.next(); + activeDirs.put(st.getRoot(), st.getStorageDirType()); + } + statusMap.put("active", activeDirs); + + List removedStorageDirs + = getFSImage().getRemovedStorageDirs(); + Map failedDirs = new HashMap(); + for (StorageDirectory st : removedStorageDirs) { + failedDirs.put(st.getRoot(), st.getStorageDirType()); + } + statusMap.put("failed", failedDirs); + + return JSON.toString(statusMap); + } + private long getLastContact(DatanodeDescriptor alivenode) { return (System.currentTimeMillis() - alivenode.getLastUpdate())/1000; } Modified: hadoop/common/branches/branch-1.0/src/hdfs/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-1.0/src/hdfs/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java?rev=1300347&r1=1300346&r2=1300347&view=diff ============================================================================== --- hadoop/common/branches/branch-1.0/src/hdfs/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java (original) +++ hadoop/common/branches/branch-1.0/src/hdfs/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java Tue Mar 13 20:40:16 2012 @@ -135,4 +135,12 @@ public interface NameNodeMXBean { * @return the decommissioning node information */ public String getDecomNodes(); + + /** + * Get status information about the directories storing image and edits logs + * of the NN. + * + * @return the name dir status information, as a JSON string. + */ + public String getNameDirStatuses(); } Modified: hadoop/common/branches/branch-1.0/src/test/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-1.0/src/test/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java?rev=1300347&r1=1300346&r2=1300347&view=diff ============================================================================== --- hadoop/common/branches/branch-1.0/src/test/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java (original) +++ hadoop/common/branches/branch-1.0/src/test/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java Tue Mar 13 20:40:16 2012 @@ -17,23 +17,32 @@ */ package org.apache.hadoop.hdfs.server.namenode; +import static org.junit.Assert.*; + +import java.io.File; import java.lang.management.ManagementFactory; +import java.util.Collection; +import java.util.Map; import javax.management.MBeanServer; import javax.management.ObjectName; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.hdfs.MiniDFSCluster; import org.apache.hadoop.hdfs.server.namenode.FSNamesystem; import org.apache.hadoop.util.VersionInfo; import org.junit.Test; +import org.mortbay.util.ajax.JSON; + import junit.framework.Assert; /** * Class for testing {@link NameNodeMXBean} implementation */ public class TestNameNodeMXBean { + @SuppressWarnings({ "unchecked", "deprecation" }) @Test public void testNameNodeMXBeanInfo() throws Exception { Configuration conf = new Configuration(); @@ -84,8 +93,44 @@ public class TestNameNodeMXBean { String deadnodeinfo = (String) (mbs.getAttribute(mxbeanName, "DeadNodes")); Assert.assertEquals(fsn.getDeadNodes(), deadnodeinfo); + // get attribute NameDirStatuses + String nameDirStatuses = (String) (mbs.getAttribute(mxbeanName, + "NameDirStatuses")); + Assert.assertEquals(fsn.getNameDirStatuses(), nameDirStatuses); + Map> statusMap = + (Map>) JSON.parse(nameDirStatuses); + Collection nameDirs = cluster.getNameDirs(); + for (File nameDir : nameDirs) { + System.out.println("Checking for the presence of " + nameDir + + " in active name dirs."); + assertTrue(statusMap.get("active").containsKey(nameDir.getAbsolutePath())); + } + assertEquals(2, statusMap.get("active").size()); + assertEquals(0, statusMap.get("failed").size()); + + // This will cause the first dir to fail. + File failedNameDir = nameDirs.toArray(new File[0])[0]; + assertEquals(0, FileUtil.chmod(failedNameDir.getAbsolutePath(), "000")); + cluster.getNameNode().rollEditLog(); + + nameDirStatuses = (String) (mbs.getAttribute(mxbeanName, + "NameDirStatuses")); + statusMap = (Map>) JSON.parse(nameDirStatuses); + for (File nameDir : nameDirs) { + String expectedStatus = + nameDir.equals(failedNameDir) ? "failed" : "active"; + System.out.println("Checking for the presence of " + nameDir + + " in " + expectedStatus + " name dirs."); + assertTrue(statusMap.get(expectedStatus).containsKey( + nameDir.getAbsolutePath())); + } + assertEquals(1, statusMap.get("active").size()); + assertEquals(1, statusMap.get("failed").size()); } finally { if (cluster != null) { + for (File dir : cluster.getNameDirs()) { + FileUtil.chmod(dir.toString(), "700"); + } cluster.shutdown(); } }