hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lewis John McGibbney (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4709) Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
Date Thu, 05 Apr 2012 21:10:25 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247674#comment-13247674
] 

Lewis John McGibbney commented on HBASE-4709:
---------------------------------------------

An example of our typical stack trace.

{code}
3287 [main] INFO org.mortbay.log - jetty-6.1.26
3333 [main] INFO org.mortbay.log - Extract jar:file:/export/home/hudson/hudson-slave/maven-repositories/0/org/apache/hadoop/hadoop-core/1.0.1/hadoop-core-1.0.1.jar!/webapps/hdfs
to /var/tmp/Jetty_localhost_54201_hdfs____txys7y/webapp
3598 [main] INFO org.mortbay.log - Started SelectChannelConnector@localhost:54201
12/04/05 16:56:06 INFO namenode.NameNode: Web-server up at: localhost:54201
12/04/05 16:56:06 INFO ipc.Server: IPC Server Responder: starting
12/04/05 16:56:06 INFO ipc.Server: IPC Server listener on 54200: starting
12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 0 on 54200: starting
12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 3 on 54200: starting
12/04/05 16:56:07 INFO ipc.Server: IPC Server handler 7 on 54200: starting
12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 2 on 54200: starting
12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 1 on 54200: starting
12/04/05 16:56:07 INFO ipc.Server: IPC Server handler 9 on 54200: starting
12/04/05 16:56:07 INFO ipc.Server: IPC Server handler 8 on 54200: starting
12/04/05 16:56:07 INFO ipc.Server: IPC Server handler 6 on 54200: starting
12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 4 on 54200: starting
12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 5 on 54200: starting
Starting DataNode 0 with dfs.data.dir: /export/home/hudson/hudson-slave/workspace/gora-trunk/trunk./gora-core/target/test-data/dfs/data/data1,/export/home/hudson/hudson-slave/workspace/gora-trunk/trunk./gora-core/target/test-data/dfs/data/data2
12/04/05 16:56:07 WARN impl.MetricsSystemImpl: Metrics system not started: Cannot locate configuration:
tried hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
12/04/05 16:56:07 WARN util.MBeans: Hadoop:service=DataNode,name=MetricsSystem,sub=Control
javax.management.InstanceAlreadyExistsException: MXBean already registered with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
	at com.sun.jmx.mbeanserver.MXBeanLookup.addReference(MXBeanLookup.java:120)
	at com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:143)
	at com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
	at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:56)
	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.initSystemMBean(MetricsSystemImpl.java:500)
	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:140)
	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1520)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1496)
	at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124)
	at org.apache.hadoop.mapred.HadoopTestCase.setUp(HadoopTestCase.java:148)
	at org.apache.gora.mapreduce.DataStoreMapReduceTestBase.setUp(DataStoreMapReduceTestBase.java:61)
	at junit.framework.TestCase.runBare(TestCase.java:132)
	at junit.framework.TestResult$1.protect(TestResult.java:110)
	at junit.framework.TestResult.runProtected(TestResult.java:128)
	at junit.framework.TestResult.run(TestResult.java:113)
	at junit.framework.TestCase.run(TestCase.java:124)
	at junit.framework.TestSuite.runTest(TestSuite.java:243)
	at junit.framework.TestSuite.run(TestSuite.java:238)
	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
{code}

You will however notice a difference between the original and my trace, where this particular
one is not using HBaseTestingUtility, however instead relying on HadoopTestCase which in turn
invokes MiniDFSCluster... the problem to that end is identical. I must state that there is
ONE previous 'metrics system not starting' warning identical to the one you see above. E.g.
two in all the 2nd being followed by the full stack trace.
                
> Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
> ----------------------------------------------------------------
>
>                 Key: HBASE-4709
>                 URL: https://issues.apache.org/jira/browse/HBASE-4709
>             Project: HBase
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.92.0, 0.94.0
>            Reporter: Gary Helmling
>            Priority: Minor
>
> Since switching over HBase to build with Hadoop 0.20.205.0, we've been getting a lot
of metrics related errors in the log files for tests:
> {noformat}
> 2011-10-30 22:00:22,858 INFO  [main] log.Slf4jLog(67): jetty-6.1.26
> 2011-10-30 22:00:22,871 INFO  [main] log.Slf4jLog(67): Extract jar:file:/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-core/0.20.205.0/hadoop-core-0.20.205.0.jar!/webapps/datanode
to /tmp/Jetty_localhost_55751_datanode____.kw16hy/webapp
> 2011-10-30 22:00:23,048 INFO  [main] log.Slf4jLog(67): Started SelectChannelConnector@localhost:55751
> Starting DataNode 1 with dfs.data.dir: /home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data3,/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data4
> 2011-10-30 22:00:23,237 WARN  [main] impl.MetricsSystemImpl(137): Metrics system not
started: Cannot locate configuration: tried hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
> 2011-10-30 22:00:23,237 WARN  [main] util.MBeans(59): Hadoop:service=DataNode,name=MetricsSystem,sub=Control
> javax.management.InstanceAlreadyExistsException: MXBean already registered with name
Hadoop:service=NameNode,name=MetricsSystem,sub=Control
> 	at com.sun.jmx.mbeanserver.MXBeanLookup.addReference(MXBeanLookup.java:120)
> 	at com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:143)
> 	at com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183)
> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941)
> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
> 	at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:56)
> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.initSystemMBean(MetricsSystemImpl.java:500)
> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:140)
> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
> 	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1483)
> 	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1459)
> 	at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
> 	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:349)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:518)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:474)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:461)
> {noformat}
> This seems to be due to errors initializing the new hadoop metrics2 code by default,
when running in a mini cluster.  The errors themselves seem to be harmless -- they're not
breaking any tests -- but we should figure out what configuration we need to eliminate them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message