Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 28 Nov 2012 17:54:58 +0000 (UTC)
From: "Elliott Clark (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <99943577.34019.1354125298598.JavaMail.jiratomcat@arcas>
In-Reply-To: 
 <1018127674.43259.1320103412439.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HBASE-4709) Hadoop metrics2 setup in test
 MiniDFSClusters spewing JMX errors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-4709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505709#comment-13505709 ] 

Elliott Clark commented on HBASE-4709:
--------------------------------------

So I had two thoughts.

# I'm not 100% sure that this kind of work around isn't needed in production.  Right now lots of messages are logged by the metrics system that are really scary to users but not really a bad thing (Specifically when starting/stopping the metrics system there are tons of errors thrown).  So we could want this logging change for more than just one logger.
# If we decide that we shouldn't silence things in a real server there are already test utilities that reach into Hadoop's guts that are in hbase-hadoop1-compat and hbase-hadoop2-compat.  Seems like we should try and make the main jar have very few places where Hadoop internals are touched just in case different versions diverge more in the future.
                
> Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
> ----------------------------------------------------------------
>
>                 Key: HBASE-4709
>                 URL: https://issues.apache.org/jira/browse/HBASE-4709
>             Project: HBase
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.92.0, 0.94.0, 0.94.1
>            Reporter: Gary Helmling
>            Priority: Minor
>             Fix For: 0.96.0
>
>         Attachments: 4709_workaround.v1.patch
>
>
> Since switching over HBase to build with Hadoop 0.20.205.0, we've been getting a lot of metrics related errors in the log files for tests:
> {noformat}
> 2011-10-30 22:00:22,858 INFO  [main] log.Slf4jLog(67): jetty-6.1.26
> 2011-10-30 22:00:22,871 INFO  [main] log.Slf4jLog(67): Extract jar:file:/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-core/0.20.205.0/hadoop-core-0.20.205.0.jar!/webapps/datanode to /tmp/Jetty_localhost_55751_datanode____.kw16hy/webapp
> 2011-10-30 22:00:23,048 INFO  [main] log.Slf4jLog(67): Started SelectChannelConnector@localhost:55751
> Starting DataNode 1 with dfs.data.dir: /home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data3,/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data4
> 2011-10-30 22:00:23,237 WARN  [main] impl.MetricsSystemImpl(137): Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
> 2011-10-30 22:00:23,237 WARN  [main] util.MBeans(59): Hadoop:service=DataNode,name=MetricsSystem,sub=Control
> javax.management.InstanceAlreadyExistsException: MXBean already registered with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
> 	at com.sun.jmx.mbeanserver.MXBeanLookup.addReference(MXBeanLookup.java:120)
> 	at com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:143)
> 	at com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183)
> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941)
> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
> 	at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:56)
> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.initSystemMBean(MetricsSystemImpl.java:500)
> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:140)
> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
> 	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1483)
> 	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1459)
> 	at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
> 	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:349)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:518)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:474)
> 	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:461)
> {noformat}
> This seems to be due to errors initializing the new hadoop metrics2 code by default, when running in a mini cluster.  The errors themselves seem to be harmless -- they're not breaking any tests -- but we should figure out what configuration we need to eliminate them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira