Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 1DEA0200D1A for ; Mon, 25 Sep 2017 04:18:11 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 1C6741609E8; Mon, 25 Sep 2017 02:18:11 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 6BE131609E6 for ; Mon, 25 Sep 2017 04:18:09 +0200 (CEST) Received: (qmail 36704 invoked by uid 500); 25 Sep 2017 02:18:08 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 36693 invoked by uid 99); 25 Sep 2017 02:18:08 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Sep 2017 02:18:08 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id F2F791A5DC9 for ; Mon, 25 Sep 2017 02:18:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id G47pw3dCVNWX for ; Mon, 25 Sep 2017 02:18:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 8F3335FDE2 for ; Mon, 25 Sep 2017 02:18:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 09CFAE0041 for ; Mon, 25 Sep 2017 02:18:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id ABA8524211 for ; Mon, 25 Sep 2017 02:18:00 +0000 (UTC) Date: Mon, 25 Sep 2017 02:18:00 +0000 (UTC) From: "wuchengzhi (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HBASE-18868) metrics error and the jmx loss many metrics like 'Hadoop:service=HBase,name=RegionServer,sub=IPC' MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 25 Sep 2017 02:18:11 -0000 [ https://issues.apache.org/jira/browse/HBASE-18868?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:all-tabpanel ] wuchengzhi updated HBASE-18868: ------------------------------- Description:=20 so many metrics error log printed in the regionserver log all the time, the= n the web-gui http://hbase-master1:16010/jmx loss some metrics like=20 'Hadoop:service=3DHBase,name=3DRegionServer,sub=3DRegions' , 'Hadoop:servi= ce=3DHBase,name=3DRegionServer,sub=3DIPC' .......=20 is there something wrong i configured at my hbase-site.xml or hbase-env.sh= ? regionserver logs: 2017-09-21 05:03:42,088 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: Sc= heduled snapshot period at 10 second(s). 2017-09-21 05:03:42,088 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: HB= ase metrics system started 2017-09-21 05:03:42,088 WARN [HBase-Metrics2-1] util.MBeans: Error creatin= g MBean object name: Hadoop:service=3DHBase,name=3DJvmMetrics org.apache.hadoop.metrics2.MetricsException: org.apache.hadoop.metrics2.Met= ricsException: Hadoop:service=3DHBase,name=3DJvmMetrics already exists! =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(Def= aultMetricsSystem.java:117) =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(Defa= ultMetricsSystem.java:102) =09at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:91) =09at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:55) =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(Metr= icsSourceAdapter.java:220) =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSou= rceAdapter.java:95) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(Metr= icsSystemImpl.java:244) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(Metrics= SystemImpl.java:228) =09at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:43) =09at java.lang.reflect.Method.invoke(Method.java:606) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSys= temImpl.java:289) =09at com.sun.proxy.$Proxy11.postStart(Unknown Source) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystem= Impl.java:184) =09at org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnable= .run(JmxCacheBuster.java:78) =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47= 1) =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.= access$201(ScheduledThreadPoolExecutor.java:178) =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.= run(ScheduledThreadPoolExecutor.java:292) =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1145) =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:615) =09at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.metrics2.MetricsException: Hadoop:service=3DHB= ase,name=3DJvmMetrics already exists! =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(Def= aultMetricsSystem.java:113) =09... 21 more 2017-09-21 05:03:42,090 WARN [HBase-Metrics2-1] util.MBeans: Failed to reg= ister MBean "null" javax.management.RuntimeOperationsException: Exception occurred trying to r= egister the MBean =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamic= MBean(DefaultMBeanServerInterceptor.java:951) =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(= DefaultMBeanServerInterceptor.java:900) =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(D= efaultMBeanServerInterceptor.java:324) =09at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.j= ava:522) =09at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57) =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(Metr= icsSourceAdapter.java:220) =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSou= rceAdapter.java:95) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(Metr= icsSystemImpl.java:244) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(Metrics= SystemImpl.java:228) =09at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:43) =09at java.lang.reflect.Method.invoke(Method.java:606) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSys= temImpl.java:289) =09at com.sun.proxy.$Proxy11.postStart(Unknown Source) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystem= Impl.java:184) =09at org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnable= .run(JmxCacheBuster.java:78) =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47= 1) =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.= access$201(ScheduledThreadPoolExecutor.java:178) =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.= run(ScheduledThreadPoolExecutor.java:292) =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1145) =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:615) =09at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalArgumentException: No object name specified =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamic= MBean(DefaultMBeanServerInterceptor.java:949) =09... 22 more 2017-09-21 05:03:42,090 WARN [HBase-Metrics2-1] util.MBeans: Error creatin= g MBean object name: Hadoop:service=3DHBase,name=3DRegionServer,sub=3DIPC org.apache.hadoop.metrics2.MetricsException: org.apache.hadoop.metrics2.Met= ricsException: Hadoop:service=3DHBase,name=3DRegionServer,sub=3DIPC already= exists! =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(Def= aultMetricsSystem.java:117) =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(Defa= ultMetricsSystem.java:102) =09at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:91) =09at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:55) =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(Metr= icsSourceAdapter.java:220) =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSou= rceAdapter.java:95) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(Metr= icsSystemImpl.java:244) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(Metrics= SystemImpl.java:228) =09at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:43) =09at java.lang.reflect.Method.invoke(Method.java:606) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSys= temImpl.java:289) =09at com.sun.proxy.$Proxy11.postStart(Unknown Source) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystem= Impl.java:184) =09at org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnable= .run(JmxCacheBuster.java:78) =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47= 1) =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.= access$201(ScheduledThreadPoolExecutor.java:178) =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.= run(ScheduledThreadPoolExecutor.java:292) =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1145) =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:615) =09at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.metrics2.MetricsException: Hadoop:service=3DHB= ase,name=3DRegionServer,sub=3DIPC already exists! =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(Def= aultMetricsSystem.java:113) =09... 21 more 2017-09-21 05:03:42,092 WARN [HBase-Metrics2-1] util.MBeans: Failed to reg= ister MBean "null" javax.management.RuntimeOperationsException: Exception occurred trying to r= egister the MBean =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamic= MBean(DefaultMBeanServerInterceptor.java:951) =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(= DefaultMBeanServerInterceptor.java:900) =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(D= efaultMBeanServerInterceptor.java:324) =09at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.j= ava:522) =09at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57) =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(Metr= icsSourceAdapter.java:220) =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSou= rceAdapter.java:95) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(Metr= icsSystemImpl.java:244) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(Metrics= SystemImpl.java:228) =09at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:43) =09at java.lang.reflect.Method.invoke(Method.java:606) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSys= temImpl.java:289) =09at com.sun.proxy.$Proxy11.postStart(Unknown Source) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystem= Impl.java:184) =09at org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnable= .run(JmxCacheBuster.java:78) =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47= 1) =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.= access$201(ScheduledThreadPoolExecutor.java:178) =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.= run(ScheduledThreadPoolExecutor.java:292) =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1145) =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:615) =09at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalArgumentException: No object name specified =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamic= MBean(DefaultMBeanServerInterceptor.java:949) =09... 22 more 2017-09-21 05:03:42,092 WARN [HBase-Metrics2-1] util.MBeans: Error creatin= g MBean object name: Hadoop:service=3DHBase,name=3DUgiMetrics org.apache.hadoop.metrics2.MetricsException: org.apache.hadoop.metrics2.Met= ricsException: Hadoop:service=3DHBase,name=3DUgiMetrics already exists! =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(Def= aultMetricsSystem.java:117) =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(Defa= ultMetricsSystem.java:102) =09at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:91) =09at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:55) =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(Metr= icsSourceAdapter.java:220) =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSou= rceAdapter.java:95) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(Metr= icsSystemImpl.java:244) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(Metrics= SystemImpl.java:228) =09at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:43) =09at java.lang.reflect.Method.invoke(Method.java:606) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSys= temImpl.java:289) =09at com.sun.proxy.$Proxy11.postStart(Unknown Source) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystem= Impl.java:184) =09at org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnable= .run(JmxCacheBuster.java:78) =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47= 1) =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.= access$201(ScheduledThreadPoolExecutor.java:178) =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.= run(ScheduledThreadPoolExecutor.java:292) =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1145) =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:615) =09at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.metrics2.MetricsException: Hadoop:service=3DHB= ase,name=3DUgiMetrics already exists! =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(Def= aultMetricsSystem.java:113) =09... 21 more 2017-09-21 05:03:42,093 WARN [HBase-Metrics2-1] util.MBeans: Failed to reg= ister MBean "null" javax.management.RuntimeOperationsException: Exception occurred trying to r= egister the MBean =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamic= MBean(DefaultMBeanServerInterceptor.java:951) =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(= DefaultMBeanServerInterceptor.java:900) =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(D= efaultMBeanServerInterceptor.java:324) =09at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.j= ava:522) =09at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57) =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(Metr= icsSourceAdapter.java:220) =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSou= rceAdapter.java:95) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(Metr= icsSystemImpl.java:244) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(Metrics= SystemImpl.java:228) =09at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:43) =09at java.lang.reflect.Method.invoke(Method.java:606) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSys= temImpl.java:289) =09at com.sun.proxy.$Proxy11.postStart(Unknown Source) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystem= Impl.java:184) =09at org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnable= .run(JmxCacheBuster.java:78) =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47= 1) =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.= access$201(ScheduledThreadPoolExecutor.java:178) =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.= run(ScheduledThreadPoolExecutor.java:292) =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1145) =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:615) =09at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalArgumentException: No object name specified =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamic= MBean(DefaultMBeanServerInterceptor.java:949) =09... 22 more 2017-09-21 05:03:42,093 WARN [HBase-Metrics2-1] util.MBeans: Error creatin= g MBean object name: Hadoop:service=3DHBase,name=3DRegionServer,sub=3DWAL org.apache.hadoop.metrics2.MetricsException: org.apache.hadoop.metrics2.Met= ricsException: Hadoop:service=3DHBase,name=3DRegionServer,sub=3DWAL already= exists! =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(Def= aultMetricsSystem.java:117) =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(Defa= ultMetricsSystem.java:102) =09at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:91) =09at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:55) =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(Metr= icsSourceAdapter.java:220) =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSou= rceAdapter.java:95) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(Metr= icsSystemImpl.java:244) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(Metrics= SystemImpl.java:228) =09at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:43) =09at java.lang.reflect.Method.invoke(Method.java:606) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSys= temImpl.java:289) =09at com.sun.proxy.$Proxy11.postStart(Unknown Source) =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystem= Impl.java:184) =09at org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnable= .run(JmxCacheBuster.java:78) =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47= 1) =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.= access$201(ScheduledThreadPoolExecutor.java:178) =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.= run(ScheduledThreadPoolExecutor.java:292) =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1145) =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:615) =09at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.metrics2.MetricsException: Hadoop:service=3DHB= ase,name=3DRegionServer,sub=3DWAL already exists! =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(Def= aultMetricsSystem.java:113) =09... 21 more was: so many metrics error log printed in the regionserver log all the time, the= n the web-gui http://hbase-master1:16010/jmx loss some metrics like=20 'Hadoop:service=3DHBase,name=3DRegionServer,sub=3DRegions' , 'Hadoop:servi= ce=3DHBase,name=3DRegionServer,sub=3DIPC' .......=20 is there something wrong i configured at my hbase-site.xml or hbase-env.sh= ? ver.thread.wakefrequency =09=091000 =09 =09 =09=09hbase.rpc.timeout =09=0954000000 =09 =09 =09=09hbase.hstore.blockingStoreFiles =09=09500 =09 =09 =09=09hbase.client.scanner.timeout.period =09=09600000 =09 =09 =09=09zookeeper.session.timeout =09=09180000 =09 =09hbase.regionserver.optionallogflushinterval =09=095000 =09 =09=09hbase.client.write.buffer =09=095242880 =09 =09 =09=09hbase.hstore.compactionThreshold =09=095 =09 =09 hbase.hstore.compaction.max 12 =09hbase.regionserver.regionSplitLimit =091 =09hbase.regionserver.thread.compaction.large =093 =09hbase.regionserver.thread.compaction.small =098 =09 =09=09hbase.master.logcleaner.ttl =09=093600000 =09 hbase.bucketcache.ioengine =09 offheap> hbase.bucketcache.percentage.in.combinedcache 0.9 hbase.bucketcache.size 26214 hbase.master.maxclockskew 150000 =09 =09 =09=09 hbase.regionserver.port =09=09 16021 =09 =09 =09 =09 hbase.regionserver.info.port=20 =09 =09 16011 =09 =20 =20 =09 =09 =09 =09 =09hbase.replication =09 =09true =09 =09 =09 =09replication.source.nb.capacity =09 =096000 =09 =09 =09 =09replication.source.size.capacity =09 =0967108864 =09 =09 =09 =09replication.source.ratio =09 =091 =09 =09 =09 =09hbase.regionserver.wal.enablecompression =09 =09false =09 =09 =09 =09 replication.sleep.before.failover =09 =095000 =09 =09 =09 =09replication.executor.workers =09 =095 =09 =09 > metrics error and the jmx loss many metrics like 'Hadoop:service=3DHBase,= name=3DRegionServer,sub=3DIPC' > -------------------------------------------------------------------------= ------------------------ > > Key: HBASE-18868 > URL: https://issues.apache.org/jira/browse/HBASE-18868 > Project: HBase > Issue Type: Bug > Components: metrics > Affects Versions: 1.2.1, 1.2.4, 1.2.6 > Environment: 2*6 cores process,E5-2620v3 , 128G memory , 1*600G S= AS=EF=BC=8C12*6T SATA > Linux version 2.6.32-504.el6.x86_64 (mockbuild@c6b9.bsys.dev.centos.org) = (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Wed Oct 15 04= :27:16 UTC 2014 > Reporter: wuchengzhi > Priority: Critical > Labels: jmx, metrics > Attachments: jmx-iffy.json > > Original Estimate: 48h > Remaining Estimate: 48h > > so many metrics error log printed in the regionserver log all the time, t= hen the web-gui http://hbase-master1:16010/jmx loss some metrics like=20 > 'Hadoop:service=3DHBase,name=3DRegionServer,sub=3DRegions' , 'Hadoop:ser= vice=3DHBase,name=3DRegionServer,sub=3DIPC' .......=20 > is there something wrong i configured at my hbase-site.xml or hbase-env.= sh? > regionserver logs: > 2017-09-21 05:03:42,088 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: = Scheduled snapshot period at 10 second(s). > 2017-09-21 05:03:42,088 INFO [HBase-Metrics2-1] impl.MetricsSystemImpl: = HBase metrics system started > 2017-09-21 05:03:42,088 WARN [HBase-Metrics2-1] util.MBeans: Error creat= ing MBean object name: Hadoop:service=3DHBase,name=3DJvmMetrics > org.apache.hadoop.metrics2.MetricsException: org.apache.hadoop.metrics2.M= etricsException: Hadoop:service=3DHBase,name=3DJvmMetrics already exists! > =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(D= efaultMetricsSystem.java:117) > =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(De= faultMetricsSystem.java:102) > =09at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:91) > =09at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:55) > =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(Me= tricsSourceAdapter.java:220) > =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsS= ourceAdapter.java:95) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(Me= tricsSystemImpl.java:244) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(Metri= csSystemImpl.java:228) > =09at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) > =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc= essorImpl.java:43) > =09at java.lang.reflect.Method.invoke(Method.java:606) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsS= ystemImpl.java:289) > =09at com.sun.proxy.$Proxy11.postStart(Unknown Source) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSyst= emImpl.java:184) > =09at org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnab= le.run(JmxCacheBuster.java:78) > =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:= 471) > =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) > =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas= k.access$201(ScheduledThreadPoolExecutor.java:178) > =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas= k.run(ScheduledThreadPoolExecutor.java:292) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1145) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:615) > =09at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.hadoop.metrics2.MetricsException: Hadoop:service=3D= HBase,name=3DJvmMetrics already exists! > =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(D= efaultMetricsSystem.java:113) > =09... 21 more > 2017-09-21 05:03:42,090 WARN [HBase-Metrics2-1] util.MBeans: Failed to r= egister MBean "null" > javax.management.RuntimeOperationsException: Exception occurred trying to= register the MBean > =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynam= icMBean(DefaultMBeanServerInterceptor.java:951) > =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObjec= t(DefaultMBeanServerInterceptor.java:900) > =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean= (DefaultMBeanServerInterceptor.java:324) > =09at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer= .java:522) > =09at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57) > =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(Me= tricsSourceAdapter.java:220) > =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsS= ourceAdapter.java:95) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(Me= tricsSystemImpl.java:244) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(Metri= csSystemImpl.java:228) > =09at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) > =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc= essorImpl.java:43) > =09at java.lang.reflect.Method.invoke(Method.java:606) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsS= ystemImpl.java:289) > =09at com.sun.proxy.$Proxy11.postStart(Unknown Source) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSyst= emImpl.java:184) > =09at org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnab= le.run(JmxCacheBuster.java:78) > =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:= 471) > =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) > =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas= k.access$201(ScheduledThreadPoolExecutor.java:178) > =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas= k.run(ScheduledThreadPoolExecutor.java:292) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1145) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:615) > =09at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.IllegalArgumentException: No object name specified > =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynam= icMBean(DefaultMBeanServerInterceptor.java:949) > =09... 22 more > 2017-09-21 05:03:42,090 WARN [HBase-Metrics2-1] util.MBeans: Error creat= ing MBean object name: Hadoop:service=3DHBase,name=3DRegionServer,sub=3DIPC > org.apache.hadoop.metrics2.MetricsException: org.apache.hadoop.metrics2.M= etricsException: Hadoop:service=3DHBase,name=3DRegionServer,sub=3DIPC alrea= dy exists! > =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(D= efaultMetricsSystem.java:117) > =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(De= faultMetricsSystem.java:102) > =09at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:91) > =09at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:55) > =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(Me= tricsSourceAdapter.java:220) > =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsS= ourceAdapter.java:95) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(Me= tricsSystemImpl.java:244) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(Metri= csSystemImpl.java:228) > =09at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) > =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc= essorImpl.java:43) > =09at java.lang.reflect.Method.invoke(Method.java:606) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsS= ystemImpl.java:289) > =09at com.sun.proxy.$Proxy11.postStart(Unknown Source) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSyst= emImpl.java:184) > =09at org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnab= le.run(JmxCacheBuster.java:78) > =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:= 471) > =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) > =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas= k.access$201(ScheduledThreadPoolExecutor.java:178) > =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas= k.run(ScheduledThreadPoolExecutor.java:292) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1145) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:615) > =09at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.hadoop.metrics2.MetricsException: Hadoop:service=3D= HBase,name=3DRegionServer,sub=3DIPC already exists! > =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(D= efaultMetricsSystem.java:113) > =09... 21 more > 2017-09-21 05:03:42,092 WARN [HBase-Metrics2-1] util.MBeans: Failed to r= egister MBean "null" > javax.management.RuntimeOperationsException: Exception occurred trying to= register the MBean > =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynam= icMBean(DefaultMBeanServerInterceptor.java:951) > =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObjec= t(DefaultMBeanServerInterceptor.java:900) > =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean= (DefaultMBeanServerInterceptor.java:324) > =09at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer= .java:522) > =09at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57) > =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(Me= tricsSourceAdapter.java:220) > =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsS= ourceAdapter.java:95) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(Me= tricsSystemImpl.java:244) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(Metri= csSystemImpl.java:228) > =09at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) > =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc= essorImpl.java:43) > =09at java.lang.reflect.Method.invoke(Method.java:606) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsS= ystemImpl.java:289) > =09at com.sun.proxy.$Proxy11.postStart(Unknown Source) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSyst= emImpl.java:184) > =09at org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnab= le.run(JmxCacheBuster.java:78) > =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:= 471) > =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) > =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas= k.access$201(ScheduledThreadPoolExecutor.java:178) > =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas= k.run(ScheduledThreadPoolExecutor.java:292) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1145) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:615) > =09at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.IllegalArgumentException: No object name specified > =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynam= icMBean(DefaultMBeanServerInterceptor.java:949) > =09... 22 more > 2017-09-21 05:03:42,092 WARN [HBase-Metrics2-1] util.MBeans: Error creat= ing MBean object name: Hadoop:service=3DHBase,name=3DUgiMetrics > org.apache.hadoop.metrics2.MetricsException: org.apache.hadoop.metrics2.M= etricsException: Hadoop:service=3DHBase,name=3DUgiMetrics already exists! > =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(D= efaultMetricsSystem.java:117) > =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(De= faultMetricsSystem.java:102) > =09at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:91) > =09at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:55) > =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(Me= tricsSourceAdapter.java:220) > =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsS= ourceAdapter.java:95) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(Me= tricsSystemImpl.java:244) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(Metri= csSystemImpl.java:228) > =09at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) > =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc= essorImpl.java:43) > =09at java.lang.reflect.Method.invoke(Method.java:606) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsS= ystemImpl.java:289) > =09at com.sun.proxy.$Proxy11.postStart(Unknown Source) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSyst= emImpl.java:184) > =09at org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnab= le.run(JmxCacheBuster.java:78) > =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:= 471) > =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) > =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas= k.access$201(ScheduledThreadPoolExecutor.java:178) > =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas= k.run(ScheduledThreadPoolExecutor.java:292) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1145) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:615) > =09at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.hadoop.metrics2.MetricsException: Hadoop:service=3D= HBase,name=3DUgiMetrics already exists! > =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(D= efaultMetricsSystem.java:113) > =09... 21 more > 2017-09-21 05:03:42,093 WARN [HBase-Metrics2-1] util.MBeans: Failed to r= egister MBean "null" > javax.management.RuntimeOperationsException: Exception occurred trying to= register the MBean > =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynam= icMBean(DefaultMBeanServerInterceptor.java:951) > =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObjec= t(DefaultMBeanServerInterceptor.java:900) > =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean= (DefaultMBeanServerInterceptor.java:324) > =09at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer= .java:522) > =09at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57) > =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(Me= tricsSourceAdapter.java:220) > =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsS= ourceAdapter.java:95) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(Me= tricsSystemImpl.java:244) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(Metri= csSystemImpl.java:228) > =09at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) > =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc= essorImpl.java:43) > =09at java.lang.reflect.Method.invoke(Method.java:606) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsS= ystemImpl.java:289) > =09at com.sun.proxy.$Proxy11.postStart(Unknown Source) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSyst= emImpl.java:184) > =09at org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnab= le.run(JmxCacheBuster.java:78) > =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:= 471) > =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) > =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas= k.access$201(ScheduledThreadPoolExecutor.java:178) > =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas= k.run(ScheduledThreadPoolExecutor.java:292) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1145) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:615) > =09at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.IllegalArgumentException: No object name specified > =09at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynam= icMBean(DefaultMBeanServerInterceptor.java:949) > =09... 22 more > 2017-09-21 05:03:42,093 WARN [HBase-Metrics2-1] util.MBeans: Error creat= ing MBean object name: Hadoop:service=3DHBase,name=3DRegionServer,sub=3DWAL > org.apache.hadoop.metrics2.MetricsException: org.apache.hadoop.metrics2.M= etricsException: Hadoop:service=3DHBase,name=3DRegionServer,sub=3DWAL alrea= dy exists! > =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(D= efaultMetricsSystem.java:117) > =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(De= faultMetricsSystem.java:102) > =09at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:91) > =09at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:55) > =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(Me= tricsSourceAdapter.java:220) > =09at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsS= ourceAdapter.java:95) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(Me= tricsSystemImpl.java:244) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(Metri= csSystemImpl.java:228) > =09at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) > =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc= essorImpl.java:43) > =09at java.lang.reflect.Method.invoke(Method.java:606) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsS= ystemImpl.java:289) > =09at com.sun.proxy.$Proxy11.postStart(Unknown Source) > =09at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSyst= emImpl.java:184) > =09at org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnab= le.run(JmxCacheBuster.java:78) > =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:= 471) > =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) > =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas= k.access$201(ScheduledThreadPoolExecutor.java:178) > =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas= k.run(ScheduledThreadPoolExecutor.java:292) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1145) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:615) > =09at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.hadoop.metrics2.MetricsException: Hadoop:service=3D= HBase,name=3DRegionServer,sub=3DWAL already exists! > =09at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(D= efaultMetricsSystem.java:113) > =09... 21 more -- This message was sent by Atlassian JIRA (v6.4.14#64029)