Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0B3531097F for ; Fri, 5 Jul 2013 07:31:53 +0000 (UTC) Received: (qmail 9586 invoked by uid 500); 5 Jul 2013 07:31:52 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 9314 invoked by uid 500); 5 Jul 2013 07:31:50 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 9301 invoked by uid 99); 5 Jul 2013 07:31:48 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Jul 2013 07:31:48 +0000 Date: Fri, 5 Jul 2013 07:31:48 +0000 (UTC) From: "Nicolas Liochon (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-8871) The region server can crash at startup MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-8871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13700500#comment-13700500 ] Nicolas Liochon commented on HBASE-8871: ---------------------------------------- committed, thanks for the review Stack! > The region server can crash at startup > -------------------------------------- > > Key: HBASE-8871 > URL: https://issues.apache.org/jira/browse/HBASE-8871 > Project: HBase > Issue Type: Bug > Components: regionserver > Affects Versions: 0.98.0, 0.95.2 > Reporter: Nicolas Liochon > Assignee: Nicolas Liochon > Attachments: 8871.v1.patch > > > I have this stack when I start a fresh region server. 5% of the time I would say (per region server). > {code} > 2013-07-04 12:00:22,609 FATAL [regionserver60020] regionserver.HRegionServer: ABORTING region server ip-10-137-7-67.ec2.internal,60020,1372939221819: Initialization of RS failed. Hence aborting RS. > java.util.ConcurrentModificationException > at java.util.Hashtable$Enumerator.next(Hashtable.java:1200) > at org.apache.hadoop.conf.Configuration.iterator(Configuration.java:1820) > at org.apache.hadoop.hbase.zookeeper.ZKConfig.makeZKProps(ZKConfig.java:92) > at org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:267) > at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:158) > at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:133) > at org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:667) > at org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:647) > at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:778) > at java.lang.Thread.run(Thread.java:722) > 2013-07-04 12:00:22,614 FATAL [regionserver60020] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [] > 2013-07-04 12:00:22,614 INFO [regionserver60020] regionserver.HRegionServer: STOPPED: Initialization of RS failed. Hence aborting RS. > 2013-07-04 12:00:22,616 FATAL [regionserver60020] regionserver.HRegionServer: ABORTING region server ip-10-137-7-67.ec2.internal,60020,1372939221819: Unhandled: null > java.lang.NullPointerException > at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:798) > at java.lang.Thread.run(Thread.java:722) > 2013-07-04 12:00:22,616 FATAL [regionserver60020] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [] > 2013-07-04 12:00:22,617 INFO [regionserver60020] regionserver.HRegionServer: STOPPED: Unhandled: null > 2013-07-04 12:00:22,767 INFO [main] regionserver.ShutdownHook: Installed shutdown hook thread: Shutdownhook:regionserver60020 > 2013-07-04 12:00:22,768 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting > java.lang.RuntimeException: HRegionServer Aborted > at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:66) > at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:78) > at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2309) > 2013-07-04 12:00:22,770 INFO [Thread-4] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@21f0dbb9 > {code} > There is one bug here in the region server: we should not start the snapshot if the machine is supposed to stop: > {code} > // start the snapshot handler, since the server is ready to run > this.snapshotManager.start(); > {code} > and the root cause is here in ZKConfig: > {code} > for (Entry entry : conf) { // <=== BUG > String key = entry.getKey(); > if (key.startsWith(HConstants.ZK_CFG_PROPERTY_PREFIX)) { > String zkKey = key.substring(HConstants.ZK_CFG_PROPERTY_PREFIX_LEN); > String value = entry.getValue(); > // If the value has variables substitutions, need to do a get. > if (value.contains(VARIABLE_START)) { > value = conf.get(key); > } > zkProperties.put(zkKey, value); > } > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira