Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C5928F9CD for ; Tue, 7 May 2013 00:38:16 +0000 (UTC) Received: (qmail 21353 invoked by uid 500); 7 May 2013 00:38:16 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 21307 invoked by uid 500); 7 May 2013 00:38:16 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 21298 invoked by uid 99); 7 May 2013 00:38:16 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 May 2013 00:38:16 +0000 Date: Tue, 7 May 2013 00:38:16 +0000 (UTC) From: "Jean-Daniel Cryans (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-6294) Detect leftover data in ZK after a user delete all its HBase data MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13650302#comment-13650302 ] Jean-Daniel Cryans commented on HBASE-6294: ------------------------------------------- At least on the tip of 0.94 in standalone mode I can wipe out HBase's root dir and restart without problems, ZK uses a new data folder when it's restarted. Not sure when this was introduced. I'm fine closing this unless [~lhofhansl], [~devaraj] or [~larsgeorge] have something against it. > Detect leftover data in ZK after a user delete all its HBase data > ----------------------------------------------------------------- > > Key: HBASE-6294 > URL: https://issues.apache.org/jira/browse/HBASE-6294 > Project: HBase > Issue Type: Improvement > Affects Versions: 0.94.0 > Reporter: Jean-Daniel Cryans > Fix For: 0.95.1 > > > It seems we have a new failure mode when a user deletes the hbase root.dir but doesn't delete the ZK data. For example a user on IRC came with this log: > {noformat} > 2012-06-30 09:07:48,017 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02. > 2012-06-30 09:07:48,017 WARN org.apache.hadoop.hbase.util.FSTableDescriptors: The following folder is in HBase's root directory and doesn't contain a table descriptor, do consider deleting it: kw > 2012-06-30 09:07:48,018 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:34193-0x1383bfe01b70001 Attempting to transition node 2e8a318837602c9c9961e9d690b7fd02 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING > 2012-06-30 09:07:48,018 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE, server=localhost,50890,1341036299694, region=2e8a318837602c9c9961e9d690b7fd02 > 2012-06-30 09:07:48,020 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_FAILED_OPEN, server=localhost,34193,1341036300138, region=b254af24c9127b8bb22cb6d24e523dad > 2012-06-30 09:07:48,020 DEBUG org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED event for b254af24c9127b8bb22cb6d24e523dad > 2012-06-30 09:07:48,020 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=kw_r,,1340981822374.b254af24c9127b8bb22cb6d24e523dad. state=CLOSED, ts=1341036467998, server=localhost,34193,1341036300138 > 2012-06-30 09:07:48,020 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:50890-0x1383bfe01b70000 Creating (or updating) unassigned node for b254af24c9127b8bb22cb6d24e523dad with OFFLINE state > 2012-06-30 09:07:48,028 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:34193-0x1383bfe01b70001 Successfully transitioned node 2e8a318837602c9c9961e9d690b7fd02 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING > 2012-06-30 09:07:48,028 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: {NAME => 'kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02.', STARTKEY => '', ENDKEY => '', ENCODED => 2e8a318837602c9c9961e9d690b7fd02,} > 2012-06-30 09:07:48,029 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02., starting to roll back the global memstore size. > java.lang.IllegalStateException: Could not instantiate a region instance. > at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3490) > at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3628) > at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332) > at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108) > at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:532) > at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3487) > ... 7 more > Caused by: java.lang.NullPointerException > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:133) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.(RegionCoprocessorHost.java:125) > at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:411) > ... 11 more > 2012-06-30 09:07:48,031 INFO org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening of region {NAME => 'kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02.', STARTKEY => '', ENDKEY => '', ENCODED => 2e8a318837602c9c9961e9d690b7fd02,} failed, marking as FAILED_OPEN in ZK > 2012-06-30 09:07:48,032 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:34193-0x1383bfe01b70001 Attempting to transition node 2e8a318837602c9c9961e9d690b7fd02 from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN > 2012-06-30 09:07:48,031 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=localhost,34193,1341036300138, region=2e8a318837602c9c9961e9d690b7fd02 > 2012-06-30 09:07:48,043 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE, server=localhost,50890,1341036299694, region=b254af24c9127b8bb22cb6d24e523dad > {noformat} > The exception itself is not very useful, nor is the NPE deep in the coproc stack. What was really useful was this: > {noformat} > 2012-06-30 09:07:48,017 WARN org.apache.hadoop.hbase.util.FSTableDescriptors: The following folder is in HBase's root directory and doesn't contain a table descriptor, do consider deleting it: kw > {noformat} > So the HBase wants to assign a region from a table that doesn't exist and we fail in an obscure way. I told the user to shut down HBase, nuke /tmp/hbase-user as it will contain both the HBase data and the ZK data, and restart. It worked. > This situation is new in 0.94, we need to detect it so our users have a better experience getting started with HBase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira