hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6294) Detect leftover data in ZK after a user delete all its HBase data
Date Tue, 28 Aug 2012 00:08:08 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442821#comment-13442821
] 

Lars Hofhansl commented on HBASE-6294:
--------------------------------------

I think this can be moved to 0.94.3 (unless somebody has a concrete plan about what to do
here).
                
> Detect leftover data in ZK after a user delete all its HBase data
> -----------------------------------------------------------------
>
>                 Key: HBASE-6294
>                 URL: https://issues.apache.org/jira/browse/HBASE-6294
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 0.94.0
>            Reporter: Jean-Daniel Cryans
>            Priority: Critical
>             Fix For: 0.96.0, 0.94.2
>
>
> It seems we have a new failure mode when a user deletes the hbase root.dir but doesn't
delete the ZK data. For example a user on IRC came with this log:
> {noformat}
> 2012-06-30 09:07:48,017 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received
request to open region: kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02.
> 2012-06-30 09:07:48,017 WARN org.apache.hadoop.hbase.util.FSTableDescriptors: The following
folder is in HBase's root directory and doesn't contain a table descriptor, do consider deleting
it: kw
> 2012-06-30 09:07:48,018 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:34193-0x1383bfe01b70001
Attempting to transition node 2e8a318837602c9c9961e9d690b7fd02 from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
> 2012-06-30 09:07:48,018 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=M_ZK_REGION_OFFLINE, server=localhost,50890,1341036299694, region=2e8a318837602c9c9961e9d690b7fd02
> 2012-06-30 09:07:48,020 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_FAILED_OPEN, server=localhost,34193,1341036300138, region=b254af24c9127b8bb22cb6d24e523dad
> 2012-06-30 09:07:48,020 DEBUG org.apache.hadoop.hbase.master.handler.ClosedRegionHandler:
Handling CLOSED event for b254af24c9127b8bb22cb6d24e523dad
> 2012-06-30 09:07:48,020 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing
OFFLINE; was=kw_r,,1340981822374.b254af24c9127b8bb22cb6d24e523dad. state=CLOSED, ts=1341036467998,
server=localhost,34193,1341036300138
> 2012-06-30 09:07:48,020 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:50890-0x1383bfe01b70000
Creating (or updating) unassigned node for b254af24c9127b8bb22cb6d24e523dad with OFFLINE state
> 2012-06-30 09:07:48,028 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:34193-0x1383bfe01b70001
Successfully transitioned node 2e8a318837602c9c9961e9d690b7fd02 from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
> 2012-06-30 09:07:48,028 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region:
{NAME => 'kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02.', STARTKEY => '', ENDKEY
=> '', ENCODED => 2e8a318837602c9c9961e9d690b7fd02,}
> 2012-06-30 09:07:48,029 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
Failed open of region=kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02., starting to roll
back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> 	at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3490)
> 	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3628)
> 	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> 	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> 	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> 	at java.lang.Thread.run(Thread.java:679)
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
> 	at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3487)
> 	... 7 more
> Caused by: java.lang.NullPointerException
> 	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:133)
> 	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:125)
> 	at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:411)
> 	... 11 more
> 2012-06-30 09:07:48,031 INFO org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
Opening of region {NAME => 'kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02.', STARTKEY
=> '', ENDKEY => '', ENCODED => 2e8a318837602c9c9961e9d690b7fd02,} failed, marking
as FAILED_OPEN in ZK
> 2012-06-30 09:07:48,032 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:34193-0x1383bfe01b70001
Attempting to transition node 2e8a318837602c9c9961e9d690b7fd02 from RS_ZK_REGION_OPENING to
RS_ZK_REGION_FAILED_OPEN
> 2012-06-30 09:07:48,031 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=localhost,34193,1341036300138, region=2e8a318837602c9c9961e9d690b7fd02
> 2012-06-30 09:07:48,043 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=M_ZK_REGION_OFFLINE, server=localhost,50890,1341036299694, region=b254af24c9127b8bb22cb6d24e523dad
> {noformat}
> The exception itself is not very useful, nor is the NPE deep in the coproc stack. What
was really useful was this:
> {noformat}
> 2012-06-30 09:07:48,017 WARN org.apache.hadoop.hbase.util.FSTableDescriptors: The following
folder is in HBase's root directory and doesn't contain a table descriptor, do consider deleting
it: kw
> {noformat}
> So the HBase wants to assign a region from a table that doesn't exist and we fail in
an obscure way. I told the user to shut down HBase, nuke /tmp/hbase-user as it will contain
both the HBase data and the ZK data, and restart. It worked.
> This situation is new in 0.94, we need to detect it so our users have a better experience
getting started with HBase.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message