hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HBASE-3190) Problem with disabling and droping table
Date Wed, 16 Jul 2014 23:17:05 GMT

     [ https://issues.apache.org/jira/browse/HBASE-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Purtell resolved HBASE-3190.
-----------------------------------

    Resolution: Cannot Reproduce

Reopen or file new issue is still relevant with modern HBase versions

> Problem with disabling and droping table
> ----------------------------------------
>
>                 Key: HBASE-3190
>                 URL: https://issues.apache.org/jira/browse/HBASE-3190
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Sebastian Bauer
>
> Table disabling was interrupted by kill -9 all part of hbase and now we cannot do anything
with this table, disabling doesn't show any exception:
> hbase(main):019:0> disable 'NGolden_CTU'
> 0 row(s) in 0.0250 seconds
> but droping show this:
> hbase(main):020:0> drop 'NGolden_CTU'   
> ERROR: org.apache.hadoop.hbase.TableNotDisabledException: org.apache.hadoop.hbase.TableNotDisabledException:
NGolden_CTU
>         at org.apache.hadoop.hbase.master.HMaster.checkTableModifiable(HMaster.java:861)
>         at org.apache.hadoop.hbase.master.handler.TableEventHandler.<init>(TableEventHandler.java:52)
>         at org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(DeleteTableHandler.java:42)
>         at org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:779)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
> Here is some help for this command:
>           Drop the named table. Table must first be disabled. If table has
>           more than one region, run a major compaction on .META.:
>           hbase> major_compact ".META."
> after this nothing strange is in logs
> when we restart hbase we get this:
> 2010-11-03 08:56:37,892 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
Processing open of NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
                                                       
> 2010-11-03 08:56:37,892 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12c10b5fb780005
Attempting to transition node 0c8579e52b0ea3f2dab5b6a857ad030b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING                                                         
> 2010-11-03 08:56:37,892 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable
while processing event M_RS_OPEN_REGION                                                  
                                                                                         
     
> java.lang.NullPointerException                                                      
                                                                                         
                                                                                         
              
>         at org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:75)    
                                                                                         
                                                                                         
              
>         at org.apache.hadoop.hbase.executor.RegionTransitionData.fromBytes(RegionTransitionData.java:198)
                                                                                         
                                                                                   
>         at org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:669)
                                                                                         
                                                                                         
           
>         at org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNodeOpening(ZKAssign.java:549)
                                                                                         
                                                                                         
    
>         at org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNodeOpening(ZKAssign.java:542)
                                                                                         
                                                                                         
    
>         at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.transitionZookeeperOfflineToOpening(OpenRegionHandler.java:208)
                                                                                         
                                                   
>         at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:89)
                                                                                         
                                                                                
>         at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151) 
                                                                                         
                                                                                         
              
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                                                                                         
                                                                                         
    
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                                                                                         
                                                                                         
        
>         at java.lang.Thread.run(Thread.java:619)                                    
                                         
> Other logs with this region:
> hbase@db2a:logs$ grep "0c8579e52b0ea3f2dab5b6a857ad030b" hbase-hbase-master-db2a.goldenline.pl.log

> 2010-11-03 08:54:02,575 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12c10b5fb780002
Async create of unassigned node for 0c8579e52b0ea3f2dab5b6a857ad030b with OFFLINE state
> 2010-11-03 08:54:03,555 DEBUG org.apache.hadoop.hbase.master.AssignmentManager$CreateUnassignedAsyncCallback:
rs=NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
state=OFFLINE, ts=1288770842575, server=db2a.goldenline.pl,60020,1288770551154
> 2010-11-03 08:54:03,777 DEBUG org.apache.hadoop.hbase.master.AssignmentManager$ExistsUnassignedAsyncCallback:
rs=NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
state=OFFLINE, ts=1288770842575
> 2010-11-03 08:54:42,836 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
state=PENDING_OPEN, ts=1288770843777
> 2010-11-03 08:54:42,836 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been PENDING_OPEN for too long, reassigning region=NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
> 2010-11-03 08:54:42,836 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing
OFFLINE; was=NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
state=PENDING_OPEN, ts=1288770843777
> 2010-11-03 08:54:42,836 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
so generated a random one; hri=NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.,
src=, dest=db2b.goldenline.pl,60020,1288770553679; 2 (online=2, exclude=null) available servers
> 2010-11-03 08:54:42,836 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning
region NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
to db2b.goldenline.pl,60020,1288770553679
> 2010-11-03 08:56:12,824 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=db2a.goldenline.pl,60020,1288770551154, region=0c8579e52b0ea3f2dab5b6a857ad030b
> 2010-11-03 08:56:12,975 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENED, server=db2a.goldenline.pl,60020,1288770551154, region=0c8579e52b0ea3f2dab5b6a857ad030b
> 2010-11-03 08:56:12,976 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler:
Handling OPENED event for 0c8579e52b0ea3f2dab5b6a857ad030b; deleting unassigned node
> 2010-11-03 08:56:12,976 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12c10b5fb780002
Deleting existing unassigned node for 0c8579e52b0ea3f2dab5b6a857ad030b that is in expected
state RS_ZK_REGION_OPENED
> 2010-11-03 08:56:12,978 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12c10b5fb780002
Successfully deleted unassigned node for region 0c8579e52b0ea3f2dab5b6a857ad030b in expected
state RS_ZK_REGION_OPENED
> 2010-11-03 08:56:13,011 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler:
Opened region NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
on db2a.goldenline.pl,60020,1288770551154
> hbase@db2a:logs$ grep "0c8579e52b0ea3f2dab5b6a857ad030b" hbase-hbase-regionserver-db2a.goldenline.pl.log

> 2010-11-03 08:54:04,470 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received
request to open region: NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
> 2010-11-03 08:56:12,770 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
Processing open of NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
> 2010-11-03 08:56:12,770 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12c10b5fb780001
Attempting to transition node 0c8579e52b0ea3f2dab5b6a857ad030b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
> 2010-11-03 08:56:12,819 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12c10b5fb780001
Successfully transitioned node 0c8579e52b0ea3f2dab5b6a857ad030b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
> 2010-11-03 08:56:12,819 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region:
REGION => {NAME => 'NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.',
STARTKEY => '3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E', ENDKEY => '3065-d_2010_9_11_47D955785DEC3CE95377D70F67BA5ABC',
ENCODED => 0c8579e52b0ea3f2dab5b6a857ad030b, TABLE => {{NAME => 'NGolden_CTU', FAMILIES
=> [{NAME => 'c', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION
=> 'LZO', VERSIONS => '1', TTL => '-1', BLOCKSIZE => '65536', IN_MEMORY =>
'false', BLOCKCACHE => 'true'}]}}
> 2010-11-03 08:56:12,819 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
> 2010-11-03 08:56:12,939 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded
row bloom filter metadata for hdfs://db2a:50001/hbase/NGolden_CTU/0c8579e52b0ea3f2dab5b6a857ad030b/c/561636411807463667
> 2010-11-03 08:56:12,939 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://db2a:50001/hbase/NGolden_CTU/0c8579e52b0ea3f2dab5b6a857ad030b/c/561636411807463667,
isReference=false, isBulkLoadResult=false, seqid=1820373362, majorCompaction=true
> 2010-11-03 08:56:12,939 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.;
next sequenceid=1820373363
> 2010-11-03 08:56:12,939 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12c10b5fb780001
Attempting to transition node 0c8579e52b0ea3f2dab5b6a857ad030b from RS_ZK_REGION_OPENING to
RS_ZK_REGION_OPENING
> 2010-11-03 08:56:12,970 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12c10b5fb780001
Successfully transitioned node 0c8579e52b0ea3f2dab5b6a857ad030b from RS_ZK_REGION_OPENING
to RS_ZK_REGION_OPENING
> 2010-11-03 08:56:12,971 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Updated row
NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
in region .META.,,1 with server=db2a.goldenline.pl:60020, startcode=1288770551154
> 2010-11-03 08:56:12,971 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12c10b5fb780001
Attempting to transition node 0c8579e52b0ea3f2dab5b6a857ad030b from RS_ZK_REGION_OPENING to
RS_ZK_REGION_OPENED
> 2010-11-03 08:56:12,973 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12c10b5fb780001
Successfully transitioned node 0c8579e52b0ea3f2dab5b6a857ad030b from RS_ZK_REGION_OPENING
to RS_ZK_REGION_OPENED
> 2010-11-03 08:56:12,973 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
Opened NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
> hbase@db2b:logs$  grep "0c8579e52b0ea3f2dab5b6a857ad030b" hbase-hbase-regionserver-db2b.goldenline.pl.log
> 2010-11-03 08:54:42,840 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received
request to open region: NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
> 2010-11-03 08:56:37,892 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
Processing open of NGolden_CTU,3065-d_2010_10_14_245FF1A15F4E236002ED3AB651BAB97E,1288046281444.0c8579e52b0ea3f2dab5b6a857ad030b.
> 2010-11-03 08:56:37,892 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12c10b5fb780005
Attempting to transition node 0c8579e52b0ea3f2dab5b6a857ad030b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message