hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephen Yuan Jiang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-13935) Orphaned namespace table ZK node should not prevent master to start
Date Fri, 19 Jun 2015 17:23:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14593663#comment-14593663
] 

Stephen Yuan Jiang commented on HBASE-13935:
--------------------------------------------

[~mbertozzi] I found the log after applying a similar fix in Branch-1.1 and resolve the issue
(not sure how we reached to this state with PV2, but the patch fixed the issue and no inconsistency
(note: the table is indeed in ENABLED state when this issue happened):

{noformat}
2015-05-30 16:56:40,213 TRACE [ProcedureExecutorThread-0] procedure.CreateTableProcedure:
CreateTableProcedure (table=hbase:namespace) user=hbase/ip-666-77-88-999.internal@TESTING.COM
(auth:KERBEROS) id=9 state=RUNNABLE execute state=CREATE_TABLE_PRE_OPERATION
2015-05-30 16:56:40,373 TRACE [ProcedureExecutorThread-0] procedure.CreateTableProcedure:
CreateTableProcedure (table=hbase:namespace) user=hbase/ip-666-77-88-999.internal@TESTING.COM
(auth:KERBEROS) id=9 state=RUNNABLE execute state=CREATE_TABLE_WRITE_FS_LAYOUT
2015-05-30 16:56:40,571 INFO  [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion:
creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', DATA_BLOCK_ENCODING
=> 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE',
VERSIONS => '10', TTL => 'FOREVER', MIN_VERSIONS => '0', CACHE_DATA_IN_L1 => 'true',
KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '8192', IN_MEMORY => 'true', BLOCKCACHE
=> 'true'} RootDir = hdfs://ip-172-31-46-122.ec2.internal:8020/apps/hbase/data/.tmp Table
name == hbase:namespace
2015-05-30 16:56:40,803 INFO  [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion:
Closed hbase:namespace,,1433004999998.9df720124fbb66974e95de65bddaee82.
2015-05-30 16:56:40,925 TRACE [ProcedureExecutorThread-0] procedure.CreateTableProcedure:
CreateTableProcedure (table=hbase:namespace) user=hbase/ip-666-77-88-999.internal@TESTING.COM
(auth:KERBEROS) id=9 state=RUNNABLE execute state=CREATE_TABLE_ADD_TO_META
2015-05-30 16:56:41,370 INFO  [ProcedureExecutorThread-0] hbase.MetaTableAccessor: Added 2
2015-05-30 16:56:41,480 TRACE [ProcedureExecutorThread-0] procedure.CreateTableProcedure:
CreateTableProcedure (table=hbase:namespace) user=hbase/ip-666-77-88-999.internal@TESTING.COM
(auth:KERBEROS) id=9 state=RUNNABLE execute state=CREATE_TABLE_ASSIGN_REGIONS
2015-05-30 16:56:41,481 INFO  [ProcedureExecutorThread-0] zookeeper.ZKTableStateManager: Moving
table hbase:namespace state from ENABLED to ENABLING
2015-05-30 16:56:41,486 INFO  [ProcedureExecutorThread-0] master.AssignmentManager: Assigning
1 region(s) to ip-666-77-88-999.ec2.internal,16020,1433004979527
2015-05-30 16:56:41,509 INFO  [ProcedureExecutorThread-0] master.RegionStates: Transition
{9df720124fbb66974e95de65bddaee82 state=OFFLINE, ts=1433005001494, server=null} to {9df720124fbb66974e95de65bddaee82
state=PENDING_OPEN, ts=1433005001509, server=ip-666-77-88-999.ec2.internal,16020,1433004979527}
2015-05-30 16:56:41,872 INFO  [ProcedureExecutorThread-0] zookeeper.ZKTableStateManager: Moving
table hbase:namespace state from ENABLING to ENABLED
2015-05-30 16:56:41,887 INFO  [AM.ZK.Worker-pool2-t5] master.RegionStates: Transition {9df720124fbb66974e95de65bddaee82
state=PENDING_OPEN, ts=1433005001509, server=ip-666-77-88-999.ec2.internal,16020,1433004979527}
to {9df720124fbb66974e95de65bddaee82 state=OPENING, ts=1433005001887, server=ip-666-77-88-999.ec2.internal,16020,1433004979527}
2015-05-30 16:56:41,997 TRACE [ProcedureExecutorThread-0] procedure.CreateTableProcedure:
CreateTableProcedure (table=hbase:namespace) user=hbase/ip-666-77-88-999.internal@TESTING.COM
(auth:KERBEROS) id=9 state=RUNNABLE execute state=CREATE_TABLE_UPDATE_DESC_CACHE
2015-05-30 16:56:42,300 TRACE [ProcedureExecutorThread-0] procedure.CreateTableProcedure:
CreateTableProcedure (table=hbase:namespace) user=hbase/ip-666-77-88-999.internal@TESTING.COM
(auth:KERBEROS) id=9 state=RUNNABLE execute state=CREATE_TABLE_POST_OPERATION
{noformat}

> Orphaned namespace table ZK node should not prevent master to start
> -------------------------------------------------------------------
>
>                 Key: HBASE-13935
>                 URL: https://issues.apache.org/jira/browse/HBASE-13935
>             Project: HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 1.0.0, 0.98.13
>            Reporter: Stephen Yuan Jiang
>            Assignee: Stephen Yuan Jiang
>             Fix For: 0.98.14, 1.0.2
>
>         Attachments: HBASE-13935.v1-0.98.patch, HBASE-13935.v1-branch-1.0.patch
>
>
> Before we have the state-of-art Procedure V2 feature (HBASE 1.0 release or older), we
frequently see the following issue (orphaned ZK node) that prevent master to start (at least
in testing):
> {noformat}
> 2015-06-16 17:54:36,472 FATAL [master:10.0.0.99:60000] master.HMaster: Unhandled exception.
Starting shutdown.
> org.apache.hadoop.hbase.TableExistsException: hbase:namespace
> 	at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:137)
> 	at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
> 	at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
> 	at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1123)
> 	at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:947)
> 	at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:618)
> 	at java.lang.Thread.run(Thread.java:745)
> 2015-06-16 17:54:36,472 INFO  [master:10.0.0.99:60000] master.HMaster: Aborting
> {noformat}
> The above call trace is from a 0.98.x test run.  We saw similar issue in 1.0.x run, too.
 
> The proposed fix is to ignore the zk node and force namespace table creation to be complete
so that master can start successfully.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message