Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 19 Jun 2015 19:12:01 +0000 (UTC)
From: "Jerry He (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12838867.1434649383000.122519.1434741121324@Atlassian.JIRA>
In-Reply-To: <JIRA.12838867.1434649383000@Atlassian.JIRA>
References: <JIRA.12838867.1434649383000@Atlassian.JIRA>
 <JIRA.12838867.1434649383110@arcas>
Subject: [jira] [Commented] (HBASE-13935) Orphaned namespace table ZK node
 should not prevent master to start
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14593813#comment-14593813 ] 

Jerry He commented on HBASE-13935:
----------------------------------

{code}
2015-05-30 16:56:41,481 INFO  [ProcedureExecutorThread-0] zookeeper.ZKTableStateManager: Moving table hbase:namespace state from ENABLED to ENABLING
2015-05-30 16:56:41,486 INFO  [ProcedureExecutorThread-0] master.AssignmentManager: Assigning 1 region(s) to ip-666-77-88-999.ec2.internal,16020,1433004979527
2015-05-30 16:56:41,509 INFO  [ProcedureExecutorThread-0] master.RegionStates: Transition {9df720124fbb66974e95de65bddaee82 state=OFFLINE, ts=1433005001494, server=null} to {9df720124fbb66974e95de65bddaee82 state=PENDING_OPEN, ts=1433005001509, server=ip-666-77-88-999.ec2.internal,16020,1433004979527}
2015-05-30 16:56:41,872 INFO  [ProcedureExecutorThread-0] zookeeper.ZKTableStateManager: Moving table hbase:namespace state from ENABLING to ENABLED
{code}
This is from your unit test run, so we started with ENABLED?
I think the patch looks ok.
We already did a prior check that the table hbase:namespace does not exist in meta accessor. So we won't mess up there. There may be table files on the filesystem? Then we will fail later.


> Orphaned namespace table ZK node should not prevent master to start
> -------------------------------------------------------------------
>
>                 Key: HBASE-13935
>                 URL: https://issues.apache.org/jira/browse/HBASE-13935
>             Project: HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 1.0.0, 0.98.13
>            Reporter: Stephen Yuan Jiang
>            Assignee: Stephen Yuan Jiang
>             Fix For: 0.98.14, 1.0.2
>
>         Attachments: HBASE-13935.v1-0.98.patch, HBASE-13935.v1-branch-1.0.patch
>
>
> Before we have the state-of-art Procedure V2 feature (HBASE 1.0 release or older), we frequently see the following issue (orphaned ZK node) that prevent master to start (at least in testing):
> {noformat}
> 2015-06-16 17:54:36,472 FATAL [master:10.0.0.99:60000] master.HMaster: Unhandled exception. Starting shutdown.
> org.apache.hadoop.hbase.TableExistsException: hbase:namespace
> 	at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:137)
> 	at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
> 	at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
> 	at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1123)
> 	at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:947)
> 	at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:618)
> 	at java.lang.Thread.run(Thread.java:745)
> 2015-06-16 17:54:36,472 INFO  [master:10.0.0.99:60000] master.HMaster: Aborting
> {noformat}
> The above call trace is from a 0.98.x test run.  We saw similar issue in 1.0.x run, too.  
> The proposed fix is to ignore the zk node and force namespace table creation to be complete so that master can start successfully.  


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)