Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8B23318081 for ; Mon, 10 Aug 2015 16:19:46 +0000 (UTC) Received: (qmail 49621 invoked by uid 500); 10 Aug 2015 16:19:46 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 49567 invoked by uid 500); 10 Aug 2015 16:19:46 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 49554 invoked by uid 99); 10 Aug 2015 16:19:46 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2015 16:19:46 +0000 Date: Mon, 10 Aug 2015 16:19:46 +0000 (UTC) From: "Ted Yu (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HBASE-14190) Assign system tables ahead of user region assignment MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-14190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14190: --------------------------- Attachment: (was: 14190-v3.txt) > Assign system tables ahead of user region assignment > ---------------------------------------------------- > > Key: HBASE-14190 > URL: https://issues.apache.org/jira/browse/HBASE-14190 > Project: HBase > Issue Type: Bug > Reporter: Ted Yu > Assignee: Ted Yu > Priority: Critical > Attachments: 14190-v1.txt, 14190-v2.txt, 14190-v3.txt, 14190-v3.txt, 14190-v4.txt, 14190-v5.txt > > > Currently the namespace table region is assigned like user regions. > I spent several hours working with a customer where master couldn't finish initialization. > Even though master was restarted quite a few times, it went down with the following: > {code} > 2015-08-05 17:16:57,530 FATAL [hdpmaster1:60000.activeMasterManager] master.HMaster: Master server abort: loaded coprocessors are: [] > 2015-08-05 17:16:57,530 FATAL [hdpmaster1:60000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown. > java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned > at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104) > at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:985) > at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:779) > at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182) > at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646) > at java.lang.Thread.run(Thread.java:744) > {code} > During previous run(s), namespace table was created, hence leaving an entry in hbase:meta. > The following if block in TableNamespaceManager#start() was skipped: > {code} > if (!MetaTableAccessor.tableExists(masterServices.getConnection(), > TableName.NAMESPACE_TABLE_NAME)) { > {code} > TableNamespaceManager#start() spins, waiting for namespace region to be assigned. > There was issue in master assigning user regions. > We tried issuing 'assign' command from hbase shell which didn't work because of the following check in MasterRpcServices#assignRegion(): > {code} > master.checkInitialized(); > {code} > This scenario can be avoided if we assign hbase:namespace table after hbase:meta is assigned but before user table region assignment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)