hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allan Yang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-14190) Assign system tables ahead of user region assignment
Date Thu, 29 Jun 2017 06:28:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067819#comment-16067819
] 

Allan Yang edited comment on HBASE-14190 at 6/29/17 6:27 AM:
-------------------------------------------------------------

{quote}
Sounds reasonable to me. Do we need change in the meta region opener in case namespace gets
scheduled ahead of meta?
{quote}
in current design, the only case the namespace gets scheduled ahead of meta is that, namespace
and meta region are host on different rs, and they crashed at the same time, and then the
SSH assign both meta region and meta to the same server, and open namespace region request
arrivals before open meta region request... That's nearly impossible

wait a minute... If this happens, open namespace region will stuck since meta region is not
online, and assign meta will stuck also because there only one thread for meta hander. maybe
this is not a good idea.


was (Author: allan163):
{quote}
Sounds reasonable to me. Do we need change in the meta region opener in case namespace gets
scheduled ahead of meta?
{quote}
No need, in current design, the only case the namespace gets scheduled ahead of meta is that,
namespace and meta region are host on different rs, and they crashed at the same time, and
then the SSH assign both meta region and meta to the same server, and open namespace region
request arrivals before open meta region request... That's nearly impossible

If no objection, I will open a new issue

> Assign system tables ahead of user region assignment
> ----------------------------------------------------
>
>                 Key: HBASE-14190
>                 URL: https://issues.apache.org/jira/browse/HBASE-14190
>             Project: HBase
>          Issue Type: Bug
>          Components: Region Assignment
>            Reporter: Ted Yu
>            Assignee: Ted Yu
>            Priority: Critical
>         Attachments: 14190-system-wal-v1.txt, 14190-v12.4.txt, 14190-v12.txt
>
>
> Currently the namespace table region is assigned like user regions.
> I spent several hours working with a customer where master couldn't finish initialization.
> Even though master was restarted quite a few times, it went down with the following:
> {code}
> 2015-08-05 17:16:57,530 FATAL [hdpmaster1:60000.activeMasterManager] master.HMaster:
Master server abort: loaded coprocessors are: []
> 2015-08-05 17:16:57,530 FATAL [hdpmaster1:60000.activeMasterManager] master.HMaster:
Unhandled exception. Starting shutdown.
> java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned
>   at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104)
>   at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:985)
>   at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:779)
>   at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
>   at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
>   at java.lang.Thread.run(Thread.java:744)
> {code}
> During previous run(s), namespace table was created, hence leaving an entry in hbase:meta.
> The following if block in TableNamespaceManager#start() was skipped:
> {code}
>     if (!MetaTableAccessor.tableExists(masterServices.getConnection(),
>       TableName.NAMESPACE_TABLE_NAME)) {
> {code}
> TableNamespaceManager#start() spins, waiting for namespace region to be assigned.
> There was issue in master assigning user regions.
> We tried issuing 'assign' command from hbase shell which didn't work because of the following
check in MasterRpcServices#assignRegion():
> {code}
>       master.checkInitialized();
> {code}
> This scenario can be avoided if we assign hbase:namespace table after hbase:meta is assigned
but before user table region assignment.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message