Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 9 Jun 2015 19:05:01 +0000 (UTC)
From: "Jerry He (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12835477.1433440763000.30834.1433876701315@Atlassian.JIRA>
In-Reply-To: <JIRA.12835477.1433440763000@Atlassian.JIRA>
References: <JIRA.12835477.1433440763000@Atlassian.JIRA>
 <JIRA.12835477.1433440763230@arcas>
Subject: [jira] [Commented] (HBASE-13845) Expire of one region server
 carrying meta can bring down the master
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-13845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14579412#comment-14579412 ] 

Jerry He commented on HBASE-13845:
----------------------------------

it seems there was a glitch applying the branch-1 patch from Hadoop QA.
It was applied fine with 'git apply' locally. There was mentioning on the hbase dev mailing list recently as I recall?

Committed the test case to the master branch. 

> Expire of one region server carrying meta can bring down the master
> -------------------------------------------------------------------
>
>                 Key: HBASE-13845
>                 URL: https://issues.apache.org/jira/browse/HBASE-13845
>             Project: HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 2.0.0, 1.1.0, 1.2.0
>            Reporter: Jerry He
>            Assignee: Jerry He
>             Fix For: 2.0.0, 1.2.0, 1.1.1
>
>         Attachments: HBASE-13845-branch-1.1.patch, HBASE-13845-branch-1.patch, HBASE-13845-master-test-case-only.patch
>
>
> There seems to be a code bug that can cause expiration of one region server carrying meta to bring down the master under certain case.
> Here is the sequence of event.
> a) The master detects the expiration of a region server on ZK, and starts to expire the region server.
> b) Since the failed region server carries meta, the shutdown handler will call verifyAndAssignMetaWithRetries() during processing the expired rs.
> c)  In verifyAndAssignMeta(), there is a logic to verifyMetaRegionLocation
> {code}
> (!server.getMetaTableLocator().verifyMetaRegionLocation(server.getConnection(),
>       this.server.getZooKeeper(), timeout)) {
>       this.services.getAssignmentManager().assignMeta
>       (HRegionInfo.FIRST_META_REGIONINFO);
>     } else if (serverName.equals(server.getMetaTableLocator().getMetaRegionLocation(
>       this.server.getZooKeeper()))) {
>       throw new IOException("hbase:meta is onlined on the dead server "
>           + serverName);
> {code}
> If we see the meta region is still alive on the expired rs, we throw an exception.
> We do some retries (default 10x1000ms) for verifyAndAssignMeta.
> If we still get the exception after retries, we abort the master.
> {code}
> 2015-05-27 06:58:30,156 FATAL [MASTER_META_SERVER_OPERATIONS-bdvs1163:60000-0] master.HMaster: Master server abort: loaded coprocessors are: []
> 2015-05-27 06:58:30,156 FATAL [MASTER_META_SERVER_OPERATIONS-bdvs1163:60000-0] master.HMaster: verifyAndAssignMeta failed after10 times retries, aborting
> java.io.IOException: hbase:meta is onlined on the dead server bdvs1164.svl.ibm.com,16020,1432681743203
>         at org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.verifyAndAssignMeta(MetaServerShutdownHandler.java:162)
>         at org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.verifyAndAssignMetaWithRetries(MetaServerShutdownHandler.java:184)
>         at org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:93)
>         at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> 2015-05-27 06:58:30,156 INFO  [MASTER_META_SERVER_OPERATIONS-bdvs1163:60000-0] regionserver.HRegionServer: STOPPED: verifyAndAssignMeta failed after10 times retries, aborting
> {code}
> The problem happens when the expired is slow processing its own expiration or has a slow death, and is still able to respond to master's meta verification in the meantime


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)