Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: <2015091021094953084020@sina.com>
References: <2015091021094953084020@sina.com>
Date: Thu, 10 Sep 2015 07:23:24 -0700
Message-ID: 
 <CALte62ybYA_O2NRoSx0JEAZD4ocmA-BsZEM1W2LsbCAnUYb5ZA@mail.gmail.com>
Subject: Re: NotServingRegion: hbase region closed forever when open region
 response time out
From: Ted Yu <yuzhihong@gmail.com>
To: "zhou_shuaifeng@sina.com" <zhou_shuaifeng@sina.com>
Cc: "dev@hbase.apache.org" <dev@hbase.apache.org>,
 =?UTF-8?B?5byg6ZOO?= <palomino219@gmail.com>,
	wangyongqiang0617 <wangyongqiang0617@163.com>
Content-Type: multipart/alternative; boundary=001a11481e0a921164051f6557d8

--001a11481e0a921164051f6557d8
Content-Type: text/plain; charset=UTF-8

Can you come up with a test that shows the problem ?

Consider opening a JIRA with anonymized master log, your test and proposed
solution (if you have one).

Cheers

On Thu, Sep 10, 2015 at 6:09 AM, zhou_shuaifeng@sina.com <
zhou_shuaifeng@sina.com> wrote:

> Hi all,
>     I found a situation may cause region closed forever, and this
> situation happend usually on my cluster, version is 0.98.10, but 1.1.2 also
> have the problem:
>     1, master send region open to regionserver
>     2, rs open a handler do openregion
>     3, rs return resopnse to master
>     3, master not received the response, or timeout, send open region again
>     4, rs already opened the region
>     5, master processAlreadyOpenedRegion, update regionstate open in
> master memory
>     6, master received zk message region opened(for some reason late, eg:
> net work), and triger update regionstate open, but find that region already
> opened, ERROR!
>     7, master send close region, and region be closed forever.
>
> may be a solution is change processAlreadyOpenedRegion in class
> AssignmentManager:
>
>
> private void processAlreadyOpenedRegion(HRegionInfo region, ServerName sn) {
>     // Remove region from in-memory transition and unassigned node from ZK
>     // While trying to enable the table the regions of the table were
>     // already enabled.
>     LOG.debug("ALREADY_OPENED " + region.getRegionNameAsString()
>       + " to " + sn);
>     String encodedName = region.getEncodedName();
>
>      /**
>      *  check region state in zk, if already opened, return; leave the
> regionStates work to zkStatus change to trigger.
>     **/
>
>
>
>     deleteNodeInStates(encodedName, "offline", sn, EventType.M_ZK_REGION_OFFLINE);
>     regionStates.regionOnline(region, sn);
>   }
> ------------------------------
> zhou_shuaifeng@sina.com
>

--001a11481e0a921164051f6557d8--