Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com
 designates 209.85.216.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=iXOIGD44fyy3B2TwUVL0AO2DQNqT2QjosRv5wSdnMoSxi3vllp4Fz4XJjPfINuRzTj
         RqmVL/YdKAGmabhTkpl4ULogvvB3Vil4Ef0jt5TcHZfGuB+YSYeZ4MX3ZUBr1SePRS67
         tGq06j8viZAzhfb0ocFIwbXCPZIsUzBkmfgz0=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: <BANLkTik3Js=6rz-RjVrwZ3J0esXKe_xGoQ@mail.gmail.com>
References: <BANLkTimDPAvCU71Y_9hFZ1mTP8bQzwuRag@mail.gmail.com>
	<BANLkTikXU9FBOxY5z8UJGmbrq-oZH8+pRA@mail.gmail.com>
	<BANLkTinQ7KZaCXH1ANmir0gmgeOOY5oQxw@mail.gmail.com>
	<BANLkTimhJVni6RwHnt7bd2aSnXYbVE=36A@mail.gmail.com>
	<BANLkTik3Js=6rz-RjVrwZ3J0esXKe_xGoQ@mail.gmail.com>
Date: Fri, 13 May 2011 21:28:17 -0700
Message-ID: <BANLkTimy4JnQhy=nYxRgBGnCAn+Lf2qfZA@mail.gmail.com>
Subject: Re: region goes missing on rs (may be during reassignment)
From: Stack <stack@duboce.net>
To: user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, May 13, 2011 at 3:01 PM, Raghu Angadi <angadi@gmail.com> wrote:
> Thanks Stack. greatly appreciate the help.
>

No problem.

> hbase.regionserver.handler.count is set to 30.
> we have not set hbase.master.assignment.timeoutmonitor.timeout. will sure=
ly
> increase to 180 seconds as HBASE-3846 does.
>

You might want to go to 0.90.3 altogether.  Is that a pain for you?


> The load on the cluster is low to moderate and HBase holds up pretty well=
.
> Most of the load consists of hourly random writes to the table and
> sequential scans from MR jobs.
>

Thanks boss.


> I will send another email with locations to full master logs.
> There are many "Regions in transition timed out" messages for this region
> and many others spread over time.
>

Grand.

I can come over any time or you should drop by our place.  Its just a
few blocks away and we can munch on lunch while we dig in your logs.

St.Ack

> Raghu.
>
> On Fri, May 13, 2011 at 11:33 AM, Stack <stack@duboce.net> wrote:
>
>> I see that we are timing out region assignment then assigning
>> elsewhere, but the region opened anyway on first server (What do you
>> have hbase.regionserver.handler.count set to? =A0The default is 10 which
>> could mean a bunch of requests hanging out in the rpc queue before
>> getting into the server to be processed). =A0One thing you could do is
>> up your region in transition timeout. =A0Default is 30 seconds which if
>> there is a bunch of churn may not be enough time for region assignment
>> to complete -- was there churn at this time? (We up the default
>> timeout in 0.90.3, see =A0'HBASE-3846 =A0Set RIT timeout higher').
>>
>> See below for more.
>>
>> On Fri, May 13, 2011 at 8:19 AM, Raghu Angadi <rangadi@apache.org> wrote=
:
>> ...
>> >> > 2011-05-12 12:05:20,987 DEBUG
>> >> > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Ope=
ned
>> >> > users,61364002,1297594642368.a0bf035ac417cdd0697464f1c48f387f.
>>
>> The region opened successfully.
>>
>> But looking at the master log, 12 seconds earlier it says:
>>
>> >>>> 2011-05-12 12:05:08,122 INFO
>> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
>> timed out: =A0users,61364002,1297594642368.a0bf035ac417cdd0697464f1c48f3=
87f.
>> state=3D3DOPENING, ts=3D3D1305201871850 2011-05-12 12:05:08,122 INFO
>> org.apache.hadoop.hbase.master.AssignmentManager: Region has been OPENIN=
G
>> for too long, reassigning
>>
>>
>> .... and then forces it reasssigned elsewhere (Your log from master
>> stops at this point. =A0I'd be interested in seeing more. =A0Send it to =
me
>> offline?).
>>
>> Thanks Raghu,
>> St.Ack
>>
>