Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates
 209.85.161.169 as permitted sender)
Subject: Re: NPE while obtaining row lock
References: <FA0733A9-183F-4CE5-9525-788874F40706@langisch.ch>
 <CADcMMgFeO2FAdS1sOrzc=7ho02qmv2B51_e5vqkdvG0RQOErdQ@mail.gmail.com>
 <DF284F4C-A6FB-4DFE-8FEE-2494F85BEEFE@langisch.ch>
 <CADcMMgGyaq82gjvCCfrY6fYsnCx_qG=390XC10Z4U+9hfqrYBQ@mail.gmail.com>
 <B4E58B8F-AB76-4785-8936-6C5081926384@langisch.ch>
 <47BB1327-B73F-44E9-8472-888FCF8F3333@langisch.ch>
From: yuzhihong@gmail.com
Content-Type: text/plain;
	charset=us-ascii
In-Reply-To: <47BB1327-B73F-44E9-8472-888FCF8F3333@langisch.ch>
Message-Id: <4D4979B8-F27F-4E50-9471-32E99649AF76@gmail.com>
Date: Tue, 10 Jan 2012 08:04:46 -0800
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (1.0)

Thanks for the analysis.=20

Do you mind opening a Jira ?


On Jan 10, 2012, at 7:51 AM, Yves Langisch <yves@langisch.ch> wrote:

> Still happens with HBase 0.90.5/Hadoop 1.0.0. But I think I have some more=
 insights on this topic. Following an up to date stack trace:
>=20
> java.lang.NullPointerException
>        at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowa=
bleToIOE(HRegionServer.java:986)
>        at org.apache.hadoop.hbase.regionserver.HRegionServer.lockRow(HRegi=
onServer.java:2008)
>        at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod=
AccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:5=
70)
>        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.=
java:1039)
> Caused by: java.lang.NullPointerException
>        at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.jav=
a:881)
>        at org.apache.hadoop.hbase.regionserver.HRegionServer.addRowLock(HR=
egionServer.java:2018)
>        at org.apache.hadoop.hbase.regionserver.HRegionServer.lockRow(HRegi=
onServer.java:2004)
>        ... 5 more
>=20
> After checking the source code I've noticed that the value which is going t=
o be put into the HashMap can be null in the case where the waitForLock flag=
 is true or the rowLockWaitDuration is expired (HRegion#internalObtainRowLoc=
k, line 2111ff). The latter I think happens in our case as we have heavy loa=
d hitting the server.
>=20
> IMHO this case should be handled somehow and must not lead to a NPE.
>=20
> -
> Yves
>=20
> On Dec 30, 2011, at 12:12 PM, Yves Langisch wrote:
>=20
>> Still happens but before I'm going to add some debugging information I'll=
 try to deploy the new version 0.90.5.
>>=20
>> -
>> Yves
>>=20
>> On Dec 18, 2011, at 12:08 AM, Stack wrote:
>>=20
>>> On Fri, Dec 16, 2011 at 8:20 AM, Yves Langisch <yves@langisch.ch> wrote:=

>>>> I'm using the async hbase client (1.0) and there is no way to choose a l=
ockId on my own:
>>>>=20
>>>> <snippet>
>>>> return database.client().lockRow(
>>>>                  new RowLockRequest(TableManager.ID_TABLE_NAME, MAXID_R=
OW)).join();
>>>>=20
>>>> </snippet>
>>>>=20
>>>> Any ideas what else could be wrong here?
>>>>=20
>>>=20
>>> Looking at the code on regionserver side,
>>> http://svn.apache.org/viewvc/hbase/tags/0.90.4/src/main/java/org/apache/=
hadoop/hbase/regionserver/HRegionServer.java?view=3Dmarkup,
>>> down around line 1994, its unlikely the region is null since we should
>>> throw NotServingRegionException if can't find region (and we check for
>>> null region name a few lines up) so maybe its something in the way we
>>> do obtainRowLock on line 1995?
>>>=20
>>> Any chance of your instrumenting the regionserver?  Adding a bit of
>>> debugging and deploying the debugging regionserver?
>>>=20
>>> My guess is we haven't seen this before because not many use rowlocks
>>> (rowlocks if long-lived and lots of contending clients could freeze
>>> you out of the server; each client blocked waiting on rowlock to clear
>>> occupies a handler of which there are a bounded number).
>>>=20
>>> St.Ack
>>>=20
>>=20
>=20