Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: Tianying Chang <tichang@ebaysf.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Subject: NullPointerException while loading large amount of new rows into
 HBase, exception is thrown when trying to obtain lock for RowKey
Thread-Topic: NullPointerException while loading large amount of new rows
 into HBase, exception is thrown when trying to obtain lock for RowKey
Thread-Index: AQHOUldhl2eR57NR9Ui6dYaaQvkSUQ==
Date: Thu, 16 May 2013 17:12:48 +0000
Message-ID: 
 <A8C751F9C8B9D448AE82296D0BD035DCD9BB06@PHX-EXRDA-S31.corp.ebay.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: multipart/alternative;
	boundary="_000_A8C751F9C8B9D448AE82296D0BD035DCD9BB06PHXEXRDAS31corpeb_"
MIME-Version: 1.0

--_000_A8C751F9C8B9D448AE82296D0BD035DCD9BB06PHXEXRDAS31corpeb_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi,

When our customers(using TSDB) loads large amount of data into HBase, we sa=
w many NullPointerException in the RS logs as below. I checked the source c=
ode, it seems when trying to obtain the lock for a rowKey, if the entry for=
 that row already exists and the "waitfoBlock" is false (therefore it won't=
 retry, but just return a NULL value). I can see in doMiniBatchMutation(), =
the waitForBlock is set to be false. (most other places "waitForBlock" is a=
lways set to true).

This exception is thrown from function lockRow(), which has been deprecated=
. I am not sure why it is deprecated, and what is used to replace it. Is th=
is normal? which implies the HBase should not throw this misleading error m=
essage to log.  Or should the client call some other API?

Thanks
Tian-Ying

2013-05-14 12:45:30,911 DEBUG org.apache.hadoop.hbase.regionserver.HRegionS=
erver: Row lock -3430274391270203797 explicitly acquired by client
2013-05-14 12:45:30,911 WARN org.apache.hadoop.ipc.HBaseServer: (responseTo=
oSlow): {"processingtimems":29783,"call":"lockRow([B@339a6a5c, [B@5ebcd87b)=
, rpc version=3D1, client version=3D29, methodsFingerPrint=3D0","client":"1=
0.53.106.37:58892","starttimems":1368560701128,"queuetimems":847,"class":"H=
RegionServer","responsesize":0,"method":"lockRow"}
2013-05-14 12:46:00,911 ERROR org.apache.hadoop.hbase.regionserver.HRegionS=
erver: Error obtaining row lock (fsOk: true)
java.lang.NullPointerException
                at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHas=
hMap.java:881)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.addRo=
wLock(HRegionServer.java:2346)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.lockR=
ow(HRegionServer.java:2332)
                at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown So=
urce)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(Delegati=
ngMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.cal=
l(WritableRpcEngine.java:384)
                at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBas=
eServer.java:1336)
2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server =
Responder, call get([B@2166c821, {"timeRange":[0,9223372036854775807],"tota=
lColumns":1,"cacheBlocks":true,"families":{"id":["tagv"]},"maxVersions":1,"=
row":"slcsn-s00314.slc.ebay.com"}), rpc version=3D1, client version=3D29, m=
ethodsFingerPrint=3D0 from 10.53.106.37:58892: output error
2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server =
handler 3 on 60020 caught: java.io.IOException: Connection reset by peer
                at sun.nio.ch.FileDispatcher.write0(Native Method)
                at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:=
29)

--_000_A8C751F9C8B9D448AE82296D0BD035DCD9BB06PHXEXRDAS31corpeb_--