Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CABWr33-zpQ3PYiaoggf42Z1_iyfnit2VqYUpZk30wGAchvhm2Q@mail.gmail.com>
References: 
 <CAAknh07DresvjSOgj4+O4s-sSpn58WfqYuicH0h8_64CF5tsaw@mail.gmail.com>
	<CABWr338DaQExTu725XYfnYSM_Wz1KtjxtpaoW08dM2jcw0m3CQ@mail.gmail.com>
	<CABWr338+bxULiaFy3TXdECEcsyo-z3_ZVsEEwhZ_91ASq19G_Q@mail.gmail.com>
	<CAAknh04rfwb2b2pZrN1jNmVviAjy8U9b4qW+iy-g26OG=j5Y+g@mail.gmail.com>
	<CABWr33-zpQ3PYiaoggf42Z1_iyfnit2VqYUpZk30wGAchvhm2Q@mail.gmail.com>
Date: Thu, 23 Jul 2015 15:04:09 +0800
Message-ID: 
 <CAAknh07w-i8hewBMnsCfEwwAB6rtnRnMKGQn=D7mcJQurOgrkQ@mail.gmail.com>
Subject: Re: Dead loop for batch put when get WrongRegionException
From: Louis Hust <louis.hust@gmail.com>
To: Victor Xu <victor.unique@gmail.com>
Cc: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=e89a8ff1cf32732efa051b857e63

--e89a8ff1cf32732efa051b857e63
Content-Type: text/plain; charset=UTF-8

My hbase version is 0.98.6,  i will try update client to 0.98.14 and keep
server at 0.98.6,
Thanks very much!

2015-07-23 15:00 GMT+08:00 Victor Xu <victor.unique@gmail.com>:

> Client-side dead loop can cause sending wrong read/write requests to the
> region servers, and you've got exactly the same log output as myself when
> the the bug happens. However, this only occurs when you are using 0.98.X
> version. 1.0 and above do not have this problem.
>
> On Thu, Jul 23, 2015 at 2:55 PM Louis Hust <louis.hust@gmail.com> wrote:
>
>> It seems that the HBASE-13896
>> <https://issues.apache.org/jira/browse/HBASE-13896>  is client-side dead
>> loop,
>> but my problem is the regionserver-side dead lock for get row lock,
>>
>> 2015-07-23 11:23 GMT+08:00 Victor Xu <victor.unique@gmail.com>:
>>
>>> FYI
>>>
>>> ---------- Forwarded message ---------
>>> From: Victor Xu <victor.unique@gmail.com>
>>> Date: Thu, Jul 23, 2015 at 11:22 AM
>>> Subject: Re: Dead loop for batch put when get WrongRegionException
>>> To: user@hbase.apache.org <user@hbase.apache.org>
>>>
>>>
>>> Any chance that this would be your problem?
>>> https://issues.apache.org/jira/browse/HBASE-13896
>>>
>>> On Thu, Jul 23, 2015 at 11:17 AM Louis Hust <louis.hust@gmail.com>
>>> wrote:
>>>
>>>> Hi ,all
>>>>
>>>> We are using batch put to insert rows, and sometimes get the following
>>>> WARN
>>>> in the region server log:
>>>>
>>>> <log>
>>>> 2015-07-23 10:08:49,684 WARN
>>>>  [B.defaultRpcServer.handler=5,queue=5,port=60020] regionserver.HRegion:
>>>> Failed getting lock in batch put, row=BHXYHZFIHHR3ECON101002150723999999
>>>> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row
>>>> out of range for row lock on HRegion
>>>> atpco:ttf_fare,C,1437145538123.9c2b8cb846b318045f2ad6b5c87fef21.,
>>>> startKey='C', getEndKey()='D', row='BHXYHZFIHHR3ECON101002150723999999'
>>>> at
>>>> org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:3456)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.regionserver.HRegion.getRowLock(HRegion.java:3474)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2394)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2261)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2213)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2217)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4386)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3588)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3477)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593)
>>>> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
>>>> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
>>>> </log>
>>>>
>>>> And the WARN message is logged no-stop. I think the batch put dived into
>>>> the dead loop.
>>>>
>>>> And i look up into the source code, and find the batch put will never
>>>> stop
>>>> if got WrongRegionException for some row.
>>>>
>>>> Any body know how to avoid this situation?
>>>>
>>>> Any idea will be appreciated!
>>>>
>>>
>>

--e89a8ff1cf32732efa051b857e63--