hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Biju N <bijuatapa...@gmail.com>
Subject Re: after server restart - getting exception - java.io.IOException: Timed out waiting for lock for row
Date Wed, 22 Jun 2016 22:38:41 GMT
Vishnu,
    Are you using "local index" on any of the tables? We have seen similar
issues while using "local index".

On Wed, Jun 22, 2016 at 12:25 PM, vishnu rao <jaihind213@gmail.com> wrote:

> the server dies when trying to take the thread dump.
>
> i believe i am experiencing this bug
>
> https://issues.apache.org/jira/browse/PHOENIX-2508
>
> On Wed, Jun 22, 2016 at 5:03 PM, Heng Chen <heng.chen.1986@gmail.com>
> wrote:
>
> > which thread hold the row lock? could you dump the jstack with 'jstack -l
> > pid' ?
> >
> > 2016-06-22 16:14 GMT+08:00 vishnu rao <jaihind213@gmail.com>:
> >
> > > hi Heng.
> > >
> > > 2016-06-22 08:13:42,256 WARN
> > > [B.defaultRpcServer.handler=32,queue=2,port=16020]
> regionserver.HRegion:
> > > Failed getting lock in batch put,
> > > row=\x01\xD6\xFD\xC9\xDC\xE4\x08\xC4\x0D\xBESM\xC2\x82\x14Z
> > >
> > > java.io.IOException: Timed out waiting for lock for row:
> > > \x01\xD6\xFD\xC9\xDC\xE4\x08\xC4\x0D\xBESM\xC2\x82\x14Z
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5051)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2944)
> > >
> > > at
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801)
> > >
> > > at
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> > >
> > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> > >
> > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> > >
> > > at
> > >
> >
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> > >
> > > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> > >
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > On Wed, Jun 22, 2016 at 3:50 PM, Heng Chen <heng.chen.1986@gmail.com>
> > > wrote:
> > >
> > > > Could you paste the whole jstack and relates rs log?   It seems row
> > write
> > > > lock was occupied by some thread.  Need more information to find it.
> > > >
> > > > 2016-06-22 13:48 GMT+08:00 vishnu rao <jaihind213@gmail.com>:
> > > >
> > > > > need some help. this has happened for 2 of my servers
> > > > > -------------
> > > > >
> > > > > *[B.defaultRpcServer.handler=2,queue=2,port=16020]
> > > regionserver.HRegion:
> > > > > Failed getting lock in batch put,
> > > > > row=a\xF7\x1D\xCBdR\xBC\xEC_\x18D>\xA2\xD0\x95\xFF*
> > > > >
> > > > > *java.io.IOException: Timed out waiting for lock for row:
> > > > > a\xF7\x1D\xCBdR\xBC\xEC_\x18D>\xA2\xD0\x95\xFF*
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5051)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2944)
> > > > >
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801)
> > > > >
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
> > > > >
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> > > > >
> > > > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> > > > >
> > > > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> > > > >
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> > > > >
> > > > > at
> > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> > > > >
> > > > > at java.lang.Thread.run(Thread.java:745)
> > > > >
> > > > > --
> > > > > with regards,
> > > > > ch Vishnu
> > > > > mash213.wordpress.com
> > > > > doodle-vishnu.blogspot.in
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > with regards,
> > > ch Vishnu
> > > mash213.wordpress.com
> > > doodle-vishnu.blogspot.in
> > >
> >
>
>
>
> --
> with regards,
> ch Vishnu
> mash213.wordpress.com
> doodle-vishnu.blogspot.in
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message