hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "聪聪" <175998...@qq.com>
Subject 回复: regionserver.HRegion: Failed getting lock in batch put
Date Tue, 21 Jul 2015 15:38:14 GMT
A regionserver today again this error!
any idea?I hope to help.




------------------ 原始邮件 ------------------
发件人: "蒲聪-北京";<175998806@qq.com>;
发送时间: 2015年7月12日(星期天) 晚上7:15
收件人: "user"<user@hbase.apache.org>; 

主题: 回复: regionserver.HRegion: Failed getting lock in batch put



hi,stack:


Quite simply,Can you explain why all regionserver has been reported to this waring.Until I
restarted the regionserver on the l-hbase1.dba.cn1 to resume.


Regionserver on the log is as follows:
2015-07-10 16:34:35,672 WARN [B.defaultRpcServer.handler=18,queue=0,port=60020] regionserver.HRegion:
Failed getting lock in batch put, row=LEDHKTABPNCB4RU004001150711999999
org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out of range for
row lock on HRegion atpco:atp_fare,M,1435898852855.e2f9f303a434f3f27dfc839c01c3d9eb., startKey='M',
getEndKey()='N', row='LEDHKTABPNCB4RU004001150711999999'
at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:3456)
at org.apache.hadoop.hbase.regionserver.HRegion.getRowLock(HRegion.java:3474)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2394)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2261)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2213)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2217)
at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4386)
at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3588)
at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3477)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:744)




------------------ 原始邮件 ------------------
发件人: "Stack";<stack@duboce.net>;
发送时间: 2015年7月11日(星期六) 凌晨0:55
收件人: "Hbase-User"<user@hbase.apache.org>; 

主题: Re: regionserver.HRegion: Failed getting lock in batch put



On Fri, Jul 10, 2015 at 4:35 AM, 聪聪 <175998806@qq.com> wrote:

> hi,all:I use the HBase version is hbase-0.98.6-cdh5.2.0.
>
> This afternoon,HBase access failure,use command 'hbase hbck' Check data
> inconsistencies。
> Hmaster on the log is as follows:
> 2015-07-10 16:24:41,285 INFO
> [FifoRpcScheduler.handler1-thread-43]master.HMaster: Client=hadoop//
> 192.168.28.12 assign
> dama:lowprice_intl_roundtrip_20150311,002687951001426005490000,1426090250918.01b4e069174f6bb786b4e0632f4e4740.
> 2015-07-10 16:24:41,285 DEBUG
> [FifoRpcScheduler.handler1-thread-43]master.AssignmentManager: Force region
> state offline
> {01b4e069174f6bb786b4e0632f4e4740 state=OPEN, ts=1436433547853, server=
> l-hbase1.dba.cn1.qunar.com,60020,1436433434134}
> 2015-07-10 16:24:56,285 INFO
> [FifoRpcScheduler.handler1-thread-43]master.AssignmentManager: Server
> l-hbase1.dba.cn1.qunar.com,60020,1436433434134 returned
> java.net.ConnectException: Connection timed out for
> dama:lowprice_intl_roundtrip_20150311,002687951001426005490000,1426090250918.01b4e069174f6bb786b4e0632f4e4740.,
> try=1 of 10
> java.net.ConnectException: Connection timed out
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868)
> at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543)
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976)
> at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1717)
> at
> org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:730)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1705)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1822)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1453)
> at org.apache.hadoop.hbase.master.HMaster.assignRegion(HMaster.java:2500)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42259)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> at
> org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
> Regionserver on the log is as follows:
> 2015-07-10 16:34:35,672
> WARN[B.defaultRpcServer.handler=18,queue=0,port=60020]regionserver.HRegion:
> Failed getting lock in batch put, row=LEDHKTABPNCB4RU004001150711999999
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row
> out of range for row lock on HRegion
> atpco:atp_fare,M,1435898852855.e2f9f303a434f3f27dfc839c01c3d9eb.,
> startKey='M', getEndKey()='N', row='LEDHKTABPNCB4RU004001150711999999'
> at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:3456)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLock(HRegion.java:3474)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2394)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2261)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2213)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2217)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4386)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3588)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3477)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> at java.lang.Thread.run(Thread.java:744)
>
> What's the problem?




You are pasting two seemingly unrelated problems -- master unable to
connect to a server (is that server up? did it come back? -- and the client
having trouble finding the right region to insert into: did hbck show
inconsistencies?

St.Ack
Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message