hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin GEORGES <ke...@d33d33.fr>
Subject Asked to modify this region's memstoreSize to a negative value which is incorrect
Date Tue, 15 May 2018 11:10:14 GMT
Hello,

We find region server abort with the following exception:

2018-05-15 08:23:23,920 ERROR [RpcServer.default.FPBQ.Fifo.handler=27,queue=7,port=16020]
regionserver.HRegion: Asked to modify this region's (continuum,R\x0C\xF6\xF2\xBD\xD4L"\xB5\xFC\
xC6b\x8D\xD7\xC8x$\x7F\xFA\x9F\xA4\x92,1524491062878.117704beb050dcd3920335b4b290a898.) memstoreSize
to a negative value which is incorrect. Current memstoreSize=-1533230, delta=480
java.lang.Exception
        at org.apache.hadoop.hbase.regionserver.HRegion.addAndGetGlobalMemstoreSize(HRegion.java:1205)
        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3534)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3102)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3044)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:894)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:822)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2376)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36621)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2352)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
2018-05-15 08:23:23,922 FATAL [regionserver/dn-35.hadoop.B.GRA.infra.metrics.ovh.net/10.0.0.35:16020-splits-1525859600420]
regionserver.HRegionServer: ABORTING region server dn-35.hadoo
p.b.gra.infra.metrics.ovh.net,16020,1525859460041: Assertion failed while closing store continuum,RH\xFD\xA6\x88\xD7\xFB5\xBBq\xD9\xE8|\xF2I_r\x7F\xFA\xB0Y\x88,1509095154547.51aea042f53
655350c0d098fd378ab9b. v. flushableSize expected=0, actual= 23429. Current memstoreSize=-34925.
Maybe a coprocessor operation failed and left the memstore in a partially updated state.
2018-05-15 08:23:23,922 FATAL [regionserver/dn-35.hadoop.B.GRA.infra.metrics.ovh.net/10.0.0.35:16020-splits-1525859600420]
regionserver.HRegionServer: RegionServer abort: loaded coproce
ssors are: [org.apache.hadoop.hbase.coprocessor.example.BulkDeleteEndpoint


The error about memstoreSize becoming negative appear at a steady rate before abort (hundreds/sec)

Any ideas?

Thanks,

Kevin
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message