From user-return-55135-archive-asf-public=cust-asf.ponee.io@hbase.apache.org Tue May 15 13:10:28 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id B091E180634 for ; Tue, 15 May 2018 13:10:27 +0200 (CEST) Received: (qmail 10763 invoked by uid 500); 15 May 2018 11:10:26 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 10748 invoked by uid 99); 15 May 2018 11:10:25 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 May 2018 11:10:25 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 078591805BE for ; Tue, 15 May 2018 11:10:25 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3 X-Spam-Level: *** X-Spam-Status: No, score=3 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_NONE=-0.0001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id Fww6LzyteUh8 for ; Tue, 15 May 2018 11:10:22 +0000 (UTC) Received: from smtp.smtpout.orange.fr (smtp06.smtpout.orange.fr [80.12.242.128]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 3E2E75F23C for ; Tue, 15 May 2018 11:10:21 +0000 (UTC) Received: from Kevins-MacBook-Pro.local.mail ([109.190.254.4]) by mwinf5d41 with ME id mbAF1x00306TypH03bAFcz; Tue, 15 May 2018 13:10:15 +0200 X-ME-Helo: Kevins-MacBook-Pro.local.mail X-ME-Auth: Z2Vvcmdlcy1rZXZpbkBvcmFuZ2UuZnI= X-ME-Date: Tue, 15 May 2018 13:10:15 +0200 X-ME-IP: 109.190.254.4 Date: Tue, 15 May 2018 13:10:14 +0200 From: Kevin GEORGES To: user@hbase.apache.org Message-ID: Subject: Asked to modify this region's memstoreSize to a negative value which is incorrect X-Mailer: Airmail (481) MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="5afac016_36fc63da_150" --5afac016_36fc63da_150 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hello, We find region server abort with the following exception: 2018-05-15 08:23:23,920 ERROR =5BRpcServer.default.=46PBQ.=46ifo.handler=3D= 27,queue=3D7,port=3D16020=5D regionserver.HRegion: Asked to modify this r= egion's (continuum,R=5Cx0C=5Cx=466=5Cx=462=5CxBD=5CxD4L=22=5CxB5=5Cx=46C=5C= xC6b=5Cx8D=5CxD7=5CxC8x=24=5Cx7=46=5Cx=46A=5Cx9=46=5CxA4=5Cx92,1524491062= 878.117704beb050dcd3920335b4b290a898.) memstoreSize to a negative value w= hich is incorrect. Current memstoreSize=3D-1533230, delta=3D480 java.lang.Exception =C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hbase.regionserver.HRegi= on.addAndGetGlobalMemstoreSize(HRegion.java:1205) =C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hbase.regionserver.HRegi= on.doMiniBatchMutation(HRegion.java:3534) =C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hbase.regionserver.HRegi= on.batchMutate(HRegion.java:3102) =C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hbase.regionserver.HRegi= on.batchMutate(HRegion.java:3044) =C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hbase.regionserver.RSRpc= Services.doBatchOp(RSRpcServices.java:894) =C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hbase.regionserver.RSRpc= Services.doNonAtomicRegionMutation(RSRpcServices.java:822) =C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hbase.regionserver.RSRpc= Services.multi(RSRpcServices.java:2376) =C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hbase.protobuf.generated= .ClientProtos=24ClientService=242.callBlockingMethod(ClientProtos.java:36= 621) =C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hbase.ipc.RpcServer.call= (RpcServer.java:2352) =C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hbase.ipc.CallRunner.run= (CallRunner.java:124) =C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hbase.ipc.RpcExecutor=24= Handler.run(RpcExecutor.java:297) =C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hbase.ipc.RpcExecutor=24= Handler.run(RpcExecutor.java:277) 2018-05-15 08:23:23,922 =46ATAL =5Bregionserver/dn-35.hadoop.B.GRA.infra.= metrics.ovh.net/10.0.0.35:16020-splits-1525859600420=5D regionserver.HReg= ionServer: ABORTING region server dn-35.hadoo p.b.gra.infra.metrics.ovh.net,16020,1525859460041: Assertion failed while= closing store continuum,RH=5Cx=46D=5CxA6=5Cx88=5CxD7=5Cx=46B5=5CxBBq=5Cx= D9=5CxE8=7C=5Cx=462I=5Fr=5Cx7=46=5Cx=46A=5CxB0Y=5Cx88,1509095154547.51aea= 042f53 655350c0d098fd378ab9b. v. flushableSize expected=3D0, actual=3D 23429. Cu= rrent memstoreSize=3D-34925. Maybe a coprocessor operation failed and lef= t the memstore in a partially updated state. 2018-05-15 08:23:23,922 =46ATAL =5Bregionserver/dn-35.hadoop.B.GRA.infra.= metrics.ovh.net/10.0.0.35:16020-splits-1525859600420=5D regionserver.HReg= ionServer: RegionServer abort: loaded coproce ssors are: =5Borg.apache.hadoop.hbase.coprocessor.example.BulkDeleteEndpo= int The error about memstoreSize becoming negative appear at a steady rate be= fore abort (hundreds/sec) Any ideas=3F Thanks, Kevin --5afac016_36fc63da_150--