Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2BEA6CC8F for ; Thu, 20 Nov 2014 12:40:04 +0000 (UTC) Received: (qmail 61355 invoked by uid 500); 20 Nov 2014 12:40:01 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 61285 invoked by uid 500); 20 Nov 2014 12:40:01 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 61273 invoked by uid 99); 20 Nov 2014 12:40:01 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Nov 2014 12:40:01 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ramkrishna.s.vasudevan@gmail.com designates 209.85.223.172 as permitted sender) Received: from [209.85.223.172] (HELO mail-ie0-f172.google.com) (209.85.223.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Nov 2014 12:39:57 +0000 Received: by mail-ie0-f172.google.com with SMTP id ar1so2611695iec.31 for ; Thu, 20 Nov 2014 04:38:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=CmldSnKtEaRNbuOGlt4qgic1yZEaujX5OcUzkMwNmdw=; b=RbawutyG0W4jxN/r7/wJnUNsmKusm/FXHfKnIOMslwSggygr+hpri21oO95JfO74Ua X/hddTP7lQkSZsFeknR2vApFP6uaItQpYY5ly7EqSQESLnGGQu4vLhAbexbSCCPaZtV5 Q9Dngip4VmtdJEdK3ddg1k7Y24Z82e93Xo/cRcCU8yqULkOpr0FWglKbrwkVOvDDayfK Wkpsmik/4fiyDWV/Ilf4t2Qjmt1IM1xZ35nBcRyCB1Xs2ZASO3uNY1rLFXyBh1mRFLK5 1AqAHo/+JrHmtc/I5HZrXMSCmkUQH4VFaRRp17Y/WA/Kg1yZBwZ3y4juiUfzJ+JyrAKg fNww== MIME-Version: 1.0 X-Received: by 10.42.136.133 with SMTP id u5mr7717557ict.33.1416487086006; Thu, 20 Nov 2014 04:38:06 -0800 (PST) Received: by 10.107.47.218 with HTTP; Thu, 20 Nov 2014 04:38:05 -0800 (PST) In-Reply-To: <7AD9FC8F-0159-45C0-A7C6-13D0A1454936@gmail.com> References: <7AD9FC8F-0159-45C0-A7C6-13D0A1454936@gmail.com> Date: Thu, 20 Nov 2014 18:08:05 +0530 Message-ID: Subject: Re: YCSB load failed because hbase region too busy From: ramkrishna vasudevan To: "user@hbase.apache.org" Content-Type: multipart/related; boundary=90e6ba6e89b2976c610508499959 X-Virus-Checked: Checked by ClamAV on apache.org --90e6ba6e89b2976c610508499959 Content-Type: multipart/alternative; boundary=90e6ba6e89b2976c5d0508499958 --90e6ba6e89b2976c5d0508499958 Content-Type: text/plain; charset=UTF-8 Check if the writes are going to that particular region and its rate is too high. Ensure that the data gets distributed among all regions. What is the memstore size? If the rate of writes is very high then the flushing will get queued and until the memstore gets flushed such that it goes down the global upper limit writes will be blocked. I don't have the code now to see the exact config related to memstore. Regards Ram On Thu, Nov 20, 2014 at 4:50 PM, louis.hust wrote: > hi all, > > I build an HBASE test environment, with three PC server, with CHD 5.1.0 > > pc1 pc2 pc3 > > pc1 and pc2 as HMASTER and hadoop namenode > pc3 as RegionServer and datanode > > Then I create user as following: > > create 'usertable', 'family', {SPLITS => (1..100).map {|i| "user#{1000+i*(9999-1000)/100}"} } > > Using YCSB for load data as following: > > ./bin/ycsb load hbase -P workloads/workloadc -p columnfamily=family > -p recordcount=1000000000 -p threadcount=32 -s > result/workloadc > > > But when after a while, the ycsb return with following error: > > 14/11/20 12:23:44 INFO client.AsyncProcess: #15, table=usertable, > attempt=35/35 failed 715 ops, last exception: > org.apache.hadoop.hbase.RegionTooBusyException: > org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, > regionName=usertable,user9099,1416453519676.2552d36eb407a8af12d2b58c973d68a9., > server=l-hbase10.dba.cn1 ,60020,1416451280772, > memstoreSize=536897120, blockingMemStoreSize=536870912 > at > org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2822) > at > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2234) > at > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2201) > at > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2205) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4253) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3469) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3359) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29503) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) > at > org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160) > at > org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38) > at > org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110) > at java.lang.Thread.run(Thread.java:744) > on l-hbase10.dba.cn1 ,60020,1416451280772, > tracking started Thu Nov 20 12:15:07 CST 2014, retrying after 20051 ms, > replay 715 ops. > > > It seems the user9099 region is too busy, so I lookup the memstore metrics > in web: > > As you see, the user9099 is bigger than other region, I think it is > flushing, but after a while, it does not change to a small size and YCSB > quit finally. > > But when i change the concurrency threads to 4, all is right. I want to > know why? > > Any idea will be appreciated. > > > --90e6ba6e89b2976c5d0508499958 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Check if the writes are going to that particular region an= d its rate is too high.=C2=A0 Ensure that the data gets distributed among a= ll regions.
What is the memstore size? =C2=A0

= If the rate of writes is very high then the flushing will get queued and un= til the memstore gets flushed such that it goes down the global upper limit= writes will be blocked.

I don't have the code= now to see the exact config related to memstore.

= Regards
Ram

On Thu, Nov 20, 2014 at 4:50 PM, louis.hust <louis.hus= t@gmail.com> wrote:
hi all,

I build an HBA= SE test environment, with three PC server, with CHD 5.1.0

pc1 pc2 pc3

pc1 and pc2 as HMASTER and had= oop namenode
pc3 as RegionServer and datanode
Then I create user as following:
create 'usertable', 'family=
9;, {SPLITS =3D> (1..100).map {|i| "user=
#{1000+i*(9999-1000)/100}"} }
Using YCSB for load data as following:

./bin/ycsb=C2=A0 load=C2=A0 = hbase=C2=A0=C2=A0 -P workloads/workloadc =C2=A0-p columnfamily=3Dfamily -p = recordcount=3D1000000000 =C2=A0 -p threadcount=3D32 =C2=A0-s=C2=A0 > res= ult/workloadc


But when after a while, the ycsb ret= urn with following error:

14/11/20 12:23:44 I= NFO client.AsyncProcess: #15, table=3Dusertable, attempt=3D35/35 failed 715= ops, last exception: org.apache.hadoop.hbase.RegionTooBusyException: org.a= pache.hadoop.hbase.RegionTooBusyException: Above memstore limit, regionName= =3Dusertable,user9099,1416453519676.2552d36e= b407a8af12d2b58c973d68a9., server=3Dl-hbase10.dba.cn1,60020,1416451280772, memstoreSize=3D5= 36897120, blockingMemStoreSize=3D536870912
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegi= on.java:2822)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hb= ase.regionserver.HRegion.batchMutate(HRegion.java:2234)
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hbase.regionserver.HRegion.batchM= utate(HRegion.java:2201)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apach= e.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2205)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hbase.regionserver.HRegi= onServer.doBatchOp(HRegionServer.java:4253)
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicReg= ionMutation(HRegionServer.java:3469)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.j= ava:3359)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hbase.= protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientPr= otos.java:29503)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop= .hbase.ipc.RpcServer.call(RpcServer.java:2012)
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98= )
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hbase.ipc.Simp= leRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.acce= ss$000(SimpleRpcScheduler.java:38)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at= org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.ja= va:110)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.lang.Thread.run(Threa= d.java:744)
=C2=A0on l-hbase10.dba.cn1,60020,1416451280772, tracking started Thu = Nov 20 12:15:07 CST 2014, retrying after 20051 ms, replay 715 ops.


It seems the user9099 region is= too busy, so I lookup the memstore metrics in web:

As you see, the user9099 is bigger= than other region, I think it is flushing, but after a while, it does not = change to a small size and YCSB quit finally.

But = when i change the concurrency threads to 4, all is right. I want to know wh= y?

Any idea will be appreciated.



--90e6ba6e89b2976c5d0508499958-- --90e6ba6e89b2976c610508499959--