hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com>
Subject Re: YCSB load failed because hbase region too busy
Date Thu, 20 Nov 2014 12:38:05 GMT
Check if the writes are going to that particular region and its rate is too
high.  Ensure that the data gets distributed among all regions.
What is the memstore size?

If the rate of writes is very high then the flushing will get queued and
until the memstore gets flushed such that it goes down the global upper
limit writes will be blocked.

I don't have the code now to see the exact config related to memstore.

Regards
Ram

On Thu, Nov 20, 2014 at 4:50 PM, louis.hust <louis.hust@gmail.com> wrote:

> hi all,
>
> I build an HBASE test environment, with three PC server, with CHD 5.1.0
>
> pc1 pc2 pc3
>
> pc1 and pc2 as HMASTER and hadoop namenode
> pc3 as RegionServer and datanode
>
> Then I create user as following:
>
> create 'usertable', 'family', {SPLITS => (1..100).map {|i| "user#{1000+i*(9999-1000)/100}"}
}
>
> Using YCSB for load data as following:
>
> ./bin/ycsb  load  hbase   -P workloads/workloadc  -p columnfamily=family
> -p recordcount=1000000000   -p threadcount=32  -s  > result/workloadc
>
>
> But when after a while, the ycsb return with following error:
>
> 14/11/20 12:23:44 INFO client.AsyncProcess: #15, table=usertable,
> attempt=35/35 failed 715 ops, last exception:
> org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=usertable,user9099,1416453519676.2552d36eb407a8af12d2b58c973d68a9.,
> server=l-hbase10.dba.cn1 <http://dba.cn1.qunar.com>,60020,1416451280772,
> memstoreSize=536897120, blockingMemStoreSize=536870912
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2822)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2234)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2201)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2205)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4253)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3469)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3359)
>         at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29503)
>         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
>         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
>         at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
>         at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
>         at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
>         at java.lang.Thread.run(Thread.java:744)
>  on l-hbase10.dba.cn1 <http://dba.cn1.qunar.com>,60020,1416451280772,
> tracking started Thu Nov 20 12:15:07 CST 2014, retrying after 20051 ms,
> replay 715 ops.
>
>
> It seems the user9099 region is too busy, so I lookup the memstore metrics
> in web:
>
> As you see, the user9099 is bigger than other region, I think it is
> flushing, but after a while, it does not change to a small size and YCSB
> quit finally.
>
> But when i change the concurrency threads to 4, all is right. I want to
> know why?
>
> Any idea will be appreciated.
>
>
>

Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message