hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "louis.hust" <louis.h...@gmail.com>
Subject YCSB load failed because hbase region too busy
Date Thu, 20 Nov 2014 11:20:08 GMT
hi all,

I build an HBASE test environment, with three PC server, with CHD 5.1.0

pc1 pc2 pc3

pc1 and pc2 as HMASTER and hadoop namenode
pc3 as RegionServer and datanode

Then I create user as following:
create 'usertable', 'family', {SPLITS => (1..100).map {|i| "user#{1000+i*(9999-1000)/100}"}
}
Using YCSB for load data as following:

./bin/ycsb  load  hbase   -P workloads/workloadc  -p columnfamily=family -p recordcount=1000000000
  -p threadcount=32  -s  > result/workloadc


But when after a while, the ycsb return with following error:

14/11/20 12:23:44 INFO client.AsyncProcess: #15, table=usertable, attempt=35/35 failed 715
ops, last exception: org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException:
Above memstore limit, regionName=usertable,user9099,1416453519676.2552d36eb407a8af12d2b58c973d68a9.,
server=l-hbase10.dba.cn1,60020,1416451280772, memstoreSize=536897120, blockingMemStoreSize=536870912
        at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2822)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2234)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2201)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2205)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4253)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3469)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3359)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29503)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
        at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
        at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
        at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
        at java.lang.Thread.run(Thread.java:744)
 on l-hbase10.dba.cn1,60020,1416451280772, tracking started Thu Nov 20 12:15:07 CST 2014,
retrying after 20051 ms, replay 715 ops.


It seems the user9099 region is too busy, so I lookup the memstore metrics in web:


As you see, the user9099 is bigger than other region, I think it is flushing, but after a
while, it does not change to a small size and YCSB quit finally.

But when i change the concurrency threads to 4, all is right. I want to know why?

Any idea will be appreciated.



Mime
View raw message