hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sumit Nigam <sumit_o...@yahoo.com.INVALID>
Subject Hbase throttling issues
Date Wed, 21 Sep 2016 17:18:34 GMT
Hi Hbase experts,

Our application is unable to scan or read from hbase tables, when throttling is set. We are
getting ThrottlingException every time. The error is seen more frequently when the number
of hbase pre splits is increased. The size tables for which this error is showing is empty
( during some runs it was very low in the order of few kbs). Have tried both rate limiters
already - average and fixed. Can't understand why read rate limit exceeds when there is hardly
any data in hbase. Anyone has faced this issue before?

Setup Details: 

Hbase version : 1.1.2 
Number of region servers :4 
Number of regions : 116 
HeapMemory for Region Server : 2GB 

Quotas set : 
TABLE => ns1:table1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
10G/sec, SCOPE => MACHINE 
TABLE => ns2:table2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
10G/sec, SCOPE => MACHINE 

Following is the error we faced. Pasting debug logs of regions server: 

2016-09-17 22:35:40,674 DEBUG [B.defaultRpcServer.handler=55,queue=1,port=58526] quotas.RegionServerQuotaManager:
Throttling exception for user=root table=ns1:table1 numWrites=0 numReads=0 numScans=1: read
size limit exceeded - wait 0.00sec 

2016-09-17 22:35:40,676 DEBUG [B.defaultRpcServer.handler=55,queue=1,port=58526] ipc.RpcServer:
B.defaultRpcServer.handler=55,queue=1,port=58526: callId: 52 service: ClientService methodName:
Scan size: 28 connection: 10.65.141.170:42806 

org.apache.hadoop.hbase.quotas.ThrottlingException: read size limit exceeded - wait 0.00sec 

at org.apache.hadoop.hbase.quotas.ThrottlingException.throwThrottlingException(ThrottlingException.java:107) 

at org.apache.hadoop.hbase.quotas.ThrottlingException.throwReadSizeExceeded(ThrottlingException.java:101) 

at org.apache.hadoop.hbase.quotas.TimeBasedLimiter.checkQuota(TimeBasedLimiter.java:139) 

at org.apache.hadoop.hbase.quotas.DefaultOperationQuota.checkQuota(DefaultOperationQuota.java:59) 

at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(RegionServerQuotaManager.java:180) 

at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(RegionServerQuotaManager.java:125) 

at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2265) 

at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) 

at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) 

at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) 

at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) 

at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) 

at java.lang.Thread.run(Thread.java:745)


Thanks,

Sumit
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message