hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Viral Bajaria <viral.baja...@gmail.com>
Subject debugging responseTooSlow
Date Sat, 16 Feb 2013 03:14:46 GMT
Hi,

(using hbase 0.94.4 and hadoop 1.0.4)

I have been seeing a lot of the following WARN in my logs:

2013-02-16 02:37:11,409 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=25.18 MB,
free=2.97 GB, max=3 GB, blocks=1, accesses=52, hits=51, hitRatio=98.07%, ,
cachingAccesses=52, cachingHits=51, cachingHitsRatio=98.07%, , evictions=0,
evicted=0, evictedPerRun=NaN
2013-02-16 02:37:33,368 WARN org.apache.hadoop.ipc.HBaseServer:
(responseTooSlow):
{"processingtimems":97509,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@1c3308bd),
rpc version=1, client version=29, methodsFingerPrint=-1368823753","client":"
10.149.10.10:41009
","starttimems":1360982155855,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
2013-02-16 02:38:37,377 WARN org.apache.hadoop.ipc.HBaseServer:
(responseTooSlow):
{"processingtimems":97191,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3eafc7ae),
rpc version=1, client version=29, methodsFingerPrint=-1368823753","client":"
10.149.10.10:41014
","starttimems":1360982220183,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
2013-02-16 02:39:29,842 WARN org.apache.hadoop.ipc.HBaseServer:
(responseTooSlow):
{"processingtimems":85300,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3d615428),
rpc version=1, client version=29, methodsFingerPrint=-1368823753","client":"
10.149.10.10:41017
","starttimems":1360982284538,"queuetimems":1,"class":"HRegionServer","responsesize":0,"method":"multi"}

It's strange because this is a new hbase setup with almost no traffic on
it. I am running a perf test and would not expect this to happen. The
regionservers have 12GB heap space and are only using 1GB when that error
happens. I just pushed close to 33K rows via a batch and I see the
responseTooSlow.

I enabled GC logging, but I don't see any GC lockups, and each GC attempt
is only taking a few 100 ms.

What else could be happening here, any pointers on debugging ? My setup is
1 Master running with 1 NN (on the same server) with 3 regionservers
running alongside the datanodes.

Thanks,
Viral

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message