hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Mor <amit.mor.m...@gmail.com>
Subject client bursts followed by drops
Date Tue, 18 Jun 2013 21:48:12 GMT
Hello,

We use hbase 0.94.2 and we are seeing (mostly reading) latency issues, and
with an interesting twist: client (100 threads) get stuck waiting on HBase,
then stops sending RPC's and then seems to be freed. When freed, while
trying to flush all request it had queued, it causes another vicious cycle
of burst - drop - burst.
I am seeing about 40K requests per seconds per RS.
The cluster is mostly read, no compaction and split storms. Just bursts and
bursts.
We set 30 rpc handlers per RS (avg scan size is 5K) and they most of the
time seem to be WAITING. IPC at DEBUG revealed (see below) that sometimes
the queueTime is very big and sometimes the responseTime is very big, but
at a very low percentage.
The burst seem to be periodic and uncorrelated to GC activity or CPU (the
burst appear the moment the RS is onlined and the heap is free)
The row keys are murmur3 hashed, and I don't really see any hotspoting

Any idea what might cause those bursts ?

Thanks,
Amit

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message