hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Mor <amit.mor.m...@gmail.com>
Subject Re: client bursts followed by drops
Date Tue, 18 Jun 2013 23:09:50 GMT
Attachment of JMX requests metric showing bursts on RS


On Wed, Jun 19, 2013 at 12:48 AM, Amit Mor <amit.mor.mail@gmail.com> wrote:

> Hello,
>
> We use hbase 0.94.2 and we are seeing (mostly reading) latency issues, and
> with an interesting twist: client (100 threads) get stuck waiting on HBase,
> then stops sending RPC's and then seems to be freed. When freed, while
> trying to flush all request it had queued, it causes another vicious cycle
> of burst - drop - burst.
> I am seeing about 40K requests per seconds per RS.
> The cluster is mostly read, no compaction and split storms. Just bursts
> and bursts.
> We set 30 rpc handlers per RS (avg scan size is 5K) and they most of the
> time seem to be WAITING. IPC at DEBUG revealed (see below) that sometimes
> the queueTime is very big and sometimes the responseTime is very big, but
> at a very low percentage.
> The burst seem to be periodic and uncorrelated to GC activity or CPU (the
> burst appear the moment the RS is onlined and the heap is free)
> The row keys are murmur3 hashed, and I don't really see any hotspoting
>
> Any idea what might cause those bursts ?
>
> Thanks,
> Amit
>
>

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message