hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Esteban Gutierrez <este...@cloudera.com>
Subject Re: IPC Queue Size
Date Thu, 07 Aug 2014 17:46:24 GMT
Hello Walter,

Do you have the RS logs and the debug dump (rs:port/dump) from this RS that
you could share? The only reason I can think of that behavior is that this
Region Server might have struggled with single large call and the call size
was never decremented due an exception. By
default ipc.client.call.purge.timeout should have purged that call from the
queue unless ipc.client.call.purge.timeout is configured with a vey large
value.

--
Cloudera, Inc.



On Thu, Aug 7, 2014 at 8:42 AM, Walter King <walter@adroll.com> wrote:

> Hi, we're running 0.98.3 hbase, and it seems like the ipc queueSize is not
> always being reduced.  Graphing the metric from jmx, at the start it stays
> at zero and spikes up and returns back to zero.  At a certain point it
> stops returning to zero, and returns to something slightly higher.
>  Eventually we ran into ipc queue size full messages being returned to
> clients trying large batch puts, as it approaches a gigabyte.  Some of our
> servers, when idle and not processing any requests, say the queue size is
> hundreds of megabytes.  I have a heap dump of one claiming to have over a
> hundred mb in the queue, and as far as i can tell the call objects are
> maybe a single mb in heap space.  We do have a sizable number of scans that
> timeout without closing, lots of increments, and a few puts.
>
> Graphs: http://imgur.com/9qDHgRd  The middle graph is what I'd expect to
> see, the bottom graph seems to stop working around 15:30, and the top graph
> never drops below 10MB.  Read/write request counts are relatively balanced
> amongst the servers.
>
> Is this normal?
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message