hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Walter King <wal...@adroll.com>
Subject IPC Queue Size
Date Thu, 07 Aug 2014 15:42:36 GMT
Hi, we're running 0.98.3 hbase, and it seems like the ipc queueSize is not
always being reduced.  Graphing the metric from jmx, at the start it stays
at zero and spikes up and returns back to zero.  At a certain point it
stops returning to zero, and returns to something slightly higher.
 Eventually we ran into ipc queue size full messages being returned to
clients trying large batch puts, as it approaches a gigabyte.  Some of our
servers, when idle and not processing any requests, say the queue size is
hundreds of megabytes.  I have a heap dump of one claiming to have over a
hundred mb in the queue, and as far as i can tell the call objects are
maybe a single mb in heap space.  We do have a sizable number of scans that
timeout without closing, lots of increments, and a few puts.

Graphs: http://imgur.com/9qDHgRd  The middle graph is what I'd expect to
see, the bottom graph seems to stop working around 15:30, and the top graph
never drops below 10MB.  Read/write request counts are relatively balanced
amongst the servers.

Is this normal?

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message