hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: RS unresponsive after series of deletes
Date Mon, 18 Jun 2012 22:17:04 GMT
Mass deleting in HBase is equivalent to mass inserting, it's just that
the former doesn't have to write values out (just keys). Almost
everything that applies to batch insert tunings applies to batch

Now the error you get comes from this:

What it means is that you have 1GB worth of _deletes_ sitting the
region server call queue. That's way too much, something's wrong, and
it doesn't seem to be making progress.

Like Stack said in his reply, have you thread dumped the slow region
servers when this happens? It would also help to see the log during
that time. Try to capture a good chunk of it and post it like you did
on pastebin.



On Mon, Jun 18, 2012 at 3:08 PM, Ted Tuttle <ted.tuttle@mentacapital.com> wrote:
> We had another of these delete-related RS hang ups.  This time we are
> getting a different error on the client:
> java.io.IOException: Call queue is full, is
> ipc.server.max.callqueue.size too small?
> full stack here: http://pastebin.com/uq68Mvhm
> Looking at the RS log, it appears the RS was working on the batch delete
> for about 1hr. There are no errors in the RS log during this time.
> There are several "responseTooSlow" messages.  Based on processingtimems
> values they all lead back to our big batch delete.
> Any theories on how a big batch of deletes could cause a RS to go
> unresponsive?

View raw message