Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of jdcryans@gmail.com designates
 209.85.213.169 as permitted sender)
MIME-Version: 1.0
Sender: jdcryans@gmail.com
In-Reply-To: 
 <E83D30E8F408F94A96F992785FC29D8206339670@s2k3mntaexc1.mentacapital.local>
References: 
 <CAPQV63Xfu4d9d1gno4D-PX_Wyo_w72L7R9OtmbYq+RC7LfC0Xw@mail.gmail.com>
	<E83D30E8F408F94A96F992785FC29D82063395D6@s2k3mntaexc1.mentacapital.local>
	<CADcMMgG82WeEuQQhcvGcsme9vOHb4=ukyLzf86m+e8Dxw8dmng@mail.gmail.com>
	<E83D30E8F408F94A96F992785FC29D8206339613@s2k3mntaexc1.mentacapital.local>
	<E83D30E8F408F94A96F992785FC29D8206339670@s2k3mntaexc1.mentacapital.local>
Date: Mon, 18 Jun 2012 15:17:04 -0700
Message-ID: 
 <CAGpTDNfWCY=5-cSxQkvLfsSnqB_mBjaxUHPTiuDQeviUVQAh6A@mail.gmail.com>
Subject: Re: RS unresponsive after series of deletes
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: user@hbase.apache.org
Cc: Development <Development@mentacapital.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Mass deleting in HBase is equivalent to mass inserting, it's just that
the former doesn't have to write values out (just keys). Almost
everything that applies to batch insert tunings applies to batch
deleting.

Now the error you get comes from this:
https://issues.apache.org/jira/browse/HBASE-5190

What it means is that you have 1GB worth of _deletes_ sitting the
region server call queue. That's way too much, something's wrong, and
it doesn't seem to be making progress.

Like Stack said in his reply, have you thread dumped the slow region
servers when this happens? It would also help to see the log during
that time. Try to capture a good chunk of it and post it like you did
on pastebin.

Thx,

J-D

On Mon, Jun 18, 2012 at 3:08 PM, Ted Tuttle <ted.tuttle@mentacapital.com> w=
rote:
> We had another of these delete-related RS hang ups. =A0This time we are
> getting a different error on the client:
>
> java.io.IOException: Call queue is full, is
> ipc.server.max.callqueue.size too small?
>
> full stack here: http://pastebin.com/uq68Mvhm
>
> Looking at the RS log, it appears the RS was working on the batch delete
> for about 1hr. There are no errors in the RS log during this time.
> There are several "responseTooSlow" messages. =A0Based on processingtimem=
s
> values they all lead back to our big batch delete.
>
> Any theories on how a big batch of deletes could cause a RS to go
> unresponsive?
>
>
>