Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of tsunanet@gmail.com designates
 74.125.82.169 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=vyJgsbDk9g9wj5MwMnN2uMrrjGyCZg+kYLS9e8iJ3z1Zr8NcA6cRwCTbN6iD0p+Sei
         Edqe8uZcH8GiMaXlK+ihWFreSXT/ISji7bzookxAbbE2HP7j8fIGoDBsl18j+QAat887
         yW4oBV9YWzBGC4Qn4nkBuTVTBTW7JAcCYsxcA=
MIME-Version: 1.0
In-Reply-To: <BANLkTikWG8SC3WbJf6=HfpNVn_=PJE6=ww@mail.gmail.com>
References: <BANLkTikWG8SC3WbJf6=HfpNVn_=PJE6=ww@mail.gmail.com>
From: tsuna <tsunanet@gmail.com>
Date: Thu, 21 Apr 2011 14:44:28 -0700
Message-ID: <BANLkTimSjSH2eCQFqPrMLDPfjRhoyKc=sw@mail.gmail.com>
Subject: Re: issue with thrift server pile up.
To: user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Wed, Apr 20, 2011 at 3:45 PM, Jack Levin <magnito@gmail.com> wrote:
> Hello -- we have an issue that looks like this. =A0 We have php app
> front end and thrift servers that live on seperate boxes away from
> HBASE cluster. =A0Everytime we do compaction on one of our 8 RS servers,
> we cause a thread pile up on Thrift servers that delay _all_ queries
> to HBASE. =A0Our usual level of threads on Thrift is around 250 threads,
> but when we do a single compaction on a single region server, the
> Thrift threads jump to about 5,000, which makes HBASE to be slow for
> all regions , not just the one that compacts. =A0 How does one solve an
> issue like that? =A0I know with mysql we can setup slaves and just
> redirect queries to slave zones that are faster, but what options do
> we have to avoid HBASE slowdown as a whole?

Writing a new Thrift server that's asynchronous and uses asynchbase
would avoid this issue, I believe, because only requests that are
blocked / slowed down by the compaction would be impacted, but other
requests would still flow normally.  I've been considering to do
something like that as I believe we have the same problem at
StumbleUpon (although to the impact is less in our case because our
cluster is bigger than 8 RS).

--=20
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com