hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tsuna <tsuna...@gmail.com>
Subject Re: issue with thrift server pile up.
Date Thu, 21 Apr 2011 21:44:28 GMT
On Wed, Apr 20, 2011 at 3:45 PM, Jack Levin <magnito@gmail.com> wrote:
> Hello -- we have an issue that looks like this.   We have php app
> front end and thrift servers that live on seperate boxes away from
> HBASE cluster.  Everytime we do compaction on one of our 8 RS servers,
> we cause a thread pile up on Thrift servers that delay _all_ queries
> to HBASE.  Our usual level of threads on Thrift is around 250 threads,
> but when we do a single compaction on a single region server, the
> Thrift threads jump to about 5,000, which makes HBASE to be slow for
> all regions , not just the one that compacts.   How does one solve an
> issue like that?  I know with mysql we can setup slaves and just
> redirect queries to slave zones that are faster, but what options do
> we have to avoid HBASE slowdown as a whole?

Writing a new Thrift server that's asynchronous and uses asynchbase
would avoid this issue, I believe, because only requests that are
blocked / slowed down by the compaction would be impacted, but other
requests would still flow normally.  I've been considering to do
something like that as I believe we have the same problem at
StumbleUpon (although to the impact is less in our case because our
cluster is bigger than 8 RS).

Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com

View raw message