hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tsuna <tsuna...@gmail.com>
Subject Re: asynchbase: A new entirely asynchronous / non-blocking HBase client
Date Sat, 11 Sep 2010 04:01:27 GMT
Due to popular demand, the license was changed to a BSD license.  I
also fixed a couple of bugs.

Our time series database (opentsdb.net) is running with this client in
prod and is writing up to 10k KeyValue per second to HBase (with the
WAL) on a crappy 4 core VM (I hate VMs, the performance is absolutely
awful).  At 10k QPS, the 4 cores allocated to the VM are all pegged at
100%, and small GC cycles take about 8ms every 25-30s (without any
specific memory or GC tuning parameter).

The code may not work with the latest version of HBase from SVN as I
think the multiPut RPC has recently changed.  Otherwise it should be
compatible with several recent 0.20.x releases and 0.89 releases.  It
doesn't care what the version of the RegionServers are running, it'll
just try to use them if it can.

One gotcha that someone recently ran into is that if you write to
HBase and then wait until the write completes before sending another
write, you'll send only 1 write per second.  That's because by default
the client flushes edits every second (I'm simplifying a bit, but
let's assume that's how it works – you can tune the flush interval,
see the javadoc).  It's an asynchronous client, so normally you're not
supposed to wait until the operation complete, you're supposed to use
a Callback.  If you do wanna wait, however, make sure you use many
concurrent threads to work around this issue.  But bear in mind that
performance is optimal when fewer threads use the client in an
asynchronous / non-blocking fashion.

-- 
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com

Mime
View raw message