hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tsuna <tsuna...@gmail.com>
Subject Re: Hbase read performance with increasing number of client threads
Date Wed, 08 Sep 2010 22:52:21 GMT
On Thu, Aug 19, 2010 at 10:15 AM, Abhijit Pol <apol@rocketfuelinc.com> wrote:
> We are using Hbase 0.20.5 drop with latest cloudera Hadoop distribution.
>
> - We are hitting 3 nodes Hbase cluster from a client which has 10
> threads each with thread local copy of HTable client object and
> established connection to server.
> - Each of 10 threads issuing 10,000 read requests of keys randomly
> selected from pool of 1000 keys. All keys are present on HBase and
> table is pinned in memory (to make sure we don't have any disk seeks).
> - If we run this test with 10 threads we get avg latency as seen by
> client = 8ms (excluding initial 10 connection setup time) . But if we
> increase # threads to 100, 250 to 500 we get increasing latency
> numbers like 26ms, 51ms, 90ms.
> - We have enabled HBase metrics on RS and we see "get_avg_time" on all
> RS between 5-15ms in all tests, consistently.
>
> Is this expected? Any tips to get consistent performance below 20ms?

If the server side latency consistently remains around 5-15ms but the
client side latency shoots up through the roof, you may be
experiencing lock contention or some other problem potentially even
unrelated to HBase in your application.  Maybe your application is
generating too much garbage and the GC has to run too frequently.
Maybe you have so many threads that you're trashing the caches of the
CPUs you're running on.

Adding more threads only makes things faster up to a certain point.
Past that point, things actually become slower.

-- 
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com

Mime
View raw message