hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saurabh Yahoo <saurabh...@yahoo.com>
Subject Re: experiencing high latency for few reads in HBase
Date Thu, 29 Aug 2013 17:47:35 GMT
Thanks Federico. I will look into this patch.

We are using .94.6.1 version. 

On Aug 29, 2013, at 8:37 AM, Federico Gaule <fgaule@despegar.com> wrote:

> In 0.94.11 Release, has been included an optimization for MultiGets: https://issues.apache.org/jira/browse/HBASE-9087
> 
> What version have you deployed?
> 
> 
> On 08/29/2013 01:29 AM, lars hofhansl wrote:
>> A 1s SLA is tough in HBase (or any large memory JVM application).
>> 
>> 
>> Maybe, if you presplit your table, play with JDK7 and the G1 collector, but nobody
here will vouch for such an SLA in the 99th percentile.
>> I heard some folks have experimented with 30GB heaps and G1 and have reported max
GC times of 200ms, but I have not verified that.
>> 
>> -- Lars
>> 
>> 
>> 
>> ----- Original Message -----
>> From: Saurabh Yahoo <saurabh_ag@yahoo.com>
>> To: "user@hbase.apache.org" <user@hbase.apache.org>
>> Cc: "user@hbase.apache.org" <user@hbase.apache.org>
>> Sent: Wednesday, August 28, 2013 3:17 PM
>> Subject: Re: experiencing high latency for few reads in HBase
>> 
>> Hi Vlad,
>> 
>> Thanks for your response.
>> 
>> 1. Our SLA is less than one sec. we cannot afford latency more than 1 sec.
>> 
>> We can increase heap size if that help, we have enough memory on server. What would
be the optimal heap size?
>> 
>> 2. Cache hit ratio is 95%.  One thing I don't understand that we have allocated only
4gb for block cache out of 12gb. That left 8gb for rest of JVM. There is no write. Memcache
is empty. Is 8gb not enough for hbase to process the requests? What are the most memory consuming
objects in region server?
>> 
>> 3. We will change the cf to IN_memory and report back performance difference.
>> 
>> Thanks,
>> Saurabh.
>> 
>> On Aug 28, 2013, at 3:15 PM, Vladimir Rodionov <vrodionov@carrieriq.com> wrote:
>> 
>>> 1. 4 sec max latency is not that bad taking into account 12GB heap.  It can be
much larger. What is your SLA?
>>> 2. Block evictions is the result of a poor cache hit rate and the root cause
of a periodic stop-the-world GC pauses (max latencies
>>>     latencies you have been observing in the test)
>>> 3. Block cache consists of 3 parts (25% young generation, 50% - tenured, 25%
- permanent). Permanent part is for CF with
>>> IN_MEMORY = true (you can specify this when you create CF).  Block first stored
in 'young gen' space, then gets promoted to 'tenured gen' space
>>> (or gets evicted). May be your 'perm gen' space is underutilized? This is exact
25% of 4GB (1GB). Although HBase LruBlockCache should use all the space allocated for block
cache -
>>> there is no guarantee (as usual). If you don have in_memory column families you
may decrease
>>> 
>>> 
>>> 
>>> Best regards,
>>> Vladimir Rodionov
>>> Principal Platform Engineer
>>> Carrier IQ, www.carrieriq.com
>>> e-mail: vrodionov@carrieriq.com
>>> 
>>> ________________________________________
>>> From: Saurabh Yahoo [saurabh_ag@yahoo.com]
>>> Sent: Wednesday, August 28, 2013 5:10 AM
>>> To: user@hbase.apache.org
>>> Subject: experiencing high latency for few reads in HBase
>>> 
>>> Hi,
>>> 
>>> We are running a stress test in our 5 node cluster and we are getting the expected
mean latency of 10ms. But we are seeing around 20 reads out of 25 million reads having latency
more than 4 seconds. Can anyone provide the insight what we can do to meet below second SLA
for each and every read?
>>> 
>>> We observe the following things -
>>> 
>>> 1. Reads are evenly distributed among 5 nodes.  CPUs remain under 5% utilized.
>>> 
>>> 2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb block cache
got filled up but around 1gb remained free. There are a large number of cache eviction.
>>> 
>>> Questions to experts -
>>> 
>>> 1. If there are still 1gb of free block cache available, why is hbase evicting
the block from cache?
>>> 
>>> 4. We are seeing memory went up to 10gb three times before dropping sharply to
5gb.
>>> 
>>> Any help is highly appreciable,
>>> 
>>> Thanks,
>>> Saurabh.
>>> 
>>> Confidentiality Notice:  The information contained in this message, including
any attachments hereto, may be confidential and is intended to be read only by the individual
or entity to whom this message is addressed. If the reader of this message is not the intended
recipient or an agent or designee of the intended recipient, please note that any review,
use, disclosure or distribution of this message or its attachments, in any form, is strictly
prohibited.  If you have received this message in error, please immediately notify the sender
and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its
attachments.
> 

Mime
View raw message