hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrien Mogenet <adrien.moge...@gmail.com>
Subject Re: experiencing high latency for few reads in HBase
Date Fri, 30 Aug 2013 06:11:23 GMT
Are your GC logs enabled? Can you see any long pause in it?


On Fri, Aug 30, 2013 at 4:45 AM, Kiru Pakkirisamy <kirupakkirisamy@yahoo.com
> wrote:

> I just moved from 0.94.10 to 0.94.11. Tremendous improvement in our app's
> query response. Went down to 1.3 sec from 1.7 sec.
> Concurrent tests are also good, but it still exponentially degrades from
> to 10 secs for 8 concurrent clients. There might a bug lurking in there
> somewhere that is probably affecting us.
>
>
> Regards,
> - kiru
>
> ________________________________
>  From: Federico Gaule <fgaule@despegar.com>
> To: user@hbase.apache.org
> Sent: Thursday, August 29, 2013 5:37 AM
> Subject: Re: experiencing high latency for few reads in HBase
>
>
> In 0.94.11 Release, has been included an optimization for MultiGets:
> https://issues.apache.org/jira/browse/HBASE-9087
>
> What version have you deployed?
>
>
> On 08/29/2013 01:29 AM, lars hofhansl wrote:
> > A 1s SLA is tough in HBase (or any large memory JVM application).
> >
> >
> > Maybe, if you presplit your table, play with JDK7 and the G1 collector,
> but nobody here will vouch for such an SLA in the 99th percentile.
> > I heard some folks have experimented with 30GB heaps and G1 and have
> reported max GC times of 200ms, but I have not verified that.
> >
> > -- Lars
> >
> >
> >
> > ----- Original Message -----
> > From: Saurabh Yahoo <saurabh_ag@yahoo.com>
> > To: "user@hbase.apache.org" <user@hbase.apache.org>
> > Cc: "user@hbase.apache.org" <user@hbase.apache.org>
> > Sent: Wednesday, August 28, 2013 3:17 PM
> > Subject: Re: experiencing high latency for few reads in HBase
> >
> > Hi Vlad,
> >
> > Thanks for your response.
> >
> > 1. Our SLA is less than one sec. we cannot afford latency more than 1
> sec.
> >
> > We can increase heap size if that help, we have enough memory on server.
> What would be the optimal heap size?
> >
> > 2. Cache hit ratio is 95%.  One thing I don't understand that we have
> allocated only 4gb for block cache out of 12gb. That left 8gb for rest of
> JVM. There is no write. Memcache is empty. Is 8gb not enough for hbase to
> process the requests? What are the most memory consuming objects in region
> server?
> >
> > 3. We will change the cf to IN_memory and report back performance
> difference.
> >
> > Thanks,
> > Saurabh.
> >
> > On Aug 28, 2013, at 3:15 PM, Vladimir Rodionov <vrodionov@carrieriq.com>
> wrote:
> >
> >> 1. 4 sec max latency is not that bad taking into account 12GB heap.  It
> can be much larger. What is your SLA?
> >> 2. Block evictions is the result of a poor cache hit rate and the root
> cause of a periodic stop-the-world GC pauses (max latencies
> >>      latencies you have been observing in the test)
> >> 3. Block cache consists of 3 parts (25% young generation, 50% -
> tenured, 25% - permanent). Permanent part is for CF with
> >> IN_MEMORY = true (you can specify this when you create CF).  Block
> first stored in 'young gen' space, then gets promoted to 'tenured gen' space
> >> (or gets evicted). May be your 'perm gen' space is underutilized? This
> is exact 25% of 4GB (1GB). Although HBase LruBlockCache should use all the
> space allocated for block cache -
> >> there is no guarantee (as usual). If you don have in_memory column
> families you may decrease
> >>
> >>
> >>
> >> Best regards,
> >> Vladimir Rodionov
> >> Principal Platform Engineer
> >> Carrier IQ, www.carrieriq.com
> >> e-mail: vrodionov@carrieriq.com
> >>
> >> ________________________________________
> >> From: Saurabh Yahoo [saurabh_ag@yahoo.com]
> >> Sent: Wednesday, August 28, 2013 5:10 AM
> >> To: user@hbase.apache.org
> >> Subject: experiencing high latency for few reads in HBase
> >>
> >> Hi,
> >>
> >> We are running a stress test in our 5 node cluster and we are getting
> the expected mean latency of 10ms. But we are seeing around 20 reads out of
> 25 million reads having latency more than 4 seconds. Can anyone provide the
> insight what we can do to meet below second SLA for each and every read?
> >>
> >> We observe the following things -
> >>
> >> 1. Reads are evenly distributed among 5 nodes.  CPUs remain under 5%
> utilized.
> >>
> >> 2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb
> block cache got filled up but around 1gb remained free. There are a large
> number of cache eviction.
> >>
> >> Questions to experts -
> >>
> >> 1. If there are still 1gb of free block cache available, why is hbase
> evicting the block from cache?
> >>
> >> 4. We are seeing memory went up to 10gb three times before dropping
> sharply to 5gb.
> >>
> >> Any help is highly appreciable,
> >>
> >> Thanks,
> >> Saurabh.
> >>
> >> Confidentiality Notice:  The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited.  If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>



-- 
Adrien Mogenet
http://www.borntosegfault.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message