Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=rKkjqkKm1PdtzgVkGFbccSHICfI65Pp0rpo67oP+Aec6MPpcAJK01YwczZNJ15t70tPiwT29Kn8wBCIyDjnAa9HMWv+DqCqOW+kwphJmFkPA4PxtOLyBHbFCvy4LUEupiJMmBhASrwk+S+5uLUzv/vlgVjtrkQwLUe7KxCdl8Ao=;
References: <1FC10E17-08E6-4674-9CAF-9FD0D2984958@yahoo.com>
 <DC5EBE7F3610EB4CA5C7E92D76873E15180DD42A9F@exchange2007.carrieriq.com>
 <1316DF98-4631-4280-B210-DD514D19653C@yahoo.com>
 <1377750572.64627.YahooMailNeo@web140602.mail.bf1.yahoo.com>
 <521F406C.5070707@despegar.com>
 <1377830723.75425.YahooMailNeo@web181604.mail.ne1.yahoo.com>
 <CADt=Q7kXkoQ5jOx2ga5EU6kLbU=_VvnyNJyjwNEd5fAjeekGZw@mail.gmail.com>
Message-ID: <1377851771.24619.YahooMailNeo@web181604.mail.ne1.yahoo.com>
Date: Fri, 30 Aug 2013 01:36:11 -0700 (PDT)
From: Kiru Pakkirisamy <kirupakkirisamy@yahoo.com>
Reply-To: Kiru Pakkirisamy <kirupakkirisamy@yahoo.com>
Subject: Re: experiencing high latency for few reads in HBase
To: "user@hbase.apache.org" <user@hbase.apache.org>
In-Reply-To: 
 <CADt=Q7kXkoQ5jOx2ga5EU6kLbU=_VvnyNJyjwNEd5fAjeekGZw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="2017413661-846613629-1377851771=:24619"

--2017413661-846613629-1377851771=:24619
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

I did check GC earlier. Not much . About .0x seconds.=0AI think there is lo=
ck/contention issue. 0.94.11 is also more stable compared to 0.94.10.=0AWit=
h 64 concurrent clients, 0.94.10 one regionserver or other will crash.=0A=
=0ARegards,=0A- kiru=0A=0A=0A________________________________=0A From: Adri=
en Mogenet <adrien.mogenet@gmail.com>=0ATo: user <user@hbase.apache.org> =
=0ASent: Thursday, August 29, 2013 11:11 PM=0ASubject: Re: experiencing hig=
h latency for few reads in HBase=0A =0A=0AAre your GC logs enabled? Can you=
 see any long pause in it?=0A=0A=0AOn Fri, Aug 30, 2013 at 4:45 AM, Kiru Pa=
kkirisamy <kirupakkirisamy@yahoo.com=0A> wrote:=0A=0A> I just moved from 0.=
94.10 to 0.94.11. Tremendous improvement in our app's=0A> query response. W=
ent down to 1.3 sec from 1.7 sec.=0A> Concurrent tests are also good, but i=
t still exponentially degrades from=0A> to 10 secs for 8 concurrent clients=
. There might a bug lurking in there=0A> somewhere that is probably affecti=
ng us.=0A>=0A>=0A> Regards,=0A> - kiru=0A>=0A> ____________________________=
____=0A>=A0 From: Federico Gaule <fgaule@despegar.com>=0A> To: user@hbase.a=
pache.org=0A> Sent: Thursday, August 29, 2013 5:37 AM=0A> Subject: Re: expe=
riencing high latency for few reads in HBase=0A>=0A>=0A> In 0.94.11 Release=
, has been included an optimization for MultiGets:=0A> https://issues.apach=
e.org/jira/browse/HBASE-9087=0A>=0A> What version have you deployed?=0A>=0A=
>=0A> On 08/29/2013 01:29 AM, lars hofhansl wrote:=0A> > A 1s SLA is tough =
in HBase (or any large memory JVM application).=0A> >=0A> >=0A> > Maybe, if=
 you presplit your table, play with JDK7 and the G1 collector,=0A> but nobo=
dy here will vouch for such an SLA in the 99th percentile.=0A> > I heard so=
me folks have experimented with 30GB heaps and G1 and have=0A> reported max=
 GC times of 200ms, but I have not verified that.=0A> >=0A> > -- Lars=0A> >=
=0A> >=0A> >=0A> > ----- Original Message -----=0A> > From: Saurabh Yahoo <=
saurabh_ag@yahoo.com>=0A> > To: "user@hbase.apache.org" <user@hbase.apache.=
org>=0A> > Cc: "user@hbase.apache.org" <user@hbase.apache.org>=0A> > Sent: =
Wednesday, August 28, 2013 3:17 PM=0A> > Subject: Re: experiencing high lat=
ency for few reads in HBase=0A> >=0A> > Hi Vlad,=0A> >=0A> > Thanks for you=
r response.=0A> >=0A> > 1. Our SLA is less than one sec. we cannot afford l=
atency more than 1=0A> sec.=0A> >=0A> > We can increase heap size if that h=
elp, we have enough memory on server.=0A> What would be the optimal heap si=
ze?=0A> >=0A> > 2. Cache hit ratio is 95%.=A0 One thing I don't understand =
that we have=0A> allocated only 4gb for block cache out of 12gb. That left =
8gb for rest of=0A> JVM. There is no write. Memcache is empty. Is 8gb not e=
nough for hbase to=0A> process the requests? What are the most memory consu=
ming objects in region=0A> server?=0A> >=0A> > 3. We will change the cf to =
IN_memory and report back performance=0A> difference.=0A> >=0A> > Thanks,=
=0A> > Saurabh.=0A> >=0A> > On Aug 28, 2013, at 3:15 PM, Vladimir Rodionov =
<vrodionov@carrieriq.com>=0A> wrote:=0A> >=0A> >> 1. 4 sec max latency is n=
ot that bad taking into account 12GB heap.=A0 It=0A> can be much larger. Wh=
at is your SLA?=0A> >> 2. Block evictions is the result of a poor cache hit=
 rate and the root=0A> cause of a periodic stop-the-world GC pauses (max la=
tencies=0A> >>=A0 =A0 =A0 latencies you have been observing in the test)=0A=
> >> 3. Block cache consists of 3 parts (25% young generation, 50% -=0A> te=
nured, 25% - permanent). Permanent part is for CF with=0A> >> IN_MEMORY =3D=
 true (you can specify this when you create CF).=A0 Block=0A> first stored =
in 'young gen' space, then gets promoted to 'tenured gen' space=0A> >> (or =
gets evicted). May be your 'perm gen' space is underutilized? This=0A> is e=
xact 25% of 4GB (1GB). Although HBase LruBlockCache should use all the=0A> =
space allocated for block cache -=0A> >> there is no guarantee (as usual). =
If you don have in_memory column=0A> families you may decrease=0A> >>=0A> >=
>=0A> >>=0A> >> Best regards,=0A> >> Vladimir Rodionov=0A> >> Principal Pla=
tform Engineer=0A> >> Carrier IQ, www.carrieriq.com=0A> >> e-mail: vrodiono=
v@carrieriq.com=0A> >>=0A> >> ________________________________________=0A> =
>> From: Saurabh Yahoo [saurabh_ag@yahoo.com]=0A> >> Sent: Wednesday, Augus=
t 28, 2013 5:10 AM=0A> >> To: user@hbase.apache.org=0A> >> Subject: experie=
ncing high latency for few reads in HBase=0A> >>=0A> >> Hi,=0A> >>=0A> >> W=
e are running a stress test in our 5 node cluster and we are getting=0A> th=
e expected mean latency of 10ms. But we are seeing around 20 reads out of=
=0A> 25 million reads having latency more than 4 seconds. Can anyone provid=
e the=0A> insight what we can do to meet below second SLA for each and ever=
y read?=0A> >>=0A> >> We observe the following things -=0A> >>=0A> >> 1. Re=
ads are evenly distributed among 5 nodes.=A0 CPUs remain under 5%=0A> utili=
zed.=0A> >>=0A> >> 2. We have 4gb block cache (30% block cache out of 12gb)=
 setup. 3gb=0A> block cache got filled up but around 1gb remained free. The=
re are a large=0A> number of cache eviction.=0A> >>=0A> >> Questions to exp=
erts -=0A> >>=0A> >> 1. If there are still 1gb of free block cache availabl=
e, why is hbase=0A> evicting the block from cache?=0A> >>=0A> >> 4. We are =
seeing memory went up to 10gb three times before dropping=0A> sharply to 5g=
b.=0A> >>=0A> >> Any help is highly appreciable,=0A> >>=0A> >> Thanks,=0A> =
>> Saurabh.=0A> >>=0A> >> Confidentiality Notice:=A0 The information contai=
ned in this message,=0A> including any attachments hereto, may be confident=
ial and is intended to be=0A> read only by the individual or entity to whom=
 this message is addressed. If=0A> the reader of this message is not the in=
tended recipient or an agent or=0A> designee of the intended recipient, ple=
ase note that any review, use,=0A> disclosure or distribution of this messa=
ge or its attachments, in any form,=0A> is strictly prohibited.=A0 If you h=
ave received this message in error, please=0A> immediately notify the sende=
r and/or Notifications@carrieriq.com and=0A> delete or destroy any copy of =
this message and its attachments.=0A>=0A=0A=0A=0A-- =0AAdrien Mogenet=0Ahtt=
p://www.borntosegfault.com
--2017413661-846613629-1377851771=:24619--