Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=3nvsWq/7minpTfJ0eGSE09bxmvzXAHqeN33QHa9wfbcnn3BMyq8haVNVg7nlqItdcneg12h3w2vimM5uI+hss1xffsyMkhgHycLrn3Z7WiFkPn6O+q5RNAQZgN4y2i7Gl8Q4eIFGpnv2aHWP5NLb9AsznZ82CP3LO2f2MDD9IGk=;
References: <1FC10E17-08E6-4674-9CAF-9FD0D2984958@yahoo.com>
 <DC5EBE7F3610EB4CA5C7E92D76873E15180DD42A9F@exchange2007.carrieriq.com>
 <1377721830.1721.YahooMailNeo@web181602.mail.ne1.yahoo.com>
 <493CA5AA-4769-4785-A724-792EDB146753@yahoo.com>
 <1377730901.39919.YahooMailNeo@web181605.mail.ne1.yahoo.com>
 <8EA7DF30-6261-4CD6-83DC-7EF586BF43BC@yahoo.com>
 <DC5EBE7F3610EB4CA5C7E92D76873E15180DD42AA1@exchange2007.carrieriq.com>
 <101B53BD-B076-4B75-9648-279F820D601E@yahoo.com>
 <DC5EBE7F3610EB4CA5C7E92D76873E15180DD42AA3@exchange2007.carrieriq.com>,<2CF1F114-1FA2-4F13-8141-BE50F80C9CFD@yahoo.com>
 <DC5EBE7F3610EB4CA5C7E92D76873E15180DD42AA4@exchange2007.carrieriq.com>
Message-ID: <1377801185.11300.YahooMailNeo@web181605.mail.ne1.yahoo.com>
Date: Thu, 29 Aug 2013 11:33:05 -0700 (PDT)
From: Kiru Pakkirisamy <kirupakkirisamy@yahoo.com>
Reply-To: Kiru Pakkirisamy <kirupakkirisamy@yahoo.com>
Subject: Re: experiencing high latency for few reads in HBase 
To: "user@hbase.apache.org" <user@hbase.apache.org>
In-Reply-To: 
 <DC5EBE7F3610EB4CA5C7E92D76873E15180DD42AA4@exchange2007.carrieriq.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="1885600119-1687031801-1377801185=:11300"

--1885600119-1687031801-1377801185=:11300
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

But locality index should not matter right if you are in IN_MEMORY most and=
 you are running the test after =A0a few runs to make sure they are already=
 in IN_MEMORY =A0(ie blockCacheHit is high or blockCacheMiss is low) =A0(?)=
=A0=0A=0ARegards,=0A- kiru=0A=0A=0AKiru Pakkirisamy | webcloudtech.wordpres=
s.com=0A=0A=0A________________________________=0A From: Vladimir Rodionov <=
vrodionov@carrieriq.com>=0ATo: "user@hbase.apache.org" <user@hbase.apache.o=
rg> =0ASent: Thursday, August 29, 2013 11:11 AM=0ASubject: RE: experiencing=
 high latency for few reads in HBase =0A =0A=0AUsually, either cluster rest=
art or major compaction helps improving locality index.=0AThere is an issue=
 in region assignment after table disable/enable in 0.94.x (x <11) which =
=0Abreaks HDFS locality. Fixed in 0.94.11 =0A=0AYou can write your own rout=
ine to manually "localize" particular table using public HBase Client API.=
=0A=0ABut this won't help you to stay withing 1 sec anyway. =0A=0ABest rega=
rds,=0AVladimir Rodionov=0APrincipal Platform Engineer=0ACarrier IQ, www.ca=
rrieriq.com=0Ae-mail: vrodionov@carrieriq.com=0A=0A________________________=
________________=0AFrom: Saurabh Yahoo [saurabh_ag@yahoo.com]=0ASent: Thurs=
day, August 29, 2013 10:52 AM=0ATo: user@hbase.apache.org=0ACc: user@hbase.=
apache.org=0ASubject: Re: experiencing high latency for few reads in HBase=
=0A=0AThanks Vlad.=0A=0AQuick question. I notice hdfsBlocksLocalityIndex is=
 around 50 in all region servers.=0A=0ADoes that could be a problem? If it =
is, how to solve that? We already ran the major compaction after ingesting =
the data.=0A=0AThanks,=0ASaurabh.=0A=0AOn Aug 29, 2013, at 12:17 PM, Vladim=
ir Rodionov <vrodionov@carrieriq.com> wrote:=0A=0A> Yes. HBase won't guaran=
tee strict sub-second latency.=0A>=0A> Best regards,=0A> Vladimir Rodionov=
=0A> Principal Platform Engineer=0A> Carrier IQ, www.carrieriq.com=0A> e-ma=
il: vrodionov@carrieriq.com=0A>=0A> _______________________________________=
_=0A> From: Saurabh Yahoo [saurabh_ag@yahoo.com]=0A> Sent: Thursday, August=
 29, 2013 2:49 AM=0A> To: user@hbase.apache.org=0A> Cc: user@hbase.apache.o=
rg=0A> Subject: Re: experiencing high latency for few reads in HBase=0A>=0A=
> Hi Vlad,=0A>=0A> We do have strict latency requirement as it is financial=
 data requiring direct access from clients.=0A>=0A> Are you saying that it =
is not possible to achieve sub second latency using hbase (because it is ba=
sed on java.) ?=0A>=0A>=0A>=0A>=0A>=0A>=0A>=0A> On Aug 28, 2013, at 8:10 PM=
, Vladimir Rodionov <vrodionov@carrieriq.com> wrote:=0A>=0A>> Increasing Ja=
va heap size will make latency worse, actually.=0A>> You can't guarantee 1 =
sec max latency if run Java app (unless your heap size is much less than 1G=
B).=0A>> I have never heard about strict maximum latency limit. Usually , i=
ts 99% , 99.9 or 99.99% query percentiles.=0A>>=0A>> You can greatly reduce=
 your 99.xxx% percentile latency by storing you data in 2 replicas to two d=
ifferent region servers.=0A>> Issue two read operations to those two region=
 servers in parallel and get the first response. Probability theory states =
that=A0 probability=0A>> of two independent events (slow requests) is=A0 th=
e product of event's probabilities themselves.=0A>>=0A>>=0A>> Best regards,=
=0A>> Vladimir Rodionov=0A>> Principal Platform Engineer=0A>> Carrier IQ, w=
ww.carrieriq.com=0A>> e-mail: vrodionov@carrieriq.com=0A>>=0A>> ___________=
_____________________________=0A>> From: Saurabh Yahoo [saurabh_ag@yahoo.co=
m]=0A>> Sent: Wednesday, August 28, 2013 4:18 PM=0A>> To: user@hbase.apache=
.org=0A>> Subject: Re: experiencing high latency for few reads in HBase=0A>=
>=0A>> Thanks Kiru,=0A>>=0A>> Scan is not an option for our use cases.=A0 O=
ur read is pretty random.=0A>>=0A>> Any other suggestion to bring down the =
latency.=0A>>=0A>> Thanks,=0A>> Saurabh.=0A>>=0A>>=0A>> On Aug 28, 2013, at=
 7:01 PM, Kiru Pakkirisamy <kirupakkirisamy@yahoo.com> wrote:=0A>>=0A>>> Sa=
urabh, we are able to 600K rowxcolumns in 400 msec. We have put what was a =
40million row table as 400K rows and columns. We Get about 100 of the rows =
from this 400K , do quite a bit of calculations in the coprocessor (almost =
a group-order by) and return in this time.=0A>>> Maybe should consider repl=
acing the MultiGets with Scan with Filter. I like the FuzzyRowFilter even t=
hough you might need to match with exact key. It works only with fixed leng=
th key.=0A>>> (I do have an issue right now, it is not scaling to multiple =
clients.)=0A>>>=0A>>> Regards,=0A>>> - kiru=0A>>>=0A>>>=0A>>> Kiru Pakkiris=
amy | webcloudtech.wordpress.com=0A>>>=0A>>>=0A>>> ________________________=
________=0A>>> From: Saurabh Yahoo <saurabh_ag@yahoo.com>=0A>>> To: "user@h=
base.apache.org" <user@hbase.apache.org>=0A>>> Cc: "user@hbase.apache.org" =
<user@hbase.apache.org>=0A>>> Sent: Wednesday, August 28, 2013 3:20 PM=0A>>=
> Subject: Re: experiencing high latency for few reads in HBase=0A>>>=0A>>>=
=0A>>> Thanks Kitu. We need less than 1 sec latency.=0A>>>=0A>>> We are usi=
ng both muliGet and get.=0A>>>=0A>>> We have three concurrent clients runni=
ng 10 threads each. ( that makes total 30 concurrent clients).=0A>>>=0A>>> =
Thanks,=0A>>> Saurabh.=0A>>>=0A>>> On Aug 28, 2013, at 4:30 PM, Kiru Pakkir=
isamy <kirupakkirisamy@yahoo.com> wrote:=0A>>>=0A>>>> Right 4 sec is good.=
=0A>>>> @Saurabh - so your read is - getting 20 out of 25 millions rows ?. =
Is this a Get or a Scan ?=0A>>>> BTW, in this stress test how many concurre=
nt clients do you have ?=0A>>>>=0A>>>> Regards,=0A>>>> - kiru=0A>>>>=0A>>>>=
=0A>>>> ________________________________=0A>>>> From: Vladimir Rodionov <vr=
odionov@carrieriq.com>=0A>>>> To: "user@hbase.apache.org" <user@hbase.apach=
e.org>=0A>>>> Sent: Wednesday, August 28, 2013 12:15 PM=0A>>>> Subject: RE:=
 experiencing high latency for few reads in HBase=0A>>>>=0A>>>>=0A>>>> 1. 4=
 sec max latency is not that bad taking into account 12GB heap.=A0 It can b=
e much larger. What is your SLA?=0A>>>> 2. Block evictions is the result of=
 a poor cache hit rate and the root cause of a periodic stop-the-world GC p=
auses (max latencies=0A>>>>=A0 =A0 latencies you have been observing in the=
 test)=0A>>>> 3. Block cache consists of 3 parts (25% young generation, 50%=
 - tenured, 25% - permanent). Permanent part is for CF with=0A>>>> IN_MEMOR=
Y =3D true (you can specify this when you create CF).=A0 Block first stored=
 in 'young gen' space, then gets promoted to 'tenured gen' space=0A>>>> (or=
 gets evicted). May be your 'perm gen' space is underutilized? This is exac=
t 25% of 4GB (1GB). Although HBase LruBlockCache should use all the space a=
llocated for block cache -=0A>>>> there is no guarantee (as usual). If you =
don have in_memory column families you may decrease=0A>>>>=0A>>>>=0A>>>>=0A=
>>>> Best regards,=0A>>>> Vladimir Rodionov=0A>>>> Principal Platform Engin=
eer=0A>>>> Carrier IQ, www.carrieriq.com=0A>>>> e-mail: vrodionov@carrieriq=
.com=0A>>>>=0A>>>> ________________________________________=0A>>>> From: Sa=
urabh Yahoo [saurabh_ag@yahoo.com]=0A>>>> Sent: Wednesday, August 28, 2013 =
5:10 AM=0A>>>> To: user@hbase.apache.org=0A>>>> Subject: experiencing high =
latency for few reads in HBase=0A>>>>=0A>>>> Hi,=0A>>>>=0A>>>> We are runni=
ng a stress test in our 5 node cluster and we are getting the expected mean=
 latency of 10ms. But we are seeing around 20 reads out of 25 million reads=
 having latency more than 4 seconds. Can anyone provide the insight what we=
 can do to meet below second SLA for each and every read?=0A>>>>=0A>>>> We =
observe the following things -=0A>>>>=0A>>>> 1. Reads are evenly distribute=
d among 5 nodes.=A0 CPUs remain under 5% utilized.=0A>>>>=0A>>>> 2. We have=
 4gb block cache (30% block cache out of 12gb) setup. 3gb block cache got f=
illed up but around 1gb remained free. There are a large number of cache ev=
iction.=0A>>>>=0A>>>> Questions to experts -=0A>>>>=0A>>>> 1. If there are =
still 1gb of free block cache available, why is hbase evicting the block fr=
om cache?=0A>>>>=0A>>>> 4. We are seeing memory went up to 10gb three times=
 before dropping sharply to 5gb.=0A>>>>=0A>>>> Any help is highly appreciab=
le,=0A>>>>=0A>>>> Thanks,=0A>>>> Saurabh.=0A>>>>=0A>>>> Confidentiality Not=
ice:=A0 The information contained in this message, including any attachment=
s hereto, may be confidential and is intended to be read only by the indivi=
dual or entity to whom this message is addressed. If the reader of this mes=
sage is not the intended recipient or an agent or designee of the intended =
recipient, please note that any review, use, disclosure or distribution of =
this message or its attachments, in any form, is strictly prohibited.=A0 If=
 you have received this message in error, please immediately notify the sen=
der and/or Notifications@carrieriq.com and delete or destroy any copy of th=
is message and its attachments.
--1885600119-1687031801-1377801185=:11300--