Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7ACE61023F for ; Thu, 29 Aug 2013 00:11:18 +0000 (UTC) Received: (qmail 72006 invoked by uid 500); 29 Aug 2013 00:11:16 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 71955 invoked by uid 500); 29 Aug 2013 00:11:16 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 71947 invoked by uid 99); 29 Aug 2013 00:11:16 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Aug 2013 00:11:16 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of vrodionov@carrieriq.com designates 204.235.122.16 as permitted sender) Received: from [204.235.122.16] (HELO obmail.carrieriq.com) (204.235.122.16) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Aug 2013 00:11:10 +0000 From: Vladimir Rodionov To: "user@hbase.apache.org" Date: Wed, 28 Aug 2013 17:10:47 -0700 Subject: RE: experiencing high latency for few reads in HBase Thread-Topic: experiencing high latency for few reads in HBase Thread-Index: Ac6kRRcFJ+10zDFkTzG3Dea8wu0PrgAA8EmQ Message-ID: References: <1FC10E17-08E6-4674-9CAF-9FD0D2984958@yahoo.com> <1377721830.1721.YahooMailNeo@web181602.mail.ne1.yahoo.com> <493CA5AA-4769-4785-A724-792EDB146753@yahoo.com> <1377730901.39919.YahooMailNeo@web181605.mail.ne1.yahoo.com>,<8EA7DF30-6261-4CD6-83DC-7EF586BF43BC@yahoo.com> In-Reply-To: <8EA7DF30-6261-4CD6-83DC-7EF586BF43BC@yahoo.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US x-kse-antivirus-interceptor-info: protection disabled Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org Increasing Java heap size will make latency worse, actually. You can't guarantee 1 sec max latency if run Java app (unless your heap siz= e is much less than 1GB).=20 I have never heard about strict maximum latency limit. Usually , its 99% , = 99.9 or 99.99% query percentiles. You can greatly reduce your 99.xxx% percentile latency by storing you data = in 2 replicas to two different region servers. Issue two read operations to those two region servers in parallel and get t= he first response. Probability theory states that probability=20 of two independent events (slow requests) is the product of event's probab= ilities themselves.=20 Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodionov@carrieriq.com ________________________________________ From: Saurabh Yahoo [saurabh_ag@yahoo.com] Sent: Wednesday, August 28, 2013 4:18 PM To: user@hbase.apache.org Subject: Re: experiencing high latency for few reads in HBase Thanks Kiru, Scan is not an option for our use cases. Our read is pretty random. Any other suggestion to bring down the latency. Thanks, Saurabh. On Aug 28, 2013, at 7:01 PM, Kiru Pakkirisamy w= rote: > Saurabh, we are able to 600K rowxcolumns in 400 msec. We have put what wa= s a 40million row table as 400K rows and columns. We Get about 100 of the r= ows from this 400K , do quite a bit of calculations in the coprocessor (alm= ost a group-order by) and return in this time. > Maybe should consider replacing the MultiGets with Scan with Filter. I li= ke the FuzzyRowFilter even though you might need to match with exact key. I= t works only with fixed length key. > (I do have an issue right now, it is not scaling to multiple clients.) > > Regards, > - kiru > > > Kiru Pakkirisamy | webcloudtech.wordpress.com > > > ________________________________ > From: Saurabh Yahoo > To: "user@hbase.apache.org" > Cc: "user@hbase.apache.org" > Sent: Wednesday, August 28, 2013 3:20 PM > Subject: Re: experiencing high latency for few reads in HBase > > > Thanks Kitu. We need less than 1 sec latency. > > We are using both muliGet and get. > > We have three concurrent clients running 10 threads each. ( that makes to= tal 30 concurrent clients). > > Thanks, > Saurabh. > > On Aug 28, 2013, at 4:30 PM, Kiru Pakkirisamy = wrote: > >> Right 4 sec is good. >> @Saurabh - so your read is - getting 20 out of 25 millions rows ?. Is th= is a Get or a Scan ? >> BTW, in this stress test how many concurrent clients do you have ? >> >> Regards, >> - kiru >> >> >> ________________________________ >> From: Vladimir Rodionov >> To: "user@hbase.apache.org" >> Sent: Wednesday, August 28, 2013 12:15 PM >> Subject: RE: experiencing high latency for few reads in HBase >> >> >> 1. 4 sec max latency is not that bad taking into account 12GB heap. It = can be much larger. What is your SLA? >> 2. Block evictions is the result of a poor cache hit rate and the root c= ause of a periodic stop-the-world GC pauses (max latencies >> latencies you have been observing in the test) >> 3. Block cache consists of 3 parts (25% young generation, 50% - tenured,= 25% - permanent). Permanent part is for CF with >> IN_MEMORY =3D true (you can specify this when you create CF). Block fir= st stored in 'young gen' space, then gets promoted to 'tenured gen' space >> (or gets evicted). May be your 'perm gen' space is underutilized? This i= s exact 25% of 4GB (1GB). Although HBase LruBlockCache should use all the s= pace allocated for block cache - >> there is no guarantee (as usual). If you don have in_memory column famil= ies you may decrease >> >> >> >> Best regards, >> Vladimir Rodionov >> Principal Platform Engineer >> Carrier IQ, www.carrieriq.com >> e-mail: vrodionov@carrieriq.com >> >> ________________________________________ >> From: Saurabh Yahoo [saurabh_ag@yahoo.com] >> Sent: Wednesday, August 28, 2013 5:10 AM >> To: user@hbase.apache.org >> Subject: experiencing high latency for few reads in HBase >> >> Hi, >> >> We are running a stress test in our 5 node cluster and we are getting th= e expected mean latency of 10ms. But we are seeing around 20 reads out of 2= 5 million reads having latency more than 4 seconds. Can anyone provide the = insight what we can do to meet below second SLA for each and every read? >> >> We observe the following things - >> >> 1. Reads are evenly distributed among 5 nodes. CPUs remain under 5% uti= lized. >> >> 2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb bloc= k cache got filled up but around 1gb remained free. There are a large numbe= r of cache eviction. >> >> Questions to experts - >> >> 1. If there are still 1gb of free block cache available, why is hbase ev= icting the block from cache? >> >> 4. We are seeing memory went up to 10gb three times before dropping shar= ply to 5gb. >> >> Any help is highly appreciable, >> >> Thanks, >> Saurabh. >> >> Confidentiality Notice: The information contained in this message, incl= uding any attachments hereto, may be confidential and is intended to be rea= d only by the individual or entity to whom this message is addressed. If th= e reader of this message is not the intended recipient or an agent or desig= nee of the intended recipient, please note that any review, use, disclosure= or distribution of this message or its attachments, in any form, is strict= ly prohibited. If you have received this message in error, please immediat= ely notify the sender and/or Notifications@carrieriq.com and delete or dest= roy any copy of this message and its attachments.