hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kiru Pakkirisamy <kirupakkiris...@yahoo.com>
Subject Re: experiencing high latency for few reads in HBase
Date Wed, 28 Aug 2013 20:30:30 GMT
Right 4 sec is good.  
@Saurabh - so your read is - getting 20 out of 25 millions rows ?. Is this a Get or a Scan
BTW, in this stress test how many concurrent clients do you have ? 
- kiru

 From: Vladimir Rodionov <vrodionov@carrieriq.com>
To: "user@hbase.apache.org" <user@hbase.apache.org> 
Sent: Wednesday, August 28, 2013 12:15 PM
Subject: RE: experiencing high latency for few reads in HBase 

1. 4 sec max latency is not that bad taking into account 12GB heap.  It can be much larger.
What is your SLA?
2. Block evictions is the result of a poor cache hit rate and the root cause of a periodic
stop-the-world GC pauses (max latencies
    latencies you have been observing in the test)
3. Block cache consists of 3 parts (25% young generation, 50% - tenured, 25% - permanent).
Permanent part is for CF with
IN_MEMORY = true (you can specify this when you create CF).  Block first stored in 'young
gen' space, then gets promoted to 'tenured gen' space
(or gets evicted). May be your 'perm gen' space is underutilized? This is exact 25% of 4GB
(1GB). Although HBase LruBlockCache should use all the space allocated for block cache -
there is no guarantee (as usual). If you don have in_memory column families you may decrease

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com

From: Saurabh Yahoo [saurabh_ag@yahoo.com]
Sent: Wednesday, August 28, 2013 5:10 AM
To: user@hbase.apache.org
Subject: experiencing high latency for few reads in HBase


We are running a stress test in our 5 node cluster and we are getting the expected mean latency
of 10ms. But we are seeing around 20 reads out of 25 million reads having latency more than
4 seconds. Can anyone provide the insight what we can do to meet below second SLA for each
and every read?

We observe the following things -

1. Reads are evenly distributed among 5 nodes.  CPUs remain under 5% utilized.

2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb block cache got filled
up but around 1gb remained free. There are a large number of cache eviction.

Questions to experts -

1. If there are still 1gb of free block cache available, why is hbase evicting the block from

4. We are seeing memory went up to 10gb three times before dropping sharply to 5gb.

Any help is highly appreciable,


Confidentiality Notice:  The information contained in this message, including any attachments
hereto, may be confidential and is intended to be read only by the individual or entity to
whom this message is addressed. If the reader of this message is not the intended recipient
or an agent or designee of the intended recipient, please note that any review, use, disclosure
or distribution of this message or its attachments, in any form, is strictly prohibited. 
If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com
and delete or destroy any copy of this message and its attachments.
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message