hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saurabh Yahoo <saurabh...@yahoo.com>
Subject experiencing high latency for few reads in HBase
Date Wed, 28 Aug 2013 12:10:58 GMT
Hi,

We are running a stress test in our 5 node cluster and we are getting the expected mean latency
of 10ms. But we are seeing around 20 reads out of 25 million reads having latency more than
4 seconds. Can anyone provide the insight what we can do to meet below second SLA for each
and every read?

We observe the following things - 

1. Reads are evenly distributed among 5 nodes.  CPUs remain under 5% utilized. 

2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb block cache got filled
up but around 1gb remained free. There are a large number of cache eviction.

Questions to experts -

1. If there are still 1gb of free block cache available, why is hbase evicting the block from
cache? 

4. We are seeing memory went up to 10gb three times before dropping sharply to 5gb. 

Any help is highly appreciable,

Thanks,
Saurabh. 
Mime
View raw message