hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From UI-JIN LIM <uj...@lgcns.com>
Subject Asking about the trend of increasing latency, hbase spike
Date Thu, 08 Oct 2015 01:24:21 GMT
Hi. This is Ui Jin, Lim in Korea, LG CNS

We had setup and are operating hbase 0.98.13 on our customer, iptv Company.
Our customer claim about this spike. Following is our status.

. We had been operating with BlockCache(heap) for 3 months. The ratio of above 100ms has been
reached 6.69% of total transaction.
The following graph show the trend of executions, above 100ms per minute, at June 17th. (This
is the response time of 2~3 Hbase transactions)
<Click to see Graph>
 http://i57.tinypic.com/mwcvtj.jpg

- After Hbase restarting at June 18th, The spikes had been reduced dramatically. But its ratio
had been increasing continuously.
   <Click to see Graph>
  http://i57.tinypic.com/71pfee.jpg

- We had adopted off-heap bucket cache at August 19th. But the increasing does not disappear.
<Click to see Graph>
http://i61.tinypic.com/2aezazt.jpg
http://i58.tinypic.com/svm32x.jpg

- We set HBase to log slow query which have above 10ms execution time. All RegionServer logs
show increasing its counts.
<Click to see Graph>
http://i59.tinypic.com/68u1ci.jpg


What we concern is not the spike itself but its incremental trend.
There is no increment of transactions and dataset.
I ask your thoughts about this issue.




-------------------------------------------------------------
UI JIN, LIM
DNA 3.0 - LG CNS
Data Architecture Advisory
Big Data Technical Support Team
Mobile : +82) 010-8307-1210

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message