hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From UI-JIN LIM <uj...@lgcns.com>
Subject Asking about the trend of increasing latency, hbase spikes.
Date Thu, 08 Oct 2015 00:35:09 GMT
Hi. This is Ui Jin, Lim in Korea, LG CNS

We had setup and are operating hbase 0.98.13 on our customer, iptv Company.
Our customer claim about this spike. Following is our status.

. We had been operating with BlockCache(heap) for 3 months. The ratio of above 100ms has been
reached 6.69% of total transaction.
  The following graph show the trend of executions, above 100ms per minute, at June 17th.
(This is the response time of 2~3 Hbase transactions)
[cid:image001.jpg@01D101AC.9FCDC380]

- After Hbase restarting at June 18th, The spikes had been reduced dramatically. But its ratio
had been increasing continuously.
[cid:image002.png@01D101AC.9FCDC380]

- We had adopted off-heap bucket cache at August 19th. But the increasing does not disappear.
[cid:image003.jpg@01D101AC.9FCDC380]
[cid:image004.jpg@01D101AC.9FCDC380]

- We set HBase to log slow query which have above 10ms execution time. All RegionServer logs
show increasing its counts.

[cid:image005.png@01D101AC.9FCDC380]


What we concern is not the spike itself but its incremental trend.
There is no increment of transactions and dataset.
I ask your thoughts about this issue.




-------------------------------------------------------------
UI JIN, LIM
DNA 3.0 - LG CNS
Data Architecture Advisory
Big Data Technical Support Team
Mobile : +82) 010-8307-1210

Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message