hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From UI-JIN LIM <uj...@lgcns.com>
Subject RE: Asking about the trend of increasing latency, hbase spikes.
Date Thu, 08 Oct 2015 01:35:20 GMT
I have post images to hosting service and share again!!

--- 
Hi. This is Ui Jin, Lim in Korea, LG CNS

We had setup and are operating hbase 0.98.13 on our customer, iptv Company. 
Our customer claim about this spike. Following is our status.

. We had been operating with BlockCache(heap) for 3 months. The ratio of above 100ms has been
reached 6.69% of total transaction.
The following graph show the trend of executions, above 100ms per minute, at June 17th. (This
is the response time of 2~3 Hbase transactions)
<Click to see Graph> 
 http://i57.tinypic.com/mwcvtj.jpg

- After Hbase restarting at June 18th, The spikes had been reduced dramatically. But its ratio
had been increasing continuously.
   <Click to see Graph>
  http://i57.tinypic.com/71pfee.jpg

- We had adopted off-heap bucket cache at August 19th. But the increasing does not disappear.
<Click to see Graph>
http://i61.tinypic.com/2aezazt.jpg
http://i58.tinypic.com/svm32x.jpg

- We set HBase to log slow query which have above 10ms execution time. All RegionServer logs
show increasing its counts.
<Click to see Graph>
http://i59.tinypic.com/68u1ci.jpg

What we concern is not the spike itself but its incremental trend. 
There is no increment of transactions and dataset.
I ask your thoughts about this issue.


-----Original Message-----
From: Vladimir Rodionov [mailto:vladrodionov@gmail.com] 
Sent: Thursday, October 08, 2015 9:50 AM
To: dev@hbase.apache.org
Subject: Re: Asking about the trend of increasing latency, hbase spikes.

That is definitely, interesting graphs, but we do not see them.
Can you host them somewhere and post here the links?

-Vlad

On Wed, Oct 7, 2015 at 5:38 PM, Biju N <bijuatapache@gmail.com> wrote:

> Looks like the attachments are not viewable.
>
> On Wed, Oct 7, 2015 at 8:35 PM, UI-JIN LIM <ujlim@lgcns.com> wrote:
>
> > Hi. This is Ui Jin, Lim in Korea, LG CNS
> >
> >
> >
> > We had setup and are operating hbase 0.98.13 on our customer, iptv 
> > Company.
> >
> > Our customer claim about this spike. Following is our status.
> >
> >
> >
> > . We had been operating with BlockCache(heap) for 3 months. The 
> > ratio of above 100ms has been reached 6.69% of total transaction.
> >
> >   The following graph show the trend of executions, above 100ms per 
> > minute, at June 17th. (This is the response time of 2~3 Hbase
> > transactions)
> >
> > [image: cid:image001.jpg@01D1011A.5E93FF70]
> >
> >
> >
> > - After Hbase restarting at June 18th, The spikes had been reduced 
> > dramatically. But its ratio had been increasing continuously.
> >
> > [image: cid:image002.png@01D1011A.5E93FF70]
> >
> >
> >
> > - We had adopted off-heap bucket cache at August 19th. But the 
> > increasing does not disappear.
> >
> > [image: cid:image002.png@01D10103.82E488C0]
> >
> > [image: cid:image002.png@01D10103.82E488C0]
> >
> >
> >
> > - We set HBase to log slow query which have above 10ms execution time.
> All
> > RegionServer logs show increasing its counts.
> >
> >
> >
> > [image: cid:image005.png@01D1011A.5E93FF70]
> >
> >
> >
> >
> >
> > What we concern is not the spike itself but its incremental trend.
> >
> > There is no increment of transactions and dataset.
> >
> > I ask your thoughts about this issue.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > -------------------------------------------------------------
> >
> > *UI JIN, LIM*
> >
> > *DNA 3.0** - **LG CNS*
> >
> > Data Architecture Advisory
> >
> > Big Data Technical Support Team
> >
> > Mobile : +82) 010-8307-1210
> >
> >
> >
>
Mime
View raw message