hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Rodionov <vladrodio...@gmail.com>
Subject Re: Asking about the trend of increasing latency, hbase spikes.
Date Thu, 08 Oct 2015 03:24:46 GMT
Can you trigger full GC across the cluster? If slow queries will go down
after that - you get the answer.
GC s-t-w frequency and duration may increase steadily until full reset
(full GC) is executed, I think

-Vlad

On Wed, Oct 7, 2015 at 6:35 PM, UI-JIN LIM <ujlim@lgcns.com> wrote:

> I have post images to hosting service and share again!!
>
> ---
> Hi. This is Ui Jin, Lim in Korea, LG CNS
>
> We had setup and are operating hbase 0.98.13 on our customer, iptv Company.
> Our customer claim about this spike. Following is our status.
>
> . We had been operating with BlockCache(heap) for 3 months. The ratio of
> above 100ms has been reached 6.69% of total transaction.
> The following graph show the trend of executions, above 100ms per minute,
> at June 17th. (This is the response time of 2~3 Hbase transactions)
> <Click to see Graph>
>  http://i57.tinypic.com/mwcvtj.jpg
>
> - After Hbase restarting at June 18th, The spikes had been reduced
> dramatically. But its ratio had been increasing continuously.
>    <Click to see Graph>
>   http://i57.tinypic.com/71pfee.jpg
>
> - We had adopted off-heap bucket cache at August 19th. But the increasing
> does not disappear.
> <Click to see Graph>
> http://i61.tinypic.com/2aezazt.jpg
> http://i58.tinypic.com/svm32x.jpg
>
> - We set HBase to log slow query which have above 10ms execution time. All
> RegionServer logs show increasing its counts.
> <Click to see Graph>
> http://i59.tinypic.com/68u1ci.jpg
>
> What we concern is not the spike itself but its incremental trend.
> There is no increment of transactions and dataset.
> I ask your thoughts about this issue.
>
>
> -----Original Message-----
> From: Vladimir Rodionov [mailto:vladrodionov@gmail.com]
> Sent: Thursday, October 08, 2015 9:50 AM
> To: dev@hbase.apache.org
> Subject: Re: Asking about the trend of increasing latency, hbase spikes.
>
> That is definitely, interesting graphs, but we do not see them.
> Can you host them somewhere and post here the links?
>
> -Vlad
>
> On Wed, Oct 7, 2015 at 5:38 PM, Biju N <bijuatapache@gmail.com> wrote:
>
> > Looks like the attachments are not viewable.
> >
> > On Wed, Oct 7, 2015 at 8:35 PM, UI-JIN LIM <ujlim@lgcns.com> wrote:
> >
> > > Hi. This is Ui Jin, Lim in Korea, LG CNS
> > >
> > >
> > >
> > > We had setup and are operating hbase 0.98.13 on our customer, iptv
> > > Company.
> > >
> > > Our customer claim about this spike. Following is our status.
> > >
> > >
> > >
> > > . We had been operating with BlockCache(heap) for 3 months. The
> > > ratio of above 100ms has been reached 6.69% of total transaction.
> > >
> > >   The following graph show the trend of executions, above 100ms per
> > > minute, at June 17th. (This is the response time of 2~3 Hbase
> > > transactions)
> > >
> > > [image: cid:image001.jpg@01D1011A.5E93FF70]
> > >
> > >
> > >
> > > - After Hbase restarting at June 18th, The spikes had been reduced
> > > dramatically. But its ratio had been increasing continuously.
> > >
> > > [image: cid:image002.png@01D1011A.5E93FF70]
> > >
> > >
> > >
> > > - We had adopted off-heap bucket cache at August 19th. But the
> > > increasing does not disappear.
> > >
> > > [image: cid:image002.png@01D10103.82E488C0]
> > >
> > > [image: cid:image002.png@01D10103.82E488C0]
> > >
> > >
> > >
> > > - We set HBase to log slow query which have above 10ms execution time.
> > All
> > > RegionServer logs show increasing its counts.
> > >
> > >
> > >
> > > [image: cid:image005.png@01D1011A.5E93FF70]
> > >
> > >
> > >
> > >
> > >
> > > What we concern is not the spike itself but its incremental trend.
> > >
> > > There is no increment of transactions and dataset.
> > >
> > > I ask your thoughts about this issue.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > -------------------------------------------------------------
> > >
> > > *UI JIN, LIM*
> > >
> > > *DNA 3.0** - **LG CNS*
> > >
> > > Data Architecture Advisory
> > >
> > > Big Data Technical Support Team
> > >
> > > Mobile : +82) 010-8307-1210
> > >
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message