hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Schless <patrick.schl...@gmail.com>
Subject Re: 3-Hour Periodic Network/CPU/Disk/Latency Spikes
Date Fri, 13 Dec 2013 23:18:37 GMT
Very interesting, I think we may be on to something. I grabbed all the
timestamps for major compactions completing and put them on a graph (see
attached). Each horizontal line is an individual server, and the dots are
when compactions complete. Each server clearly has a cluster of compactions
about every 3 hours, and several of the servers are aligned such that they
are compacting at the same time.

Should we be managing these compactions ourselves? Would it make more sense
to have them less frequently (but presumably more expensive), or closer
together?

Thanks,
Patrick


On Fri, Dec 13, 2013 at 2:19 PM, Bryan Beaudreault <bbeaudreault@hubspot.com
> wrote:

> Have you taken a look at the logs on the RegionServers during the period?
>
> One possibility is compactions happening organically.  If you were
> sustaining a certain level of writes most of the time, I could maybe see
> that every 3 hours enough store files build up to require compactions.
>
> There's nothing else automated in HDFS or HBase that I could see causing
> this.
>
> On Fri, Dec 13, 2013 at 3:07 PM, Patrick Schless
> <patrick.schless@gmail.com>wrote:
>
> > CDH4.1.2
> > HBase 0.92.1
> > HDFS 2.0.0
> >
> >
> > Every 3 hours, our production HBase cluster does something that causes
> all
> > the data nodes to have a sustained spike in CPU/network/disk. The spike
> > lasts about 30 mins, and during this time the cluster has greatly
> increased
> > latencies for our typical application usage.
> >
> > I can't find anything in our application that would have such a periodic
> > and significant behavior. Is there anything that HBase/HDFS might be
> doing
> > on it's own that would cause this? We're on the default schedule for
> major
> > compactions, but I thought that was daily.
> >
> > Any ideas what could be causing this?
> >
> > Thanks,
> >
> > Patrick
> >
>

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message