hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Schless <patrick.schl...@gmail.com>
Subject Re: 3-Hour Periodic Network/CPU/Disk/Latency Spikes
Date Sat, 14 Dec 2013 00:36:48 GMT
Ah, sorry about the attachment (didn't realize they weren't allowed).
Here's the picture I was trying to attach:
http://www.plainlystated.com/hbase_major_compactions.png

It sounds like you're right, Vladimir, about the compaction storm, though I
don't understand why it's about every three hours instead of every day. In
the book [1] I see the suggestion that they be managed manually. I don't
see, however, any advice on what do do after turning auto-compaction off.
Are there best practices around scheduling and monitoring the process?

Thanks,
Patrick

[1]
http://hbase.apache.org/book/important_configurations.html#managed.compactions


On Fri, Dec 13, 2013 at 6:07 PM, Vladimir Rodionov
<vrodionov@carrieriq.com>wrote:

> You forgot to mention that it won't go through because Apache mail server
> blocks attachments.
> What Patrick is observing is called compaction storms. The best way (as
> since its 0.92.x) is to disable automatic compactions
> and manage them manually (see HBase book how to do this).
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: Ted Yu [yuzhihong@gmail.com]
> Sent: Friday, December 13, 2013 3:33 PM
> To: user@hbase.apache.org
> Cc: user
> Subject: Re: 3-Hour Periodic Network/CPU/Disk/Latency Spikes
>
> Patrick:
> Attachment didn't go through.
>
> Cheers
>
> On Dec 13, 2013, at 3:18 PM, Patrick Schless <patrick.schless@gmail.com>
> wrote:
>
> > Very interesting, I think we may be on to something. I grabbed all the
> timestamps for major compactions completing and put them on a graph (see
> attached). Each horizontal line is an individual server, and the dots are
> when compactions complete. Each server clearly has a cluster of compactions
> about every 3 hours, and several of the servers are aligned such that they
> are compacting at the same time.
> >
> > Should we be managing these compactions ourselves? Would it make more
> sense to have them less frequently (but presumably more expensive), or
> closer together?
> >
> > Thanks,
> > Patrick
> >
> >
> > On Fri, Dec 13, 2013 at 2:19 PM, Bryan Beaudreault <
> bbeaudreault@hubspot.com> wrote:
> >> Have you taken a look at the logs on the RegionServers during the
> period?
> >>
> >> One possibility is compactions happening organically.  If you were
> >> sustaining a certain level of writes most of the time, I could maybe see
> >> that every 3 hours enough store files build up to require compactions.
> >>
> >> There's nothing else automated in HDFS or HBase that I could see causing
> >> this.
> >>
> >> On Fri, Dec 13, 2013 at 3:07 PM, Patrick Schless
> >> <patrick.schless@gmail.com>wrote:
> >>
> >> > CDH4.1.2
> >> > HBase 0.92.1
> >> > HDFS 2.0.0
> >> >
> >> >
> >> > Every 3 hours, our production HBase cluster does something that
> causes all
> >> > the data nodes to have a sustained spike in CPU/network/disk. The
> spike
> >> > lasts about 30 mins, and during this time the cluster has greatly
> increased
> >> > latencies for our typical application usage.
> >> >
> >> > I can't find anything in our application that would have such a
> periodic
> >> > and significant behavior. Is there anything that HBase/HDFS might be
> doing
> >> > on it's own that would cause this? We're on the default schedule for
> major
> >> > compactions, but I thought that was daily.
> >> >
> >> > Any ideas what could be causing this?
> >> >
> >> > Thanks,
> >> >
> >> > Patrick
> >> >
> >
>
> Confidentiality Notice:  The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited.  If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message