hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Gray <jg...@fb.com>
Subject RE: Cluster Wide Pauses
Date Fri, 14 Jan 2011 17:29:30 GMT
These are a different kind of pause (those caused by blockingStoreFiles).

This is HBase stepping in and actually blocking updates to a region because compactions have
not been able to keep up with the write load.  It could manifest itself in the same way but
this is different than shorter pauses caused by periodic offlining of regions during balancing
and splits.

Wayne, have you confirmed in your RegionServer logs that the pauses are associated with splits
or region movement, and that you are not seeing the blocking store files issue?

JG

> -----Original Message-----
> From: cft@tarnas.org [mailto:cft@tarnas.org] On Behalf Of Christopher
> Tarnas
> Sent: Friday, January 14, 2011 7:29 AM
> To: user@hbase.apache.org
> Subject: Re: Cluster Wide Pauses
> 
> I have been seeing similar problems and found by raising the
> hbase.hregion.memstore.block.multiplier
> to above 12 (default is two) and the hbase.hstore.blockingStoreFiles to 16 I
> managed to reduce the frequency of the pauses during loads.  My nodes are
> pretty beefy (48 GB of ram) so I had room to experiment.
> 
> From what I understand that gave the regionservers more buffer before
> they had to halt the world to catch up. The pauses still happen but their
> impact is less now.
> 
> -chris
> 
> On Fri, Jan 14, 2011 at 8:34 AM, Wayne <wav100@gmail.com> wrote:
> 
> > We have not found any smoking gun here. Most likely these are region
> > splits on a quickly growing/hot region that all clients get caught waiting for.
> >
> >
> > On Thu, Jan 13, 2011 at 7:49 AM, Wayne <wav100@gmail.com> wrote:
> >
> > > Thank you for the lead! We will definitely look closer at the OS logs.
> > >
> > >
> > > On Thu, Jan 13, 2011 at 6:59 AM, Tatsuya Kawano
> > ><tatsuya6502@gmail.com
> > >wrote:
> > >
> > >>
> > >> Hi Wayne,
> > >>
> > >> > We are seeing some TCP Resets on all nodes at the same time, and
> > >> sometimes
> > >> > quite a lot of them.
> > >>
> > >>
> > >> Have you checked this article from Andrei and Cosmin? They had a
> > >> busy firewall to cause network blackout.
> > >>
> > >> http://hstack.org/hbase-performance-testing/
> > >>
> > >> Maybe it's not your case but just for sure.
> > >>
> > >> Thanks,
> > >>
> > >> --
> > >> Tatsuya Kawano (Mr.)
> > >> Tokyo, Japan
> > >>
> > >>
> > >> On Jan 13, 2011, at 4:52 AM, Wayne <wav100@gmail.com> wrote:
> > >>
> > >> > We are seeing some TCP Resets on all nodes at the same time, and
> > >> sometimes
> > >> > quite a lot of them. We have yet to correlate the pauses to the
> > >> > TCP
> > >> resets
> > >> > but I am starting to wonder if this is partly a network problem.
> > >> > Does Gigabit Ethernet break down on high volume nodes? Do high
> > >> > volume nodes
> > >> use
> > >> > 10G or Infiniband?
> > >> >
> > >> >
> > >> > On Wed, Jan 12, 2011 at 1:52 PM, Stack <stack@duboce.net> wrote:
> > >> >
> > >> >> Jon asks that you describe your loading in the issue.  Would you
> > >> >> mind doing so.  Ted, stick up in the issue the workload and
> > >> >> configs. you are running if you don't mind.  I'd like to try it
over here.
> > >> >> Thanks lads,
> > >> >> St.Ack
> > >> >>
> > >> >>
> > >> >> On Wed, Jan 12, 2011 at 9:03 AM, Wayne <wav100@gmail.com>
> wrote:
> > >> >>> Added: https://issues.apache.org/jira/browse/HBASE-3438.
> > >> >>>
> > >> >>> On Wed, Jan 12, 2011 at 11:40 AM, Wayne <wav100@gmail.com>
> wrote:
> > >> >>>
> > >> >>>> We are using 0.89.20100924, r1001068
> > >> >>>>
> > >> >>>> We are seeing see it during heavy write load (which is
all the
> > time),
> > >> >> but
> > >> >>>> yesterday we had read load as well as write load and saw
both
> > >> >>>> reads
> > >> and
> > >> >>>> writes stop for 10+ seconds. The region size is the biggest
> > >> >>>> clue we
> > >> have
> > >> >>>> found from our tests as setting up a new cluster with
a 1GB
> > >> >>>> max
> > >> region
> > >> >> size
> > >> >>>> and starting to load heavily we will see this a lot for
long
> > >> >>>> long
> > >> time
> > >> >>>> frames. Maybe the bigger file gets hung up more easily
with a
> > split?
> > >> >> Your
> > >> >>>> description below also fits in that early on the load
is not
> > balanced
> > >> so
> > >> >> it
> > >> >>>> is easier to stop everything on one node as the balance
is not
> > great
> > >> >> early
> > >> >>>> on. I will file a JIRA. I will also try to dig deeper
into the
> > >> >>>> logs
> > >> >> during
> > >> >>>> the pauses to find a node that might be stuck in a split.
> > >> >>>>
> > >> >>>>
> > >> >>>>
> > >> >>>> On Wed, Jan 12, 2011 at 11:17 AM, Stack <stack@duboce.net>
> wrote:
> > >> >>>>
> > >> >>>>> On Tue, Jan 11, 2011 at 2:34 PM, Wayne <wav100@gmail.com>
> wrote:
> > >> >>>>>> We have very frequent cluster wide pauses that
stop all
> > >> >>>>>> reads and
> > >> >>>>> writes
> > >> >>>>>> for seconds.
> > >> >>>>>
> > >> >>>>> All reads and all writes?
> > >> >>>>>
> > >> >>>>> I've seen the pause too for writes.  Its something
I've
> > >> >>>>> always
> > meant
> > >> >>>>> to look into.  Friso postulates one cause.  Another
that
> > >> >>>>> we've
> > >> talked
> > >> >>>>> of is a region taking a while to come back on line
after a
> > >> >>>>> split
> > or
> > >> a
> > >> >>>>> rebalance for whatever reason.  Client loading might
be 'random'
> > >> >>>>> spraying over lots of random regions but they all
get stuck
> > waiting
> > >> on
> > >> >>>>> one particular region to come back online.
> > >> >>>>>
> > >> >>>>> I suppose reads could be blocked for same reason if
all are
> > >> >>>>> trying
> > >> to
> > >> >>>>> read from the offlined region.
> > >> >>>>>
> > >> >>>>> What version of hbase are you using?  Splits should
be faster
> > >> >>>>> in
> > >> 0.90
> > >> >>>>> now that the split daughters come up on the same region.
> > >> >>>>>
> > >> >>>>> Sorry I don't have a better answer for you.  Need
to dig in.
> > >> >>>>>
> > >> >>>>> File a JIRA.  If you want to help out some, stick
some data
> > >> >>>>> up in
> > >> it.
> > >> >>>>> Some suggestions would be to enable logging of when
we
> lookup
> > region
> > >> >>>>> locations in client and then note when requests go
to zero.
> > >> >>>>> Can
> > you
> > >> >>>>> figure what region the clients are waiting on (if
they are
> > >> >>>>> waiting
> > >> on
> > >> >>>>> any).  If you can pull out a particular one, try and
elicit
> > >> >>>>> its history at time of blockage.  Is it being moved
or
> > >> >>>>> mid-split?  I suppose it makes sense that bigger regions
> > >> >>>>> would make the
> > situation
> > >> >>>>> 'worse'.  I can take a look at it too.
> > >> >>>>>
> > >> >>>>> St.Ack
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> We are constantly loading data to this cluster of
10 nodes.
> > >> >>>>>> These pauses can happen as frequently as every
minute but
> > sometimes
> > >> >> are
> > >> >>>>> not
> > >> >>>>>> seen for 15+ minutes. Basically watching the Region
server
> > >> >>>>>> list
> > >> with
> > >> >>>>> request
> > >> >>>>>> counts is the only evidence of what is going on.
All reads
> > >> >>>>>> and
> > >> writes
> > >> >>>>>> totally stop and if there is ever any activity
it is on the
> > >> >>>>>> node
> > >> >> hosting
> > >> >>>>> the
> > >> >>>>>> .META. table with a request count of region count
+ 1. This
> > problem
> > >> >>>>> seems to
> > >> >>>>>> be worse with a larger region size. We tried a
1GB region
> > >> >>>>>> size
> > and
> > >> >> saw
> > >> >>>>> this
> > >> >>>>>> more than we saw actual activity (and stopped
using a larger
> > region
> > >> >> size
> > >> >>>>>> because of it). We went back to the default region
size and
> > >> >>>>>> it
> > was
> > >> >>>>> better,
> > >> >>>>>> but we had too many regions so now we are up to
512M for a
> > >> >>>>>> region
> > >> >> size
> > >> >>>>> and
> > >> >>>>>> we are seeing it more again.
> > >> >>>>>>
> > >> >>>>>> Does anyone know what this is? We have dug into
all of the
> > >> >>>>>> logs
> > to
> > >> >> find
> > >> >>>>> some
> > >> >>>>>> sort of pause but are not able to find anything.
Is this an
> > >> >>>>>> wal
> > >> hlog
> > >> >>>>> roll?
> > >> >>>>>> Is this a region split or compaction? Of course
our biggest
> > >> >>>>>> fear
> > is
> > >> a
> > >> >> GC
> > >> >>>>>> pause on the master but we do not have java logging
turned
> > >> >>>>>> on
> > with
> > >> >> the
> > >> >>>>>> master to tell. What could possibly stop the entire
cluster
> > >> >>>>>> from
> > >> >> working
> > >> >>>>> for
> > >> >>>>>> seconds at a time very frequently?
> > >> >>>>>>
> > >> >>>>>> Thanks in advance for any ideas of what could
be causing this.
> > >> >>>>>>
> > >> >>>>>
> > >> >>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> > >>
> > >
> >

Mime
View raw message