hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Tarnas <...@email.com>
Subject Re: Cluster Wide Pauses
Date Fri, 14 Jan 2011 15:28:33 GMT
I have been seeing similar problems and found by raising the
hbase.hregion.memstore.block.multiplier
to above 12 (default is two) and the hbase.hstore.blockingStoreFiles to 16 I
managed to reduce the frequency of the pauses during loads.  My nodes are
pretty beefy (48 GB of ram) so I had room to experiment.

>From what I understand that gave the regionservers more buffer before they
had to halt the world to catch up. The pauses still happen but their impact
is less now.

-chris

On Fri, Jan 14, 2011 at 8:34 AM, Wayne <wav100@gmail.com> wrote:

> We have not found any smoking gun here. Most likely these are region splits
> on a quickly growing/hot region that all clients get caught waiting for.
>
>
> On Thu, Jan 13, 2011 at 7:49 AM, Wayne <wav100@gmail.com> wrote:
>
> > Thank you for the lead! We will definitely look closer at the OS logs.
> >
> >
> > On Thu, Jan 13, 2011 at 6:59 AM, Tatsuya Kawano <tatsuya6502@gmail.com
> >wrote:
> >
> >>
> >> Hi Wayne,
> >>
> >> > We are seeing some TCP Resets on all nodes at the same time, and
> >> sometimes
> >> > quite a lot of them.
> >>
> >>
> >> Have you checked this article from Andrei and Cosmin? They had a busy
> >> firewall to cause network blackout.
> >>
> >> http://hstack.org/hbase-performance-testing/
> >>
> >> Maybe it's not your case but just for sure.
> >>
> >> Thanks,
> >>
> >> --
> >> Tatsuya Kawano (Mr.)
> >> Tokyo, Japan
> >>
> >>
> >> On Jan 13, 2011, at 4:52 AM, Wayne <wav100@gmail.com> wrote:
> >>
> >> > We are seeing some TCP Resets on all nodes at the same time, and
> >> sometimes
> >> > quite a lot of them. We have yet to correlate the pauses to the TCP
> >> resets
> >> > but I am starting to wonder if this is partly a network problem. Does
> >> > Gigabit Ethernet break down on high volume nodes? Do high volume nodes
> >> use
> >> > 10G or Infiniband?
> >> >
> >> >
> >> > On Wed, Jan 12, 2011 at 1:52 PM, Stack <stack@duboce.net> wrote:
> >> >
> >> >> Jon asks that you describe your loading in the issue.  Would you mind
> >> >> doing so.  Ted, stick up in the issue the workload and configs. you
> >> >> are running if you don't mind.  I'd like to try it over here.
> >> >> Thanks lads,
> >> >> St.Ack
> >> >>
> >> >>
> >> >> On Wed, Jan 12, 2011 at 9:03 AM, Wayne <wav100@gmail.com> wrote:
> >> >>> Added: https://issues.apache.org/jira/browse/HBASE-3438.
> >> >>>
> >> >>> On Wed, Jan 12, 2011 at 11:40 AM, Wayne <wav100@gmail.com>
wrote:
> >> >>>
> >> >>>> We are using 0.89.20100924, r1001068
> >> >>>>
> >> >>>> We are seeing see it during heavy write load (which is all
the
> time),
> >> >> but
> >> >>>> yesterday we had read load as well as write load and saw both
reads
> >> and
> >> >>>> writes stop for 10+ seconds. The region size is the biggest
clue we
> >> have
> >> >>>> found from our tests as setting up a new cluster with a 1GB
max
> >> region
> >> >> size
> >> >>>> and starting to load heavily we will see this a lot for long
long
> >> time
> >> >>>> frames. Maybe the bigger file gets hung up more easily with
a
> split?
> >> >> Your
> >> >>>> description below also fits in that early on the load is not
> balanced
> >> so
> >> >> it
> >> >>>> is easier to stop everything on one node as the balance is
not
> great
> >> >> early
> >> >>>> on. I will file a JIRA. I will also try to dig deeper into
the logs
> >> >> during
> >> >>>> the pauses to find a node that might be stuck in a split.
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> On Wed, Jan 12, 2011 at 11:17 AM, Stack <stack@duboce.net>
wrote:
> >> >>>>
> >> >>>>> On Tue, Jan 11, 2011 at 2:34 PM, Wayne <wav100@gmail.com>
wrote:
> >> >>>>>> We have very frequent cluster wide pauses that stop
all reads and
> >> >>>>> writes
> >> >>>>>> for seconds.
> >> >>>>>
> >> >>>>> All reads and all writes?
> >> >>>>>
> >> >>>>> I've seen the pause too for writes.  Its something I've
always
> meant
> >> >>>>> to look into.  Friso postulates one cause.  Another that
we've
> >> talked
> >> >>>>> of is a region taking a while to come back on line after
a split
> or
> >> a
> >> >>>>> rebalance for whatever reason.  Client loading might be
'random'
> >> >>>>> spraying over lots of random regions but they all get stuck
> waiting
> >> on
> >> >>>>> one particular region to come back online.
> >> >>>>>
> >> >>>>> I suppose reads could be blocked for same reason if all
are trying
> >> to
> >> >>>>> read from the offlined region.
> >> >>>>>
> >> >>>>> What version of hbase are you using?  Splits should be
faster in
> >> 0.90
> >> >>>>> now that the split daughters come up on the same region.
> >> >>>>>
> >> >>>>> Sorry I don't have a better answer for you.  Need to dig
in.
> >> >>>>>
> >> >>>>> File a JIRA.  If you want to help out some, stick some
data up in
> >> it.
> >> >>>>> Some suggestions would be to enable logging of when we
lookup
> region
> >> >>>>> locations in client and then note when requests go to zero.
 Can
> you
> >> >>>>> figure what region the clients are waiting on (if they
are waiting
> >> on
> >> >>>>> any).  If you can pull out a particular one, try and elicit
its
> >> >>>>> history at time of blockage.  Is it being moved or mid-split?
 I
> >> >>>>> suppose it makes sense that bigger regions would make the
> situation
> >> >>>>> 'worse'.  I can take a look at it too.
> >> >>>>>
> >> >>>>> St.Ack
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> We are constantly loading data to this cluster of 10 nodes.
> >> >>>>>> These pauses can happen as frequently as every minute
but
> sometimes
> >> >> are
> >> >>>>> not
> >> >>>>>> seen for 15+ minutes. Basically watching the Region
server list
> >> with
> >> >>>>> request
> >> >>>>>> counts is the only evidence of what is going on. All
reads and
> >> writes
> >> >>>>>> totally stop and if there is ever any activity it is
on the node
> >> >> hosting
> >> >>>>> the
> >> >>>>>> .META. table with a request count of region count +
1. This
> problem
> >> >>>>> seems to
> >> >>>>>> be worse with a larger region size. We tried a 1GB
region size
> and
> >> >> saw
> >> >>>>> this
> >> >>>>>> more than we saw actual activity (and stopped using
a larger
> region
> >> >> size
> >> >>>>>> because of it). We went back to the default region
size and it
> was
> >> >>>>> better,
> >> >>>>>> but we had too many regions so now we are up to 512M
for a region
> >> >> size
> >> >>>>> and
> >> >>>>>> we are seeing it more again.
> >> >>>>>>
> >> >>>>>> Does anyone know what this is? We have dug into all
of the logs
> to
> >> >> find
> >> >>>>> some
> >> >>>>>> sort of pause but are not able to find anything. Is
this an wal
> >> hlog
> >> >>>>> roll?
> >> >>>>>> Is this a region split or compaction? Of course our
biggest fear
> is
> >> a
> >> >> GC
> >> >>>>>> pause on the master but we do not have java logging
turned on
> with
> >> >> the
> >> >>>>>> master to tell. What could possibly stop the entire
cluster from
> >> >> working
> >> >>>>> for
> >> >>>>>> seconds at a time very frequently?
> >> >>>>>>
> >> >>>>>> Thanks in advance for any ideas of what could be causing
this.
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message