hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abe Weinograd <...@flonet.com>
Subject Re: Region balancing query
Date Wed, 18 Feb 2015 19:04:15 GMT
Great.  Thanks for the clarification.  In 0.98.1 I thought we saw a
different behavior and regions were distributed when we split.

Thanks again for the help.

Abe

On Wed, Feb 18, 2015 at 12:01 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> When a region splits, the two daughter regions stay on the same host as the
> parent region did.
>
> It is through balancing that the daughter regions (may) be assigned to
> other server(s).
>
> Cheers
>
> On Wed, Feb 18, 2015 at 7:35 AM, Abe Weinograd <abe@flonet.com> wrote:
>
> > Another question, would this help when we split or only when we balance?
> >
> > Thanks,
> > Abe
> >
> > On Mon, Feb 16, 2015 at 11:24 AM, Abe Weinograd <abe@flonet.com> wrote:
> >
> > > balancer said "true" and it is not disabled.  Thanks again for your
> help.
> > >
> > > Abe
> > >
> > > On Mon, Feb 16, 2015 at 11:23 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > >> What was the output from 'balancer' command ?
> > >>
> > >> Was is possible that balancer was disabled ?
> > >>
> > >> Cheers
> > >>
> > >> On Mon, Feb 16, 2015 at 8:04 AM, Abe Weinograd <abe@flonet.com>
> wrote:
> > >>
> > >> > Ok.  I forced the balancer run and am not seeing anything after a
> few
> > >> > minutes.  Master logs isn't showing anything.  Should I look at the
> RS
> > >> ones
> > >> > instead?
> > >> >
> > >> > On Mon, Feb 16, 2015 at 11:03 AM, Ted Yu <yuzhihong@gmail.com>
> wrote:
> > >> >
> > >> > > You should see effect in the next balancer run.
> > >> > >
> > >> > > Cheers
> > >> > >
> > >> > > On Mon, Feb 16, 2015 at 7:52 AM, Abe Weinograd <abe@flonet.com>
> > >> wrote:
> > >> > >
> > >> > > > Excellent. If i trigger the balancer, should this start
to help
> or
> > >> only
> > >> > > for
> > >> > > > future region creation?
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Abe
> > >> > > >
> > >> > > > On Mon, Feb 16, 2015 at 9:35 AM, Ted Yu <yuzhihong@gmail.com>
> > >> wrote:
> > >> > > >
> > >> > > > > Yes. This setting should be modified on Master.
> > >> > > > >
> > >> > > > > Cheers
> > >> > > > >
> > >> > > > > On Mon, Feb 16, 2015 at 6:27 AM, Abe Weinograd <
> abe@flonet.com>
> > >> > wrote:
> > >> > > > >
> > >> > > > > > Thanks Ted.  We are putting this in the hbase-site.xml
for
> the
> > >> > > Master?
> > >> > > > > >
> > >> > > > > > Abe
> > >> > > > > >
> > >> > > > > > On Fri, Feb 13, 2015 at 5:03 PM, Shahab Yunus
<
> > >> > > shahab.yunus@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Thanks, we will try that and report back.
> > >> > > > > > >
> > >> > > > > > > Regards,
> > >> > > > > > > Shahab
> > >> > > > > > >
> > >> > > > > > > On Fri, Feb 13, 2015 at 4:56 PM, Ted Yu <
> > yuzhihong@gmail.com>
> > >> > > wrote:
> > >> > > > > > >
> > >> > > > > > > > You can make TableSkewCostFunction more
prominent by
> > >> increasing
> > >> > > the
> > >> > > > > > value
> > >> > > > > > > > for config parameter:
> > >> > > > > > > >
> > >> > > > > > > > hbase.master.balancer.stochastic.tableSkewCost
> > >> > > > > > > >
> > >> > > > > > > > Its default is 35.
> > >> > > > > > > >
> > >> > > > > > > > See if raising to 100 or 200 helps.
> > >> > > > > > > >
> > >> > > > > > > > On Fri, Feb 13, 2015 at 1:09 PM, Shahab
Yunus <
> > >> > > > > shahab.yunus@gmail.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Yes, this sever hosts other regions
from other tables
> as
> > >> > well.
> > >> > > > > > > > >
> > >> > > > > > > > > Regards
> > >> > > > > > > > > Shahab
> > >> > > > > > > > >
> > >> > > > > > > > > On Fri, Feb 13, 2015 at 1:45 PM,
Ted Yu <
> > >> yuzhihong@gmail.com
> > >> > >
> > >> > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > Interesting,
> server7.ec3.internal,60020,1423845018628
> > >> was
> > >> > > > > > > consistently
> > >> > > > > > > > > > chosen as destination for
the table.
> > >> > > > > > > > > > Did server7.ec3.internal,60020,1423845018628
host
> > >> regions
> > >> > > from
> > >> > > > > > other
> > >> > > > > > > > > table
> > >> > > > > > > > > > ?
> > >> > > > > > > > > >
> > >> > > > > > > > > > Cheers
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Fri, Feb 13, 2015 at 10:27
AM, Shahab Yunus <
> > >> > > > > > > shahab.yunus@gmail.com
> > >> > > > > > > > >
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Table name is:
> > >> > > > > > > > > > > MYTABLE_RECENT_4W_V2
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Pastebin snippet 1: http://pastebin.com/dQzMhGyP
> > >> > > > > > > > > > > Pastebin snippet 2: http://pastebin.com/Y7ZsNAgF
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > This is the master log
after invoking balancer
> > command
> > >> > from
> > >> > > > > hbase
> > >> > > > > > > > > shell.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Regards,
> > >> > > > > > > > > > > Shahab
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Fri, Feb 13, 2015
at 12:00 PM, Ted Yu <
> > >> > > > yuzhihong@gmail.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > > bq. all the regions
of this table were back on
> > this
> > >> > same
> > >> > > > RS!
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Interesting. Please
check master log around the
> > time
> > >> > this
> > >> > > > RS
> > >> > > > > > was
> > >> > > > > > > > > > brought
> > >> > > > > > > > > > > > online. You can
pastebin the relevant snippet.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Thanks
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > On Fri, Feb 13,
2015 at 8:55 AM, Shahab Yunus <
> > >> > > > > > > > > shahab.yunus@gmail.com>
> > >> > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > > Hi Ted.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Yes, the cluster
itself is balanced. On
> average
> > >> 300
> > >> > > > regions
> > >> > > > > > per
> > >> > > > > > > > > node
> > >> > > > > > > > > > on
> > >> > > > > > > > > > > > 10
> > >> > > > > > > > > > > > > nodes.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > # of tables
is 53 of varying sizes.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Balancer was
invoked and it didn't do anything
> > >> (i.e.
> > >> > no
> > >> > > > > > > movement
> > >> > > > > > > > of
> > >> > > > > > > > > > > > > regions) but
we didn't check the master's
> logs.
> > We
> > >> > can
> > >> > > do
> > >> > > > > > that.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Interestingly,
we restarted the RS which was
> > >> holding
> > >> > > all
> > >> > > > > the
> > >> > > > > > > > > regions
> > >> > > > > > > > > > of
> > >> > > > > > > > > > > > > this one table.
The regions were nicely spread
> > >> out to
> > >> > > the
> > >> > > > > > > > remaining
> > >> > > > > > > > > > RS.
> > >> > > > > > > > > > > > But
> > >> > > > > > > > > > > > > when we brought
back this RS, all the regions
> of
> > >> this
> > >> > > > table
> > >> > > > > > > were
> > >> > > > > > > > > back
> > >> > > > > > > > > > > on
> > >> > > > > > > > > > > > > this same RS!
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Thanks.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Regards,
> > >> > > > > > > > > > > > > Shahab
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > On Fri, Feb
13, 2015 at 11:46 AM, Ted Yu <
> > >> > > > > > yuzhihong@gmail.com>
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > How many
tables are there in your cluster ?
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Is the
cluster balanced overall (in terms of
> > >> number
> > >> > > of
> > >> > > > > > > regions
> > >> > > > > > > > > per
> > >> > > > > > > > > > > > > server)
> > >> > > > > > > > > > > > > > but this
table is not ?
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > What happens
(check master log) when you
> issue
> > >> > > > 'balancer'
> > >> > > > > > > > command
> > >> > > > > > > > > > > > through
> > >> > > > > > > > > > > > > > shell
?
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Cheers
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > On Fri,
Feb 13, 2015 at 8:19 AM, Shahab
> Yunus
> > <
> > >> > > > > > > > > > > shahab.yunus@gmail.com>
> > >> > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > CDH
5.3
> > >> > > > > > > > > > > > > > > HBase
98.6
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > We
are writing data to an HBase table
> > through
> > >> a
> > >> > M/R
> > >> > > > > job.
> > >> > > > > > We
> > >> > > > > > > > pre
> > >> > > > > > > > > > > split
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > table
before each job run. The problem is
> > that
> > >> > most
> > >> > > > of
> > >> > > > > > the
> > >> > > > > > > > > > regions
> > >> > > > > > > > > > > > end
> > >> > > > > > > > > > > > > up
> > >> > > > > > > > > > > > > > > on
the same RS. This results in that one
> RS
> > >> being
> > >> > > > > > severely
> > >> > > > > > > > > > > overloaded
> > >> > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > subsequent
M/R jobs failing trying to
> write
> > to
> > >> > the
> > >> > > > > > regions
> > >> > > > > > > on
> > >> > > > > > > > > > that
> > >> > > > > > > > > > > > RS.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > The
balancer is on and the split policy is
> > >> > default.
> > >> > > > No
> > >> > > > > > > > changes
> > >> > > > > > > > > > > there.
> > >> > > > > > > > > > > > > It
> > >> > > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > a
10 node cluster.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > All
other related properties are defaults
> > too.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Any
idea, how can we force balancing of
> the
> > >> new
> > >> > > > > regions?
> > >> > > > > > Do
> > >> > > > > > > > we
> > >> > > > > > > > > > have
> > >> > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > consider
compaction into the equation as
> > well?
> > >> > > > Thanks.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Regards,
> > >> > > > > > > > > > > > > > > Shahab
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message