hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Region balancing query
Date Wed, 18 Feb 2015 17:01:14 GMT
When a region splits, the two daughter regions stay on the same host as the
parent region did.

It is through balancing that the daughter regions (may) be assigned to
other server(s).

Cheers

On Wed, Feb 18, 2015 at 7:35 AM, Abe Weinograd <abe@flonet.com> wrote:

> Another question, would this help when we split or only when we balance?
>
> Thanks,
> Abe
>
> On Mon, Feb 16, 2015 at 11:24 AM, Abe Weinograd <abe@flonet.com> wrote:
>
> > balancer said "true" and it is not disabled.  Thanks again for your help.
> >
> > Abe
> >
> > On Mon, Feb 16, 2015 at 11:23 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> >> What was the output from 'balancer' command ?
> >>
> >> Was is possible that balancer was disabled ?
> >>
> >> Cheers
> >>
> >> On Mon, Feb 16, 2015 at 8:04 AM, Abe Weinograd <abe@flonet.com> wrote:
> >>
> >> > Ok.  I forced the balancer run and am not seeing anything after a few
> >> > minutes.  Master logs isn't showing anything.  Should I look at the RS
> >> ones
> >> > instead?
> >> >
> >> > On Mon, Feb 16, 2015 at 11:03 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >> >
> >> > > You should see effect in the next balancer run.
> >> > >
> >> > > Cheers
> >> > >
> >> > > On Mon, Feb 16, 2015 at 7:52 AM, Abe Weinograd <abe@flonet.com>
> >> wrote:
> >> > >
> >> > > > Excellent. If i trigger the balancer, should this start to help
or
> >> only
> >> > > for
> >> > > > future region creation?
> >> > > >
> >> > > > Thanks,
> >> > > > Abe
> >> > > >
> >> > > > On Mon, Feb 16, 2015 at 9:35 AM, Ted Yu <yuzhihong@gmail.com>
> >> wrote:
> >> > > >
> >> > > > > Yes. This setting should be modified on Master.
> >> > > > >
> >> > > > > Cheers
> >> > > > >
> >> > > > > On Mon, Feb 16, 2015 at 6:27 AM, Abe Weinograd <abe@flonet.com>
> >> > wrote:
> >> > > > >
> >> > > > > > Thanks Ted.  We are putting this in the hbase-site.xml
for the
> >> > > Master?
> >> > > > > >
> >> > > > > > Abe
> >> > > > > >
> >> > > > > > On Fri, Feb 13, 2015 at 5:03 PM, Shahab Yunus <
> >> > > shahab.yunus@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Thanks, we will try that and report back.
> >> > > > > > >
> >> > > > > > > Regards,
> >> > > > > > > Shahab
> >> > > > > > >
> >> > > > > > > On Fri, Feb 13, 2015 at 4:56 PM, Ted Yu <
> yuzhihong@gmail.com>
> >> > > wrote:
> >> > > > > > >
> >> > > > > > > > You can make TableSkewCostFunction more prominent
by
> >> increasing
> >> > > the
> >> > > > > > value
> >> > > > > > > > for config parameter:
> >> > > > > > > >
> >> > > > > > > > hbase.master.balancer.stochastic.tableSkewCost
> >> > > > > > > >
> >> > > > > > > > Its default is 35.
> >> > > > > > > >
> >> > > > > > > > See if raising to 100 or 200 helps.
> >> > > > > > > >
> >> > > > > > > > On Fri, Feb 13, 2015 at 1:09 PM, Shahab Yunus
<
> >> > > > > shahab.yunus@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Yes, this sever hosts other regions
from other tables as
> >> > well.
> >> > > > > > > > >
> >> > > > > > > > > Regards
> >> > > > > > > > > Shahab
> >> > > > > > > > >
> >> > > > > > > > > On Fri, Feb 13, 2015 at 1:45 PM, Ted
Yu <
> >> yuzhihong@gmail.com
> >> > >
> >> > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Interesting, server7.ec3.internal,60020,1423845018628
> >> was
> >> > > > > > > consistently
> >> > > > > > > > > > chosen as destination for the table.
> >> > > > > > > > > > Did server7.ec3.internal,60020,1423845018628
host
> >> regions
> >> > > from
> >> > > > > > other
> >> > > > > > > > > table
> >> > > > > > > > > > ?
> >> > > > > > > > > >
> >> > > > > > > > > > Cheers
> >> > > > > > > > > >
> >> > > > > > > > > > On Fri, Feb 13, 2015 at 10:27 AM,
Shahab Yunus <
> >> > > > > > > shahab.yunus@gmail.com
> >> > > > > > > > >
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Table name is:
> >> > > > > > > > > > > MYTABLE_RECENT_4W_V2
> >> > > > > > > > > > >
> >> > > > > > > > > > > Pastebin snippet 1: http://pastebin.com/dQzMhGyP
> >> > > > > > > > > > > Pastebin snippet 2: http://pastebin.com/Y7ZsNAgF
> >> > > > > > > > > > >
> >> > > > > > > > > > > This is the master log after
invoking balancer
> command
> >> > from
> >> > > > > hbase
> >> > > > > > > > > shell.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Regards,
> >> > > > > > > > > > > Shahab
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Fri, Feb 13, 2015 at 12:00
PM, Ted Yu <
> >> > > > yuzhihong@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > bq. all the regions of
this table were back on
> this
> >> > same
> >> > > > RS!
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Interesting. Please check
master log around the
> time
> >> > this
> >> > > > RS
> >> > > > > > was
> >> > > > > > > > > > brought
> >> > > > > > > > > > > > online. You can pastebin
the relevant snippet.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Thanks
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Fri, Feb 13, 2015
at 8:55 AM, Shahab Yunus <
> >> > > > > > > > > shahab.yunus@gmail.com>
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Hi Ted.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Yes, the cluster
itself is balanced. On average
> >> 300
> >> > > > regions
> >> > > > > > per
> >> > > > > > > > > node
> >> > > > > > > > > > on
> >> > > > > > > > > > > > 10
> >> > > > > > > > > > > > > nodes.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > # of tables is 53
of varying sizes.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Balancer was invoked
and it didn't do anything
> >> (i.e.
> >> > no
> >> > > > > > > movement
> >> > > > > > > > of
> >> > > > > > > > > > > > > regions) but we
didn't check the master's logs.
> We
> >> > can
> >> > > do
> >> > > > > > that.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Interestingly, we
restarted the RS which was
> >> holding
> >> > > all
> >> > > > > the
> >> > > > > > > > > regions
> >> > > > > > > > > > of
> >> > > > > > > > > > > > > this one table.
The regions were nicely spread
> >> out to
> >> > > the
> >> > > > > > > > remaining
> >> > > > > > > > > > RS.
> >> > > > > > > > > > > > But
> >> > > > > > > > > > > > > when we brought
back this RS, all the regions of
> >> this
> >> > > > table
> >> > > > > > > were
> >> > > > > > > > > back
> >> > > > > > > > > > > on
> >> > > > > > > > > > > > > this same RS!
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Thanks.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Regards,
> >> > > > > > > > > > > > > Shahab
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > On Fri, Feb 13,
2015 at 11:46 AM, Ted Yu <
> >> > > > > > yuzhihong@gmail.com>
> >> > > > > > > > > > wrote:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > > How many tables
are there in your cluster ?
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Is the cluster
balanced overall (in terms of
> >> number
> >> > > of
> >> > > > > > > regions
> >> > > > > > > > > per
> >> > > > > > > > > > > > > server)
> >> > > > > > > > > > > > > > but this table
is not ?
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > What happens
(check master log) when you issue
> >> > > > 'balancer'
> >> > > > > > > > command
> >> > > > > > > > > > > > through
> >> > > > > > > > > > > > > > shell ?
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Cheers
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > On Fri, Feb
13, 2015 at 8:19 AM, Shahab Yunus
> <
> >> > > > > > > > > > > shahab.yunus@gmail.com>
> >> > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > CDH 5.3
> >> > > > > > > > > > > > > > > HBase
98.6
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > We are
writing data to an HBase table
> through
> >> a
> >> > M/R
> >> > > > > job.
> >> > > > > > We
> >> > > > > > > > pre
> >> > > > > > > > > > > split
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > table
before each job run. The problem is
> that
> >> > most
> >> > > > of
> >> > > > > > the
> >> > > > > > > > > > regions
> >> > > > > > > > > > > > end
> >> > > > > > > > > > > > > up
> >> > > > > > > > > > > > > > > on the
same RS. This results in that one RS
> >> being
> >> > > > > > severely
> >> > > > > > > > > > > overloaded
> >> > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > subsequent
M/R jobs failing trying to write
> to
> >> > the
> >> > > > > > regions
> >> > > > > > > on
> >> > > > > > > > > > that
> >> > > > > > > > > > > > RS.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > The balancer
is on and the split policy is
> >> > default.
> >> > > > No
> >> > > > > > > > changes
> >> > > > > > > > > > > there.
> >> > > > > > > > > > > > > It
> >> > > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > > a 10 node
cluster.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > All other
related properties are defaults
> too.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Any idea,
how can we force balancing of the
> >> new
> >> > > > > regions?
> >> > > > > > Do
> >> > > > > > > > we
> >> > > > > > > > > > have
> >> > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > consider
compaction into the equation as
> well?
> >> > > > Thanks.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Regards,
> >> > > > > > > > > > > > > > > Shahab
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message