hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Region balancing query
Date Fri, 13 Feb 2015 21:56:30 GMT
You can make TableSkewCostFunction more prominent by increasing the value
for config parameter:

hbase.master.balancer.stochastic.tableSkewCost

Its default is 35.

See if raising to 100 or 200 helps.

On Fri, Feb 13, 2015 at 1:09 PM, Shahab Yunus <shahab.yunus@gmail.com>
wrote:

> Yes, this sever hosts other regions from other tables as well.
>
> Regards
> Shahab
>
> On Fri, Feb 13, 2015 at 1:45 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > Interesting, server7.ec3.internal,60020,1423845018628 was consistently
> > chosen as destination for the table.
> > Did server7.ec3.internal,60020,1423845018628 host regions from other
> table
> > ?
> >
> > Cheers
> >
> > On Fri, Feb 13, 2015 at 10:27 AM, Shahab Yunus <shahab.yunus@gmail.com>
> > wrote:
> >
> > > Table name is:
> > > MYTABLE_RECENT_4W_V2
> > >
> > > Pastebin snippet 1: http://pastebin.com/dQzMhGyP
> > > Pastebin snippet 2: http://pastebin.com/Y7ZsNAgF
> > >
> > > This is the master log after invoking balancer command from hbase
> shell.
> > >
> > > Regards,
> > > Shahab
> > >
> > > On Fri, Feb 13, 2015 at 12:00 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > bq. all the regions of this table were back on this same RS!
> > > >
> > > > Interesting. Please check master log around the time this RS was
> > brought
> > > > online. You can pastebin the relevant snippet.
> > > >
> > > > Thanks
> > > >
> > > > On Fri, Feb 13, 2015 at 8:55 AM, Shahab Yunus <
> shahab.yunus@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Ted.
> > > > >
> > > > > Yes, the cluster itself is balanced. On average 300 regions per
> node
> > on
> > > > 10
> > > > > nodes.
> > > > >
> > > > > # of tables is 53 of varying sizes.
> > > > >
> > > > > Balancer was invoked and it didn't do anything (i.e. no movement
of
> > > > > regions) but we didn't check the master's logs. We can do that.
> > > > >
> > > > > Interestingly, we restarted the RS which was holding all the
> regions
> > of
> > > > > this one table. The regions were nicely spread out to the remaining
> > RS.
> > > > But
> > > > > when we brought back this RS, all the regions of this table were
> back
> > > on
> > > > > this same RS!
> > > > >
> > > > > Thanks.
> > > > >
> > > > >
> > > > > Regards,
> > > > > Shahab
> > > > >
> > > > > On Fri, Feb 13, 2015 at 11:46 AM, Ted Yu <yuzhihong@gmail.com>
> > wrote:
> > > > >
> > > > > > How many tables are there in your cluster ?
> > > > > >
> > > > > > Is the cluster balanced overall (in terms of number of regions
> per
> > > > > server)
> > > > > > but this table is not ?
> > > > > >
> > > > > > What happens (check master log) when you issue 'balancer' command
> > > > through
> > > > > > shell ?
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > > On Fri, Feb 13, 2015 at 8:19 AM, Shahab Yunus <
> > > shahab.yunus@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > CDH 5.3
> > > > > > > HBase 98.6
> > > > > > >
> > > > > > > We are writing data to an HBase table through a M/R job.
We pre
> > > split
> > > > > the
> > > > > > > table before each job run. The problem is that most of
the
> > regions
> > > > end
> > > > > up
> > > > > > > on the same RS. This results in that one RS being severely
> > > overloaded
> > > > > and
> > > > > > > subsequent M/R jobs failing trying to write to the regions
on
> > that
> > > > RS.
> > > > > > >
> > > > > > > The balancer is on and the split policy is default. No
changes
> > > there.
> > > > > It
> > > > > > is
> > > > > > > a 10 node cluster.
> > > > > > >
> > > > > > > All other related properties are defaults too.
> > > > > > >
> > > > > > > Any idea, how can we force balancing of the new regions?
Do we
> > have
> > > > to
> > > > > > > consider compaction into the equation as well? Thanks.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Shahab
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message