hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abe Weinograd <...@flonet.com>
Subject Re: Region balancing query
Date Wed, 18 Feb 2015 15:35:48 GMT
Another question, would this help when we split or only when we balance?

Thanks,
Abe

On Mon, Feb 16, 2015 at 11:24 AM, Abe Weinograd <abe@flonet.com> wrote:

> balancer said "true" and it is not disabled.  Thanks again for your help.
>
> Abe
>
> On Mon, Feb 16, 2015 at 11:23 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
>> What was the output from 'balancer' command ?
>>
>> Was is possible that balancer was disabled ?
>>
>> Cheers
>>
>> On Mon, Feb 16, 2015 at 8:04 AM, Abe Weinograd <abe@flonet.com> wrote:
>>
>> > Ok.  I forced the balancer run and am not seeing anything after a few
>> > minutes.  Master logs isn't showing anything.  Should I look at the RS
>> ones
>> > instead?
>> >
>> > On Mon, Feb 16, 2015 at 11:03 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>> >
>> > > You should see effect in the next balancer run.
>> > >
>> > > Cheers
>> > >
>> > > On Mon, Feb 16, 2015 at 7:52 AM, Abe Weinograd <abe@flonet.com>
>> wrote:
>> > >
>> > > > Excellent. If i trigger the balancer, should this start to help or
>> only
>> > > for
>> > > > future region creation?
>> > > >
>> > > > Thanks,
>> > > > Abe
>> > > >
>> > > > On Mon, Feb 16, 2015 at 9:35 AM, Ted Yu <yuzhihong@gmail.com>
>> wrote:
>> > > >
>> > > > > Yes. This setting should be modified on Master.
>> > > > >
>> > > > > Cheers
>> > > > >
>> > > > > On Mon, Feb 16, 2015 at 6:27 AM, Abe Weinograd <abe@flonet.com>
>> > wrote:
>> > > > >
>> > > > > > Thanks Ted.  We are putting this in the hbase-site.xml for
the
>> > > Master?
>> > > > > >
>> > > > > > Abe
>> > > > > >
>> > > > > > On Fri, Feb 13, 2015 at 5:03 PM, Shahab Yunus <
>> > > shahab.yunus@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Thanks, we will try that and report back.
>> > > > > > >
>> > > > > > > Regards,
>> > > > > > > Shahab
>> > > > > > >
>> > > > > > > On Fri, Feb 13, 2015 at 4:56 PM, Ted Yu <yuzhihong@gmail.com>
>> > > wrote:
>> > > > > > >
>> > > > > > > > You can make TableSkewCostFunction more prominent
by
>> increasing
>> > > the
>> > > > > > value
>> > > > > > > > for config parameter:
>> > > > > > > >
>> > > > > > > > hbase.master.balancer.stochastic.tableSkewCost
>> > > > > > > >
>> > > > > > > > Its default is 35.
>> > > > > > > >
>> > > > > > > > See if raising to 100 or 200 helps.
>> > > > > > > >
>> > > > > > > > On Fri, Feb 13, 2015 at 1:09 PM, Shahab Yunus
<
>> > > > > shahab.yunus@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Yes, this sever hosts other regions from
other tables as
>> > well.
>> > > > > > > > >
>> > > > > > > > > Regards
>> > > > > > > > > Shahab
>> > > > > > > > >
>> > > > > > > > > On Fri, Feb 13, 2015 at 1:45 PM, Ted Yu <
>> yuzhihong@gmail.com
>> > >
>> > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Interesting, server7.ec3.internal,60020,1423845018628
>> was
>> > > > > > > consistently
>> > > > > > > > > > chosen as destination for the table.
>> > > > > > > > > > Did server7.ec3.internal,60020,1423845018628
host
>> regions
>> > > from
>> > > > > > other
>> > > > > > > > > table
>> > > > > > > > > > ?
>> > > > > > > > > >
>> > > > > > > > > > Cheers
>> > > > > > > > > >
>> > > > > > > > > > On Fri, Feb 13, 2015 at 10:27 AM, Shahab
Yunus <
>> > > > > > > shahab.yunus@gmail.com
>> > > > > > > > >
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Table name is:
>> > > > > > > > > > > MYTABLE_RECENT_4W_V2
>> > > > > > > > > > >
>> > > > > > > > > > > Pastebin snippet 1: http://pastebin.com/dQzMhGyP
>> > > > > > > > > > > Pastebin snippet 2: http://pastebin.com/Y7ZsNAgF
>> > > > > > > > > > >
>> > > > > > > > > > > This is the master log after invoking
balancer command
>> > from
>> > > > > hbase
>> > > > > > > > > shell.
>> > > > > > > > > > >
>> > > > > > > > > > > Regards,
>> > > > > > > > > > > Shahab
>> > > > > > > > > > >
>> > > > > > > > > > > On Fri, Feb 13, 2015 at 12:00 PM,
Ted Yu <
>> > > > yuzhihong@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > bq. all the regions of this
table were back on this
>> > same
>> > > > RS!
>> > > > > > > > > > > >
>> > > > > > > > > > > > Interesting. Please check
master log around the time
>> > this
>> > > > RS
>> > > > > > was
>> > > > > > > > > > brought
>> > > > > > > > > > > > online. You can pastebin the
relevant snippet.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Fri, Feb 13, 2015 at 8:55
AM, Shahab Yunus <
>> > > > > > > > > shahab.yunus@gmail.com>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Hi Ted.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Yes, the cluster itself
is balanced. On average
>> 300
>> > > > regions
>> > > > > > per
>> > > > > > > > > node
>> > > > > > > > > > on
>> > > > > > > > > > > > 10
>> > > > > > > > > > > > > nodes.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > # of tables is 53 of
varying sizes.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Balancer was invoked
and it didn't do anything
>> (i.e.
>> > no
>> > > > > > > movement
>> > > > > > > > of
>> > > > > > > > > > > > > regions) but we didn't
check the master's logs. We
>> > can
>> > > do
>> > > > > > that.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Interestingly, we restarted
the RS which was
>> holding
>> > > all
>> > > > > the
>> > > > > > > > > regions
>> > > > > > > > > > of
>> > > > > > > > > > > > > this one table. The regions
were nicely spread
>> out to
>> > > the
>> > > > > > > > remaining
>> > > > > > > > > > RS.
>> > > > > > > > > > > > But
>> > > > > > > > > > > > > when we brought back
this RS, all the regions of
>> this
>> > > > table
>> > > > > > > were
>> > > > > > > > > back
>> > > > > > > > > > > on
>> > > > > > > > > > > > > this same RS!
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Regards,
>> > > > > > > > > > > > > Shahab
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Fri, Feb 13, 2015
at 11:46 AM, Ted Yu <
>> > > > > > yuzhihong@gmail.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > How many tables
are there in your cluster ?
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Is the cluster balanced
overall (in terms of
>> number
>> > > of
>> > > > > > > regions
>> > > > > > > > > per
>> > > > > > > > > > > > > server)
>> > > > > > > > > > > > > > but this table is
not ?
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > What happens (check
master log) when you issue
>> > > > 'balancer'
>> > > > > > > > command
>> > > > > > > > > > > > through
>> > > > > > > > > > > > > > shell ?
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Cheers
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Fri, Feb 13,
2015 at 8:19 AM, Shahab Yunus <
>> > > > > > > > > > > shahab.yunus@gmail.com>
>> > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > CDH 5.3
>> > > > > > > > > > > > > > > HBase 98.6
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > We are writing
data to an HBase table through
>> a
>> > M/R
>> > > > > job.
>> > > > > > We
>> > > > > > > > pre
>> > > > > > > > > > > split
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > table before
each job run. The problem is that
>> > most
>> > > > of
>> > > > > > the
>> > > > > > > > > > regions
>> > > > > > > > > > > > end
>> > > > > > > > > > > > > up
>> > > > > > > > > > > > > > > on the same
RS. This results in that one RS
>> being
>> > > > > > severely
>> > > > > > > > > > > overloaded
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > subsequent
M/R jobs failing trying to write to
>> > the
>> > > > > > regions
>> > > > > > > on
>> > > > > > > > > > that
>> > > > > > > > > > > > RS.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > The balancer
is on and the split policy is
>> > default.
>> > > > No
>> > > > > > > > changes
>> > > > > > > > > > > there.
>> > > > > > > > > > > > > It
>> > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > a 10 node cluster.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > All other related
properties are defaults too.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Any idea, how
can we force balancing of the
>> new
>> > > > > regions?
>> > > > > > Do
>> > > > > > > > we
>> > > > > > > > > > have
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > > consider compaction
into the equation as well?
>> > > > Thanks.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Regards,
>> > > > > > > > > > > > > > > Shahab
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message