Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates
 209.85.213.50 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAGg3V121iTXwP6kUL=UiZDf25OZQGb=xcMFSjup4tYOOQseFRA@mail.gmail.com>
References: 
 <CAEo-6+RaJXNfdhTm+G98P9Xhn3E1A8oAOfW-YhirqL2XZsBiTg@mail.gmail.com>
	<CALte62wSsyA-4fi+Nwggg_9s3uL6u+nk9tnvp66M26jfSufrjw@mail.gmail.com>
	<CAEo-6+QiJ-GBNaxLEL-6=QUtr+m_97nuxcRx0Bq=7zd8_6C6zA@mail.gmail.com>
	<CALte62zc55wxxeW+8-rravd+FRr5ucMiB1qAHtnW6wyPiwZVnw@mail.gmail.com>
	<CAEo-6+SkbTMoCaKZ8JmBe6n3KwxLuZc2TJPc4sO4Z2rxps+vJQ@mail.gmail.com>
	<CALte62xGxxNAUw_npPY72nNXre-4z8wsVUBgncXpAtN5TRoWdA@mail.gmail.com>
	<CAEo-6+QbuvKYkumY4y_mvSmukeSDv3wYUir0=7O2rYGF1SMU7g@mail.gmail.com>
	<CALte62xiEpPYdTvC26SGZub2uoJ_gh9OCe0eyYSFK0AwLPWDDg@mail.gmail.com>
	<CAEo-6+QVdAG7EeFOnSRAdXsHJQXuvjrY8gbd0O+qndKj4gDGBA@mail.gmail.com>
	<CAGg3V12fCzKPMSs1-biQvRz3pLve+CoECxCiCLEUOAnJwxEL7A@mail.gmail.com>
	<CALte62ximd38OJjROKCTZadNFSSAwbZmbpUScHdH_a6ncgWhAg@mail.gmail.com>
	<CAGg3V10iVd-GoRzSLsnmiSzgLqEid0RzxZirT8BRUpCmOGtAJQ@mail.gmail.com>
	<CALte62y25siRQ4YqtdTVhVYjogNKLGXCj7bYf0c29OhxQtQpDw@mail.gmail.com>
	<CAGg3V12z2iejGMCNmfzriLbeyJ9598nAqMCWg44uuUoNr4h=+Q@mail.gmail.com>
	<CALte62z0d14d04SV4qenyNM5GNxeKnZhws_m+Q2YuJAq7CN9XA@mail.gmail.com>
	<CAGg3V10cbtsQft0svf1M0e5x9jLzwSLG40Fxryc+nRLVK5D6cw@mail.gmail.com>
	<CAGg3V121iTXwP6kUL=UiZDf25OZQGb=xcMFSjup4tYOOQseFRA@mail.gmail.com>
Date: Wed, 18 Feb 2015 09:01:14 -0800
Message-ID: 
 <CALte62w1UAkrqgPMNRAO34GO5atev_KNjzXzz-9GH-+NGJTK0g@mail.gmail.com>
Subject: Re: Region balancing query
From: Ted Yu <yuzhihong@gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=089e011845aa5a2e22050f5fc4f7

--089e011845aa5a2e22050f5fc4f7
Content-Type: text/plain; charset=UTF-8

When a region splits, the two daughter regions stay on the same host as the
parent region did.

It is through balancing that the daughter regions (may) be assigned to
other server(s).

Cheers

On Wed, Feb 18, 2015 at 7:35 AM, Abe Weinograd <abe@flonet.com> wrote:

> Another question, would this help when we split or only when we balance?
>
> Thanks,
> Abe
>
> On Mon, Feb 16, 2015 at 11:24 AM, Abe Weinograd <abe@flonet.com> wrote:
>
> > balancer said "true" and it is not disabled.  Thanks again for your help.
> >
> > Abe
> >
> > On Mon, Feb 16, 2015 at 11:23 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> >> What was the output from 'balancer' command ?
> >>
> >> Was is possible that balancer was disabled ?
> >>
> >> Cheers
> >>
> >> On Mon, Feb 16, 2015 at 8:04 AM, Abe Weinograd <abe@flonet.com> wrote:
> >>
> >> > Ok.  I forced the balancer run and am not seeing anything after a few
> >> > minutes.  Master logs isn't showing anything.  Should I look at the RS
> >> ones
> >> > instead?
> >> >
> >> > On Mon, Feb 16, 2015 at 11:03 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >> >
> >> > > You should see effect in the next balancer run.
> >> > >
> >> > > Cheers
> >> > >
> >> > > On Mon, Feb 16, 2015 at 7:52 AM, Abe Weinograd <abe@flonet.com>
> >> wrote:
> >> > >
> >> > > > Excellent. If i trigger the balancer, should this start to help or
> >> only
> >> > > for
> >> > > > future region creation?
> >> > > >
> >> > > > Thanks,
> >> > > > Abe
> >> > > >
> >> > > > On Mon, Feb 16, 2015 at 9:35 AM, Ted Yu <yuzhihong@gmail.com>
> >> wrote:
> >> > > >
> >> > > > > Yes. This setting should be modified on Master.
> >> > > > >
> >> > > > > Cheers
> >> > > > >
> >> > > > > On Mon, Feb 16, 2015 at 6:27 AM, Abe Weinograd <abe@flonet.com>
> >> > wrote:
> >> > > > >
> >> > > > > > Thanks Ted.  We are putting this in the hbase-site.xml for the
> >> > > Master?
> >> > > > > >
> >> > > > > > Abe
> >> > > > > >
> >> > > > > > On Fri, Feb 13, 2015 at 5:03 PM, Shahab Yunus <
> >> > > shahab.yunus@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Thanks, we will try that and report back.
> >> > > > > > >
> >> > > > > > > Regards,
> >> > > > > > > Shahab
> >> > > > > > >
> >> > > > > > > On Fri, Feb 13, 2015 at 4:56 PM, Ted Yu <
> yuzhihong@gmail.com>
> >> > > wrote:
> >> > > > > > >
> >> > > > > > > > You can make TableSkewCostFunction more prominent by
> >> increasing
> >> > > the
> >> > > > > > value
> >> > > > > > > > for config parameter:
> >> > > > > > > >
> >> > > > > > > > hbase.master.balancer.stochastic.tableSkewCost
> >> > > > > > > >
> >> > > > > > > > Its default is 35.
> >> > > > > > > >
> >> > > > > > > > See if raising to 100 or 200 helps.
> >> > > > > > > >
> >> > > > > > > > On Fri, Feb 13, 2015 at 1:09 PM, Shahab Yunus <
> >> > > > > shahab.yunus@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Yes, this sever hosts other regions from other tables as
> >> > well.
> >> > > > > > > > >
> >> > > > > > > > > Regards
> >> > > > > > > > > Shahab
> >> > > > > > > > >
> >> > > > > > > > > On Fri, Feb 13, 2015 at 1:45 PM, Ted Yu <
> >> yuzhihong@gmail.com
> >> > >
> >> > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Interesting, server7.ec3.internal,60020,1423845018628
> >> was
> >> > > > > > > consistently
> >> > > > > > > > > > chosen as destination for the table.
> >> > > > > > > > > > Did server7.ec3.internal,60020,1423845018628 host
> >> regions
> >> > > from
> >> > > > > > other
> >> > > > > > > > > table
> >> > > > > > > > > > ?
> >> > > > > > > > > >
> >> > > > > > > > > > Cheers
> >> > > > > > > > > >
> >> > > > > > > > > > On Fri, Feb 13, 2015 at 10:27 AM, Shahab Yunus <
> >> > > > > > > shahab.yunus@gmail.com
> >> > > > > > > > >
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Table name is:
> >> > > > > > > > > > > MYTABLE_RECENT_4W_V2
> >> > > > > > > > > > >
> >> > > > > > > > > > > Pastebin snippet 1: http://pastebin.com/dQzMhGyP
> >> > > > > > > > > > > Pastebin snippet 2: http://pastebin.com/Y7ZsNAgF
> >> > > > > > > > > > >
> >> > > > > > > > > > > This is the master log after invoking balancer
> command
> >> > from
> >> > > > > hbase
> >> > > > > > > > > shell.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Regards,
> >> > > > > > > > > > > Shahab
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Fri, Feb 13, 2015 at 12:00 PM, Ted Yu <
> >> > > > yuzhihong@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > bq. all the regions of this table were back on
> this
> >> > same
> >> > > > RS!
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Interesting. Please check master log around the
> time
> >> > this
> >> > > > RS
> >> > > > > > was
> >> > > > > > > > > > brought
> >> > > > > > > > > > > > online. You can pastebin the relevant snippet.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Thanks
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Fri, Feb 13, 2015 at 8:55 AM, Shahab Yunus <
> >> > > > > > > > > shahab.yunus@gmail.com>
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Hi Ted.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Yes, the cluster itself is balanced. On average
> >> 300
> >> > > > regions
> >> > > > > > per
> >> > > > > > > > > node
> >> > > > > > > > > > on
> >> > > > > > > > > > > > 10
> >> > > > > > > > > > > > > nodes.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > # of tables is 53 of varying sizes.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Balancer was invoked and it didn't do anything
> >> (i.e.
> >> > no
> >> > > > > > > movement
> >> > > > > > > > of
> >> > > > > > > > > > > > > regions) but we didn't check the master's logs.
> We
> >> > can
> >> > > do
> >> > > > > > that.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Interestingly, we restarted the RS which was
> >> holding
> >> > > all
> >> > > > > the
> >> > > > > > > > > regions
> >> > > > > > > > > > of
> >> > > > > > > > > > > > > this one table. The regions were nicely spread
> >> out to
> >> > > the
> >> > > > > > > > remaining
> >> > > > > > > > > > RS.
> >> > > > > > > > > > > > But
> >> > > > > > > > > > > > > when we brought back this RS, all the regions of
> >> this
> >> > > > table
> >> > > > > > > were
> >> > > > > > > > > back
> >> > > > > > > > > > > on
> >> > > > > > > > > > > > > this same RS!
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Thanks.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Regards,
> >> > > > > > > > > > > > > Shahab
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > On Fri, Feb 13, 2015 at 11:46 AM, Ted Yu <
> >> > > > > > yuzhihong@gmail.com>
> >> > > > > > > > > > wrote:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > > How many tables are there in your cluster ?
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Is the cluster balanced overall (in terms of
> >> number
> >> > > of
> >> > > > > > > regions
> >> > > > > > > > > per
> >> > > > > > > > > > > > > server)
> >> > > > > > > > > > > > > > but this table is not ?
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > What happens (check master log) when you issue
> >> > > > 'balancer'
> >> > > > > > > > command
> >> > > > > > > > > > > > through
> >> > > > > > > > > > > > > > shell ?
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Cheers
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > On Fri, Feb 13, 2015 at 8:19 AM, Shahab Yunus
> <
> >> > > > > > > > > > > shahab.yunus@gmail.com>
> >> > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > CDH 5.3
> >> > > > > > > > > > > > > > > HBase 98.6
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > We are writing data to an HBase table
> through
> >> a
> >> > M/R
> >> > > > > job.
> >> > > > > > We
> >> > > > > > > > pre
> >> > > > > > > > > > > split
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > table before each job run. The problem is
> that
> >> > most
> >> > > > of
> >> > > > > > the
> >> > > > > > > > > > regions
> >> > > > > > > > > > > > end
> >> > > > > > > > > > > > > up
> >> > > > > > > > > > > > > > > on the same RS. This results in that one RS
> >> being
> >> > > > > > severely
> >> > > > > > > > > > > overloaded
> >> > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > subsequent M/R jobs failing trying to write
> to
> >> > the
> >> > > > > > regions
> >> > > > > > > on
> >> > > > > > > > > > that
> >> > > > > > > > > > > > RS.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > The balancer is on and the split policy is
> >> > default.
> >> > > > No
> >> > > > > > > > changes
> >> > > > > > > > > > > there.
> >> > > > > > > > > > > > > It
> >> > > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > > a 10 node cluster.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > All other related properties are defaults
> too.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Any idea, how can we force balancing of the
> >> new
> >> > > > > regions?
> >> > > > > > Do
> >> > > > > > > > we
> >> > > > > > > > > > have
> >> > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > consider compaction into the equation as
> well?
> >> > > > Thanks.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Regards,
> >> > > > > > > > > > > > > > > Shahab
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

--089e011845aa5a2e22050f5fc4f7--