Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: error (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CALte62z0d14d04SV4qenyNM5GNxeKnZhws_m+Q2YuJAq7CN9XA@mail.gmail.com>
References: 
 <CAEo-6+RaJXNfdhTm+G98P9Xhn3E1A8oAOfW-YhirqL2XZsBiTg@mail.gmail.com>
 <CALte62wSsyA-4fi+Nwggg_9s3uL6u+nk9tnvp66M26jfSufrjw@mail.gmail.com>
 <CAEo-6+QiJ-GBNaxLEL-6=QUtr+m_97nuxcRx0Bq=7zd8_6C6zA@mail.gmail.com>
 <CALte62zc55wxxeW+8-rravd+FRr5ucMiB1qAHtnW6wyPiwZVnw@mail.gmail.com>
 <CAEo-6+SkbTMoCaKZ8JmBe6n3KwxLuZc2TJPc4sO4Z2rxps+vJQ@mail.gmail.com>
 <CALte62xGxxNAUw_npPY72nNXre-4z8wsVUBgncXpAtN5TRoWdA@mail.gmail.com>
 <CAEo-6+QbuvKYkumY4y_mvSmukeSDv3wYUir0=7O2rYGF1SMU7g@mail.gmail.com>
 <CALte62xiEpPYdTvC26SGZub2uoJ_gh9OCe0eyYSFK0AwLPWDDg@mail.gmail.com>
 <CAEo-6+QVdAG7EeFOnSRAdXsHJQXuvjrY8gbd0O+qndKj4gDGBA@mail.gmail.com>
 <CAGg3V12fCzKPMSs1-biQvRz3pLve+CoECxCiCLEUOAnJwxEL7A@mail.gmail.com>
 <CALte62ximd38OJjROKCTZadNFSSAwbZmbpUScHdH_a6ncgWhAg@mail.gmail.com>
 <CAGg3V10iVd-GoRzSLsnmiSzgLqEid0RzxZirT8BRUpCmOGtAJQ@mail.gmail.com>
 <CALte62y25siRQ4YqtdTVhVYjogNKLGXCj7bYf0c29OhxQtQpDw@mail.gmail.com>
 <CAGg3V12z2iejGMCNmfzriLbeyJ9598nAqMCWg44uuUoNr4h=+Q@mail.gmail.com>
 <CALte62z0d14d04SV4qenyNM5GNxeKnZhws_m+Q2YuJAq7CN9XA@mail.gmail.com>
From: Abe Weinograd <abe@flonet.com>
Date: Mon, 16 Feb 2015 11:24:21 -0500
Message-ID: 
 <CAGg3V10cbtsQft0svf1M0e5x9jLzwSLG40Fxryc+nRLVK5D6cw@mail.gmail.com>
Subject: Re: Region balancing query
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=001a1136860e9164ea050f37068f

--001a1136860e9164ea050f37068f
Content-Type: text/plain; charset=UTF-8

balancer said "true" and it is not disabled.  Thanks again for your help.

Abe

On Mon, Feb 16, 2015 at 11:23 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> What was the output from 'balancer' command ?
>
> Was is possible that balancer was disabled ?
>
> Cheers
>
> On Mon, Feb 16, 2015 at 8:04 AM, Abe Weinograd <abe@flonet.com> wrote:
>
> > Ok.  I forced the balancer run and am not seeing anything after a few
> > minutes.  Master logs isn't showing anything.  Should I look at the RS
> ones
> > instead?
> >
> > On Mon, Feb 16, 2015 at 11:03 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > You should see effect in the next balancer run.
> > >
> > > Cheers
> > >
> > > On Mon, Feb 16, 2015 at 7:52 AM, Abe Weinograd <abe@flonet.com> wrote:
> > >
> > > > Excellent. If i trigger the balancer, should this start to help or
> only
> > > for
> > > > future region creation?
> > > >
> > > > Thanks,
> > > > Abe
> > > >
> > > > On Mon, Feb 16, 2015 at 9:35 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> > > >
> > > > > Yes. This setting should be modified on Master.
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Mon, Feb 16, 2015 at 6:27 AM, Abe Weinograd <abe@flonet.com>
> > wrote:
> > > > >
> > > > > > Thanks Ted.  We are putting this in the hbase-site.xml for the
> > > Master?
> > > > > >
> > > > > > Abe
> > > > > >
> > > > > > On Fri, Feb 13, 2015 at 5:03 PM, Shahab Yunus <
> > > shahab.yunus@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks, we will try that and report back.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Shahab
> > > > > > >
> > > > > > > On Fri, Feb 13, 2015 at 4:56 PM, Ted Yu <yuzhihong@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > You can make TableSkewCostFunction more prominent by
> increasing
> > > the
> > > > > > value
> > > > > > > > for config parameter:
> > > > > > > >
> > > > > > > > hbase.master.balancer.stochastic.tableSkewCost
> > > > > > > >
> > > > > > > > Its default is 35.
> > > > > > > >
> > > > > > > > See if raising to 100 or 200 helps.
> > > > > > > >
> > > > > > > > On Fri, Feb 13, 2015 at 1:09 PM, Shahab Yunus <
> > > > > shahab.yunus@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Yes, this sever hosts other regions from other tables as
> > well.
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > Shahab
> > > > > > > > >
> > > > > > > > > On Fri, Feb 13, 2015 at 1:45 PM, Ted Yu <
> yuzhihong@gmail.com
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Interesting, server7.ec3.internal,60020,1423845018628 was
> > > > > > > consistently
> > > > > > > > > > chosen as destination for the table.
> > > > > > > > > > Did server7.ec3.internal,60020,1423845018628 host regions
> > > from
> > > > > > other
> > > > > > > > > table
> > > > > > > > > > ?
> > > > > > > > > >
> > > > > > > > > > Cheers
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 13, 2015 at 10:27 AM, Shahab Yunus <
> > > > > > > shahab.yunus@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Table name is:
> > > > > > > > > > > MYTABLE_RECENT_4W_V2
> > > > > > > > > > >
> > > > > > > > > > > Pastebin snippet 1: http://pastebin.com/dQzMhGyP
> > > > > > > > > > > Pastebin snippet 2: http://pastebin.com/Y7ZsNAgF
> > > > > > > > > > >
> > > > > > > > > > > This is the master log after invoking balancer command
> > from
> > > > > hbase
> > > > > > > > > shell.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Shahab
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Feb 13, 2015 at 12:00 PM, Ted Yu <
> > > > yuzhihong@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > bq. all the regions of this table were back on this
> > same
> > > > RS!
> > > > > > > > > > > >
> > > > > > > > > > > > Interesting. Please check master log around the time
> > this
> > > > RS
> > > > > > was
> > > > > > > > > > brought
> > > > > > > > > > > > online. You can pastebin the relevant snippet.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Feb 13, 2015 at 8:55 AM, Shahab Yunus <
> > > > > > > > > shahab.yunus@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Ted.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes, the cluster itself is balanced. On average 300
> > > > regions
> > > > > > per
> > > > > > > > > node
> > > > > > > > > > on
> > > > > > > > > > > > 10
> > > > > > > > > > > > > nodes.
> > > > > > > > > > > > >
> > > > > > > > > > > > > # of tables is 53 of varying sizes.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Balancer was invoked and it didn't do anything
> (i.e.
> > no
> > > > > > > movement
> > > > > > > > of
> > > > > > > > > > > > > regions) but we didn't check the master's logs. We
> > can
> > > do
> > > > > > that.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Interestingly, we restarted the RS which was
> holding
> > > all
> > > > > the
> > > > > > > > > regions
> > > > > > > > > > of
> > > > > > > > > > > > > this one table. The regions were nicely spread out
> to
> > > the
> > > > > > > > remaining
> > > > > > > > > > RS.
> > > > > > > > > > > > But
> > > > > > > > > > > > > when we brought back this RS, all the regions of
> this
> > > > table
> > > > > > > were
> > > > > > > > > back
> > > > > > > > > > > on
> > > > > > > > > > > > > this same RS!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > Shahab
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Feb 13, 2015 at 11:46 AM, Ted Yu <
> > > > > > yuzhihong@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > How many tables are there in your cluster ?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Is the cluster balanced overall (in terms of
> number
> > > of
> > > > > > > regions
> > > > > > > > > per
> > > > > > > > > > > > > server)
> > > > > > > > > > > > > > but this table is not ?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > What happens (check master log) when you issue
> > > > 'balancer'
> > > > > > > > command
> > > > > > > > > > > > through
> > > > > > > > > > > > > > shell ?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Feb 13, 2015 at 8:19 AM, Shahab Yunus <
> > > > > > > > > > > shahab.yunus@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > CDH 5.3
> > > > > > > > > > > > > > > HBase 98.6
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > We are writing data to an HBase table through a
> > M/R
> > > > > job.
> > > > > > We
> > > > > > > > pre
> > > > > > > > > > > split
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > table before each job run. The problem is that
> > most
> > > > of
> > > > > > the
> > > > > > > > > > regions
> > > > > > > > > > > > end
> > > > > > > > > > > > > up
> > > > > > > > > > > > > > > on the same RS. This results in that one RS
> being
> > > > > > severely
> > > > > > > > > > > overloaded
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > subsequent M/R jobs failing trying to write to
> > the
> > > > > > regions
> > > > > > > on
> > > > > > > > > > that
> > > > > > > > > > > > RS.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The balancer is on and the split policy is
> > default.
> > > > No
> > > > > > > > changes
> > > > > > > > > > > there.
> > > > > > > > > > > > > It
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > a 10 node cluster.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > All other related properties are defaults too.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Any idea, how can we force balancing of the new
> > > > > regions?
> > > > > > Do
> > > > > > > > we
> > > > > > > > > > have
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > consider compaction into the equation as well?
> > > > Thanks.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > > > Shahab
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--001a1136860e9164ea050f37068f--