Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of bharathv@cloudera.com
 designates 209.85.218.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CALte62zoS_ypeHkSRysx=8ZONEp4jv2oJHsYGf4-CAUXk2j+vQ@mail.gmail.com>
References: 
 <CACA1tWLzQVRmsnphLcdfagBqDaRZzvZiZsrgU1a6HK751JXBiw@mail.gmail.com>
 <CAPQV63X1Xr8hhQsRG8torj7sPsQD+qdPgERnpP-NcQp_HPZ=uw@mail.gmail.com>
 <CACA1tWJE_BA9LJN-xCOi6BNpqjK4HFi+-SwHihsaixqr9KsXGw@mail.gmail.com>
 <D02F2B3E.14117%maganazook1@llnl.gov>
 <CACA1tWLm0L2D0-uYC1xvLwghU4XTHV_N9QBap5c1BLg8c7M5wg@mail.gmail.com>
 <CAEo-6+Q_cZp8Key7+RCfGS1=Ox7X=J2hosUenc3F+Zx8hCf7dw@mail.gmail.com>
 <CACA1tW+FZOrzum1Q_FNUWuEANshkKmqvjU97DqQMQe0svqsFrg@mail.gmail.com>
 <CALte62yhnRCzh8KNbzurEn3eXFX_c-BYrCSSgTZy4by+tHLzQA@mail.gmail.com>
 <CACA1tWLJY4ht4ACQkCcOe6wj0qkgJpnjt7QGOLG9=9s173j8Gg@mail.gmail.com>
 <CALte62wpfuKZn9QQnwqDp9D_NE_SaRqdE060m4AiQyCk7xfEkQ@mail.gmail.com>
 <CACA1tWJquFzcgnjiQi=GJ1ZwR18t9m+-k5Y-GHzF2DVF4GxRNA@mail.gmail.com>
 <CALte62zNEes3RkCPycOiwaPqg_H=rj7RtgmGiuhyT+XsUUO2_w@mail.gmail.com>
 <CACA1tWLoc88uJVoLV_u0QQoHCsXP8b=qKw2gTuK2dk6PNFX+XQ@mail.gmail.com>
 <CALte62zAAiBmcd+Jc_cf3WQYjg6Dasm6ZH4rj3YJEY+M=56VQg@mail.gmail.com>
 <CACA1tWJDtmYJe1ZU3_j+WGwj5z1J+vc6-QswAvXppmZjmKZcFA@mail.gmail.com>
 <CALte62wSpNfsMu5vbWSG5Ewv5_1XDjLWGbEuL=ZQgjnRvg_2QQ@mail.gmail.com>
 <CACA1tWL1+Xt79y7kq-H_WkH1HjAWWjGLEWx=R-_3pgUNaW_VxA@mail.gmail.com>
 <CACA1tWKB2dNG+11BZEB=QEvXx=61HM64_wCWC__fHeUSBYQGbA@mail.gmail.com>
 <CACA1tWK3JsoLTWygaQWHkqFKSgLZKPtUr223uXgRgSQi1i_RoA@mail.gmail.com>
 <CALte62zoS_ypeHkSRysx=8ZONEp4jv2oJHsYGf4-CAUXk2j+vQ@mail.gmail.com>
From: Bharath Vissapragada <bharathv@cloudera.com>
Date: Mon, 22 Sep 2014 11:12:24 -0700
Message-ID: 
 <CAFWiQHYTdzNZanug9aKUuHm5cSTqzs0_L43e7uRY_+9dHBThpQ@mail.gmail.com>
Subject: Re: Help: RegionTooBusyException: failed to get a lock in 60000 ms
To: hbase-user <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=089e013cb946c2c9d80503ab6590

--089e013cb946c2c9d80503ab6590
Content-Type: text/plain; charset=UTF-8

https://issues.apache.org/jira/browse/HBASE-11368 related to the original
issue too.

On Mon, Sep 22, 2014 at 10:18 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> As you noted in the FIXME, there're some factors which should be tackled by
> balancer / assignment manager.
>
> Please continue digging up master log so that we can find the cause for
> balancer not fulfilling your goal.
>
> Cheers
>
> On Mon, Sep 22, 2014 at 10:09 AM, Jianshi Huang <jianshi.huang@gmail.com>
> wrote:
>
> > Ok, I fixed this by manually reassign region servers to newly created
> ones.
> >
> >   def reassignRegionServer(admin: HBaseAdmin, regions: Seq[HRegionInfo],
> > regionServers: Seq[ServerName]): Unit = {
> >     val rand = new Random()
> >     regions.foreach { r =>
> >       val idx = rand.nextInt(regionServers.size)
> >       val server = regionServers(idx)
> >       // FIXME: what if selected region server is dead?
> >         admin.move(r.getEncodedNameAsBytes,
> > server.getServerName.getBytes("UTF8"))
> >     }
> >   }
> >
> > er...
> >
> > Jianshi
> >
> > On Tue, Sep 23, 2014 at 12:24 AM, Jianshi Huang <jianshi.huang@gmail.com
> >
> > wrote:
> >
> > > Hmm...any workaround? I only want to do this:
> > >
> > > Rebalance the new regions *evenly* to all servers after manually adding
> > > splits, so later bulk insertions won't cause contention.
> > >
> > > P.S.
> > > Looks like two of the region servers which had majority of the regions
> > > were down during Major compaction... I guess it had too much data.
> > >
> > >
> > > Jianshi
> > >
> > > On Tue, Sep 23, 2014 at 12:13 AM, Jianshi Huang <
> jianshi.huang@gmail.com
> > >
> > > wrote:
> > >
> > >> Yes, I have access to Master UI, however logs/*.log cannot be opened
> or
> > >> downloaded, must be some security restrictions in the proxy...
> > >>
> > >> Jianshi
> > >>
> > >> On Tue, Sep 23, 2014 at 12:06 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >>
> > >>> Do you have access to Master UI ?
> > >>>
> > >>> <master-address>:60010/logs/ would show you list of log files.
> > >>>
> > >>> The you can view
> > <master-address>:60010/logs/hbase-<user>-master-XXX.log
> > >>>
> > >>> Cheers
> > >>>
> > >>> On Mon, Sep 22, 2014 at 9:00 AM, Jianshi Huang <
> > jianshi.huang@gmail.com>
> > >>> wrote:
> > >>>
> > >>> > Ah... I don't have access to HMaster logs... I need to ask the
> admin.
> > >>> >
> > >>> > Jianshi
> > >>> >
> > >>> > On Mon, Sep 22, 2014 at 11:49 PM, Ted Yu <yuzhihong@gmail.com>
> > wrote:
> > >>> >
> > >>> > > bq. assign per-table balancer class
> > >>> > >
> > >>> > > No that I know of.
> > >>> > > Can you pastebin master log involving output from balancer ?
> > >>> > >
> > >>> > > Cheers
> > >>> > >
> > >>> > > On Mon, Sep 22, 2014 at 8:29 AM, Jianshi Huang <
> > >>> jianshi.huang@gmail.com>
> > >>> > > wrote:
> > >>> > >
> > >>> > > > Hi Ted,
> > >>> > > >
> > >>> > > > I moved setBalancerRunning before balancer and run them twice.
> > >>> However
> > >>> > I
> > >>> > > > still got highly skewed region distribution.
> > >>> > > >
> > >>> > > > I guess it's because of the StochasticLoadBalancer, can I
> assign
> > >>> > > per-table
> > >>> > > > balancer class in HBase?
> > >>> > > >
> > >>> > > >
> > >>> > > > Jianshi
> > >>> > > >
> > >>> > > > On Mon, Sep 22, 2014 at 9:50 PM, Ted Yu <yuzhihong@gmail.com>
> > >>> wrote:
> > >>> > > >
> > >>> > > > > admin.setBalancerRunning() call should precede the call to
> > >>> > > > > admin.balancer().
> > >>> > > > >
> > >>> > > > > You can inspect master log to see whether regions are being
> > >>> moved off
> > >>> > > the
> > >>> > > > > heavily loaded server.
> > >>> > > > >
> > >>> > > > > Cheers
> > >>> > > > >
> > >>> > > > > On Mon, Sep 22, 2014 at 1:42 AM, Jianshi Huang <
> > >>> > > jianshi.huang@gmail.com>
> > >>> > > > > wrote:
> > >>> > > > >
> > >>> > > > > > Hi Ted and others,
> > >>> > > > > >
> > >>> > > > > > I did the following after adding splits (without data) to
> my
> > >>> table,
> > >>> > > > > however
> > >>> > > > > > the region is still very imbalanced (one region server has
> > 221
> > >>> > > regions
> > >>> > > > > and
> > >>> > > > > > other 50 region servers have about 4~8 regions each).
> > >>> > > > > >
> > >>> > > > > >       admin.balancer()
> > >>> > > > > >       admin.setBalancerRunning(true, true)
> > >>> > > > > >
> > >>> > > > > > The balancer class in my HBase cluster is
> > >>> > > > > >
> > >>> > > > > >
> > org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer
> > >>> > > > > >
> > >>> > > > > > So, is this behavior expected? Can I assign different
> > balancer
> > >>> > class
> > >>> > > to
> > >>> > > > > my
> > >>> > > > > > tables (I don't have HBase admin permission)? Which one
> > should
> > >>> I
> > >>> > use?
> > >>> > > > > >
> > >>> > > > > > I just want HBase to evenly distribute the regions even
> they
> > >>> don't
> > >>> > > have
> > >>> > > > > > data (that's the purpose of pre-split I think).
> > >>> > > > > >
> > >>> > > > > >
> > >>> > > > > > Jianshi
> > >>> > > > > >
> > >>> > > > > >
> > >>> > > > > > On Sat, Sep 6, 2014 at 12:45 AM, Ted Yu <
> yuzhihong@gmail.com
> > >
> > >>> > wrote:
> > >>> > > > > >
> > >>> > > > > > > Yes. See the following method in HBaseAdmin:
> > >>> > > > > > >
> > >>> > > > > > >   public boolean balancer()
> > >>> > > > > > >
> > >>> > > > > > >
> > >>> > > > > > > On Fri, Sep 5, 2014 at 9:38 AM, Jianshi Huang <
> > >>> > > > jianshi.huang@gmail.com
> > >>> > > > > >
> > >>> > > > > > > wrote:
> > >>> > > > > > >
> > >>> > > > > > > > Thanks Ted!
> > >>> > > > > > > >
> > >>> > > > > > > > Didn't know I still need to run the 'balancer' command.
> > >>> > > > > > > >
> > >>> > > > > > > > Is there a way to do it programmatically?
> > >>> > > > > > > >
> > >>> > > > > > > > Jianshi
> > >>> > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > > > On Sat, Sep 6, 2014 at 12:29 AM, Ted Yu <
> > >>> yuzhihong@gmail.com>
> > >>> > > > wrote:
> > >>> > > > > > > >
> > >>> > > > > > > > > After splitting the region, you may need to run
> > balancer
> > >>> to
> > >>> > > > spread
> > >>> > > > > > the
> > >>> > > > > > > > new
> > >>> > > > > > > > > regions out.
> > >>> > > > > > > > >
> > >>> > > > > > > > > Cheers
> > >>> > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > > > On Fri, Sep 5, 2014 at 9:25 AM, Jianshi Huang <
> > >>> > > > > > jianshi.huang@gmail.com
> > >>> > > > > > > >
> > >>> > > > > > > > > wrote:
> > >>> > > > > > > > >
> > >>> > > > > > > > > > Hi Shahab,
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > I see, that seems to be the right way...
> > >>> > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > On Sat, Sep 6, 2014 at 12:21 AM, Shahab Yunus <
> > >>> > > > > > > shahab.yunus@gmail.com>
> > >>> > > > > > > > > > wrote:
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > > Shahab
> > >>> > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > --
> > >>> > > > > > > > > > Jianshi Huang
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > LinkedIn: jianshi
> > >>> > > > > > > > > > Twitter: @jshuang
> > >>> > > > > > > > > > Github & Blog: http://huangjs.github.com/
> > >>> > > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > > > --
> > >>> > > > > > > > Jianshi Huang
> > >>> > > > > > > >
> > >>> > > > > > > > LinkedIn: jianshi
> > >>> > > > > > > > Twitter: @jshuang
> > >>> > > > > > > > Github & Blog: http://huangjs.github.com/
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > > >
> > >>> > > > > >
> > >>> > > > > > --
> > >>> > > > > > Jianshi Huang
> > >>> > > > > >
> > >>> > > > > > LinkedIn: jianshi
> > >>> > > > > > Twitter: @jshuang
> > >>> > > > > > Github & Blog: http://huangjs.github.com/
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > > >
> > >>> > > >
> > >>> > > > --
> > >>> > > > Jianshi Huang
> > >>> > > >
> > >>> > > > LinkedIn: jianshi
> > >>> > > > Twitter: @jshuang
> > >>> > > > Github & Blog: http://huangjs.github.com/
> > >>> > > >
> > >>> > >
> > >>> >
> > >>> >
> > >>> >
> > >>> > --
> > >>> > Jianshi Huang
> > >>> >
> > >>> > LinkedIn: jianshi
> > >>> > Twitter: @jshuang
> > >>> > Github & Blog: http://huangjs.github.com/
> > >>> >
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Jianshi Huang
> > >>
> > >> LinkedIn: jianshi
> > >> Twitter: @jshuang
> > >> Github & Blog: http://huangjs.github.com/
> > >>
> > >
> > >
> > >
> > > --
> > > Jianshi Huang
> > >
> > > LinkedIn: jianshi
> > > Twitter: @jshuang
> > > Github & Blog: http://huangjs.github.com/
> > >
> >
> >
> >
> > --
> > Jianshi Huang
> >
> > LinkedIn: jianshi
> > Twitter: @jshuang
> > Github & Blog: http://huangjs.github.com/
> >
>


-- 
Bharath Vissapragada
<http://www.cloudera.com>

--089e013cb946c2c9d80503ab6590--