hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tao Xiao <xiaotao.cs....@gmail.com>
Subject Re: All regions stay on two nodes out of 18 nodes
Date Thu, 17 Apr 2014 01:24:54 GMT
Actually, open that link and then click on the picture, it will zoom in and
become quite clear.

I checked the HMaster UI just now and I am sure that these regions are
always in transition,  I suppose there would be some exceptions happening.
How to prevent regions from being in transition for a long time ?


2014-04-17 9:00 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:

> The picture is not very clear.
> I don't see E_MP_DAY_READ having regions in transition.
>
> Anyway, as long as there is region in transition, balancer would not run.
>
> Cheers
>
>
> On Wed, Apr 16, 2014 at 5:52 PM, Tao Xiao <xiaotao.cs.nju@gmail.com>
> wrote:
>
> > Ted,
> >
> > I can see some regions of other tables in transition now , but I'm not
> sure
> > how long have them been in transition and I will check the HBase master
> UI
> > later. Here is the
> > screenshot<
> >
> http://picpaste.com/Regions_in_Transition_-_2014-04-17_08-38-qyf5anz8.png
> > >.
> > From the screenshot, there is a region with state of FAILED_OPEN, which
> is
> > in red, and there are 9 regions in transition for more than 60 seconds.
> >
> > Note that the table whose regions all stay in 2 nodes is E_MP_DAY_READ,
> > while the other tables shown in the screenshot are named as
> > E_MP_DAY_READ_20140315, E_MP_DAY_READ_20140322, E_MP_DAY_READ_20140324,
> and
> > so on.
> >
> > Thanks.
> >
> >
> > 2014-04-16 23:10 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:
> >
> > > bq. found some regions of other tables in transition, not of this
> table.
> > >
> > > That can explain why "balancer" command returned false.
> > > Are those regions stuck in transition ?
> > >
> > > Cheers
> > >
> > >
> > > On Tue, Apr 15, 2014 at 10:47 PM, Tao Xiao <xiaotao.cs.nju@gmail.com>
> > > wrote:
> > >
> > > > The command "balance_switch true" returns true, but the command
> > > "balancer"
> > > > returns false. I checked the HMaster UI and found some regions of
> other
> > > > tables in transition, not of this table.
> > > >
> > > > This table's name is E_MP_DAY_READ, I did grep it in the master log
> and
> > > > found only the following lines:
> > > >
> > > > 2014-04-15 15:50:59,925 INFO  [MASTER_SERVER_OPERATIONS-b03:60000-1]
> > > > handler.ServerShutdownHandler: Skip assigning region
> > > >
> > > >
> > >
> >
> E_MP_DAY_READ,160001123745_2014-01-25:00:00:00,1395753408476.ba5c8291f8dad37d5b9621b7334c17b0.
> > > > because it has been opened in a04.jsepc.com,60020,1397548219084
> > > > 2014-04-15 15:50:59,926 INFO  [MASTER_SERVER_OPERATIONS-b03:60000-1]
> > > > handler.ServerShutdownHandler: Skip assigning region
> > > >
> > > >
> > >
> >
> E_MP_DAY_READ,300007915618_2014-03-13:00:00:00,1395994146202.ec4e397baffd1cc40bdc18ce0ab2f28a.
> > > > because it has been opened in a04.jsepc.com,60020,1397548219084
> > > > 2014-04-15 15:50:59,926 INFO  [MASTER_SERVER_OPERATIONS-b03:60000-1]
> > > > handler.ServerShutdownHandler: Skip assigning region
> > > >
> > > >
> > >
> >
> E_MP_DAY_READ,300013608840_2014-02-21:00:00:00,1395749573711.744bab52befec279a7ee97497801e10f.
> > > > because it has been opened in a04.jsepc.com,60020,1397548219084
> > > > 2014-04-15 15:50:59,937 INFO  [MASTER_SERVER_OPERATIONS-b03:60000-2]
> > > > handler.ServerShutdownHandler: Skip assigning region
> > > >
> > > >
> > >
> >
> E_MP_DAY_READ,300000497780_2014-01-23:00:00:00,1395746363941.79b831e698053b1005f7a97c9f2a6ddc.
> > > > because it has been opened in a04.jsepc.com,60020,1397548219084
> > > > 2014-04-15 15:50:59,938 INFO  [MASTER_SERVER_OPERATIONS-b03:60000-2]
> > > > handler.ServerShutdownHandler: Skip assigning region
> > > >
> > > >
> > >
> >
> E_MP_DAY_READ,300008188567_2014-03-04:00:00:00,1395756104426.eb1806c2dc5833152b6b5e7b5e4a88b8.
> > > > because it has been opened in a04.jsepc.com,60020,1397548219084
> > > > 2014-04-15 15:50:59,940 INFO  [MASTER_SERVER_OPERATIONS-b03:60000-2]
> > > > handler.ServerShutdownHandler: Skip assigning region
> > > >
> > > >
> > >
> >
> E_MP_DAY_READ,300016987143_2014-01-21:00:00:00,1395986789897.e4d143865d354bdc2a427c1f00df6ad7.
> > > > because it has been opened in a04.jsepc.com,60020,1397548219084
> > > >
> > > > so few logging lines about it, looks strange ?
> > > >
> > > >
> > > > BTW, I can spread the regions of this table evenly across the whole
> > > cluster
> > > > after I shutdown the two region servers where the regions of this
> table
> > > > resided originally.
> > > >
> > > >
> > > > 2014-04-15 19:47 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:
> > > >
> > > > > Is load balancer enabled ?
> > > > >
> > > > > Can you grep this table in master log and pastebin what you found
?
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Apr 15, 2014, at 4:40 AM, Tao Xiao <xiaotao.cs.nju@gmail.com>
> > > wrote:
> > > > >
> > > > > > I am using HDP 2.0.6, which has 18 nodes(region servers). One
of
> my
> > > > HBase
> > > > > > tables has 50 regions but I found that the 50 regions all stay
in
> > > just
> > > > > two
> > > > > > nodes, not spread evenly in the 18 nodes. I did not pre-create
> > splits
> > > > so
> > > > > > this table was gradually split into 50 regions itself.
> > > > > >
> > > > > > I'd like to know why all the regions stay in just two nodes,
not
> > the
> > > 18
> > > > > > nodes of the cluster, and how to spread the regions evenly across
> > all
> > > > the
> > > > > > region servers. Thanks.
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message