hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: problem starting HBase
Date Tue, 23 Nov 2010 21:50:09 GMT
I was my fault running 0.20.5 on the same cluster as 0.20.6
Switching back to 0.20.6.jar on all servers solved the problem.

On Tue, Nov 23, 2010 at 10:52 AM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:

> So before that first line... everything was fine? What happened there?
>
> On Tue, Nov 23, 2010 at 10:48 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> > Here is relevant portion of master log:
> > http://pastebin.com/HiembAxc
> >
> > On Tue, Nov 23, 2010 at 10:37 AM, Jean-Daniel Cryans <
> jdcryans@apache.org>wrote:
> >
> >> Can you dig up to where it started doing that?
> >>
> >> On Tue, Nov 23, 2010 at 10:27 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >> > http://pastebin.com/E86iPnK4
> >> >
> >> > On Tue, Nov 23, 2010 at 10:23 AM, Jean-Daniel Cryans <
> >> jdcryans@apache.org>wrote:
> >> >
> >> >> Is that really from the master log? Can we get the full log in a
> >> pastebin?
> >> >>
> >> >> J-D
> >> >>
> >> >> On Tue, Nov 23, 2010 at 7:40 AM, Ted Yu <yuzhihong@gmail.com>
wrote:
> >> >> > I backed up zookeeper dataDir to another location.
> >> >> > After clearing zookeeper dataDir, HMaster still couldn't start:
> >> >> >
> >> >> > 2010-11-23 15:30:56,095 DEBUG
> >> >> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> >> >> /hbase/master
> >> >> > got 10.202.50.100:60000
> >> >> > 2010-11-23 15:30:56,119 DEBUG
> >> >> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to
read:
> >> >> > org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode
> >> =
> >> >> > NoNode for /hbase/root-region-server
> >> >> > 2010-11-23 15:30:56,120 DEBUG
> >> >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
> >> Sleeping
> >> >> > 5000ms, waiting for root region.
> >> >> > 2010-11-23 15:31:01,125 DEBUG
> >> >> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to
read:
> >> >> > org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode
> >> =
> >> >> > NoNode for /hbase/root-region-server
> >> >> > 2010-11-23 15:31:01,125 DEBUG
> >> >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
> >> Sleeping
> >> >> > 5000ms, waiting for root region.
> >> >> >
> >> >> > Disk isn't full:
> >> >> > /dev/md2             2786058952 186234928 2456017408   8% /
> >> >> >
> >> >> > Comment is appreciated.
> >> >> >
> >> >> > On Tue, Nov 23, 2010 at 5:34 AM, Ted Yu <yuzhihong@gmail.com>
> wrote:
> >> >> >
> >> >> >> I tried to restart hbase. But the region server identified
by
> ZNode
> >> >> >> /hbase/root-region-server declared that it is not serving
root
> >> region:
> >> >> >>
> >> >> >> 2010-11-23 13:26:49,617 DEBUG
> >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
> >> >> >> locateRegionInMeta attempt 1 of 3 failed; retrying after sleep
of
> >> 5000
> >> >> >> because: Timed out trying to locate root region because:
> >> >> >> org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> >> >> >>         at
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2274)
> >> >> >>         at
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:1711)
> >> >> >>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> Method)
> >> >> >>         at
> >> >> >>
> >> >>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >> >>         at
> >> >> >>
> >> >>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >> >>         at java.lang.reflect.Method.invoke(Method.java:597)
> >> >> >>         at
> >> >> >>
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> >> >> >>         at
> >> >> >>
> >> >>
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:998)
> >> >> >>
> >> >> >> 2010-11-23 13:26:54,622 DEBUG
> >> >> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> >> >> >> /hbase/root-region-server got 10.202.50.111:60020
> >> >> >> 2010-11-23 13:26:54,624 DEBUG
> >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
> Root
> >> >> region
> >> >> >> location changed. Sleeping.
> >> >> >> 2010-11-23 13:26:59,626 DEBUG
> >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
> Wake.
> >> >> Retry
> >> >> >> finding root region.
> >> >> >> 2010-11-23 13:26:59,629 DEBUG
> >> >> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> >> >> >> /hbase/root-region-server got 10.202.50.111:60020
> >> >> >> 2010-11-23 13:26:59,630 DEBUG
> >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
> Root
> >> >> region
> >> >> >> location changed. Sleeping.
> >> >> >> 2010-11-23 13:27:04,632 DEBUG
> >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
> Wake.
> >> >> Retry
> >> >> >> finding root region.
> >> >> >> 2010-11-23 13:27:04,635 DEBUG
> >> >> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> >> >> >> /hbase/root-region-server got 10.202.50.111:60020
> >> >> >>
> >> >> >> What should I do next ?
> >> >> >>
> >> >> >> Thanks
> >> >> >>
> >> >> >>
> >> >> >> On Tue, Nov 23, 2010 at 1:37 AM, Lars George <
> lars.george@gmail.com
> >> >> >wrote:
> >> >> >>
> >> >> >>> Hi Ted,
> >> >> >>>
> >> >> >>> So one of the regions is not being released? Could you
try and
> see
> >> >> >>> from .META. which is still deployed and use the shell's
> >> "close_region"
> >> >> >>> to close it while looking at the master and region server
logs to
> >> see
> >> >> >>> what is going on? Maybe best if you switch the RS to DEBUG
level
> >> >> >>> logging first to get some info?
> >> >> >>>
> >> >> >>> Lars
> >> >> >>>
> >> >> >>> On Tue, Nov 23, 2010 at 8:25 AM, Ted Yu <yuzhihong@gmail.com>
> >> wrote:
> >> >> >>> > Hi
> >> >> >>> > We use 0.20.6
> >> >> >>> >
> >> >> >>> > I tried to disable packageindex table. From master
log:
> >> >> >>> >
> >> >> >>> > 2010-11-23 07:21:06,326 DEBUG
> >> >> >>> > org.apache.hadoop.hbase.master.ChangeTableState:
Adding region
> >> >> >>> > packageindex,CC7E6FEA4CDCF19C6F4AC9BB51EF6A33,1290230596786
to
> >> >> >>> setClosing
> >> >> >>> > list for us01-ciqps1-grid10.carrieriq.com,60020,1290493641949
> >> >> >>> > 2010-11-23 07:21:06,326 DEBUG
> >> >> >>> > org.apache.hadoop.hbase.master.ChangeTableState:
Adding region
> >> >> >>> > packageindex,F2A18967F48C9FDA9C23BF9A8210ED17,1290230394345
to
> >> >> >>> setClosing
> >> >> >>> > list for us01-ciqps1-grid11.carrieriq.com,60020,1290493641228
> >> >> >>> > 2010-11-23 07:21:06,326 DEBUG
> >> >> >>> > org.apache.hadoop.hbase.master.ChangeTableState:
Adding region
> >> >> >>> > packageindex,E8FA713B2F030EF012E5AB0A641CB1DB,1290230356969
to
> >> >> >>> setClosing
> >> >> >>> > list for us01-ciqps1-grid11.carrieriq.com,60020,1290493641228
> >> >> >>> > 2010-11-23 07:21:06,327 DEBUG
> >> >> >>> > org.apache.hadoop.hbase.master.ChangeTableState:
Adding region
> >> >> >>> > packageindex,5B10CA26DCAEFBFF4A63DB7D0432D628,1290229869191
to
> >> >> >>> setClosing
> >> >> >>> > list for us01-ciqps1-grid12.carrieriq.com,60020,1290493641232
> >> >> >>> > 2010-11-23 07:21:20,178 INFO
> >> >> >>> org.apache.hadoop.hbase.master.ServerManager:
> >> >> >>> > 15 region servers, 0 dead, average load 123.66666666666667
> >> >> >>> > 2010-11-23 07:21:20,252 INFO
> >> >> org.apache.hadoop.hbase.master.BaseScanner:
> >> >> >>> > RegionManager.rootScanner scanning meta region {server:
> >> >> >>> 10.202.50.111:60020,
> >> >> >>> > regionname: -ROOT-,,0, startKey: <>}
> >> >> >>> > 2010-11-23 07:21:20,257 INFO
> >> >> org.apache.hadoop.hbase.master.BaseScanner:
> >> >> >>> > RegionManager.rootScanner scan of 1 row(s) of meta
region
> {server:
> >> >> >>> > 10.202.50.111:60020, regionname: -ROOT-,,0, startKey:
<>}
> >> complete
> >> >> >>> > 2010-11-23 07:21:22,838 INFO
> >> >> org.apache.hadoop.hbase.master.BaseScanner:
> >> >> >>> > RegionManager.metaScanner scanning meta region {server:
> >> >> >>> 10.202.50.101:60020,
> >> >> >>> > regionname: .META.,,1, startKey: <>}
> >> >> >>> > 2010-11-23 07:21:24,731 INFO
> >> >> org.apache.hadoop.hbase.master.BaseScanner:
> >> >> >>> > RegionManager.metaScanner scan of 2086 row(s) of
meta region
> >> {server:
> >> >> >>> > 10.202.50.101:60020, regionname: .META.,,1, startKey:
<>}
> >> complete
> >> >> >>> > 2010-11-23 07:21:24,731 INFO
> >> >> org.apache.hadoop.hbase.master.BaseScanner:
> >> >> >>> All
> >> >> >>> > 1 .META. region(s) scanned
> >> >> >>> >
> >> >> >>> > But I always got:
> >> >> >>> > hbase(main):004:0> disable 'packageindex'
> >> >> >>> > NativeException: org.apache.hadoop.hbase.RegionException:
> Retries
> >> >> >>> exhausted,
> >> >> >>> > it took too long to wait for the table packageindex
to be
> >> disabled.
> >> >> >>> >
> >> >> >>> > What should I do to disable the table ?
> >> >> >>> >
> >> >> >>> > Thanks
> >> >> >>> >
> >> >> >>>
> >> >> >>
> >> >> >>
> >> >> >
> >> >>
> >> >
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message