hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xian Woo <infinity0...@gmail.com>
Subject Re: Region is not online: -ROOT-,,0
Date Fri, 29 Jul 2011 14:45:12 GMT
hmm, I am not  sure about that , let me check again. Thank you about that.
 ^_^

2011/7/29 Gan, Xiyun <ganxiyun@gmail.com>

> Refer to https://issues.apache.org/jira/browse/HBASE-3669
> Probably it's a bug.
>
> Is this issue reproducible?
>
> On Fri, Jul 29, 2011 at 1:22 PM, Xian Woo <infinity0222@gmail.com> wrote:
> > Actually it happened when I did a test to manually reboot one of the
>  region
> > servers.  Note that the region server I rebooted was the one that
> contained
> > -ROOT-, &.META as online regions.
> > After I rebooted the server, the cluster crashed. I had to reboot the
> entire
> > cluster , hoping that the cluster would fix this itself.  But to my
> > disappointed, the cluster didn't recover and the log showed the error
> which
> > have been listed in this mail.
> > Speaking of this, do you know anything about my problem?shouldn't
> > the  -ROOT-, &.META keep synchronized among all the servers?
> > Please contact with me if u are interested in this issue.
> > Much Thanks.
> >
> > 2011/7/29 Gan, Xiyun <ganxiyun@gmail.com>
> >>
> >> Following is my understanding. Please correct me if there are any
> >> mistakes.
> >>
> >> Did you change Zookeeper quorum configuration?
> >> Zookeeper tracks table schema and regionservers. If ZK quorum changes,
> >> the data Zookeeper saved is corrupted since the integration of the
> >> data has been destroyed.
> >> You may see some snippets from the log which says  the cluster can't
> >> connect to a previously selected ZK node.
> >>
> >> As far as I know, table data will not be lost after cleaning up the
> >> zookeeper data, but I'm not sure.
> >>
> >> On Fri, Jul 29, 2011 at 11:33 AM, Xian Woo <infinity0222@gmail.com>
> wrote:
> >> > Wow,it really helps~~ Thanks pal~~
> >> > But in fact, I really don't understand why this issue can be solved
> just
> >> > by
> >> > cleaning up the zookeeper data on each zk node.  So I really hope you
> >> > can
> >> > tell me something more about that. Thanks~
> >> > Best wishes~
> >> > Woo
> >> > 2011/7/29 Gan, Xiyun <ganxiyun@gmail.com>
> >> >>
> >> >> Hi, Woo
> >> >>     I have been confronted with this issue in our cluster. Have you
> >> >> started hbase cluster successfully ever before? If yes, try to clean
> >> >> up the zookeeper data on each zk node.
> >> >>
> >> >>
> >> >> On Fri, Jul 29, 2011 at 1:03 AM, Xian Woo <infinity0222@gmail.com>
> >> >> wrote:
> >> >> > Excuse me for clicking the 'send' button accidently..
> >> >> >
> >> >> > Log in Server4.yun.com:
> >> >> > Here are some segments:
> >> >> >
> >> >> > 2011-07-28 20:48:41,123 DEBUG
> >> >> > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats:
> total=1.64
> >> >> > MB,
> >> >> > free=197.54 MB, max=199.18 MB, blocks=0, accesses=0, hits=0,
> >> >> > hitRatio=�%,
> >> >> > cachingAccesses=0, cachingHits=0, cachingHitsRatio=�%, evictions=0,
> >> >> > evicted=0, evictedPerRun=NaN
> >> >> >
> >> >> > 2011-07-28 20:38:41,109 INFO org.apache.hadoop.ipc.HBaseServer:
IPC
> >> >> > Server
> >> >> > handler 2 on 60020: starting
> >> >> >
> >> >> > 2011-07-28 20:38:40,598 DEBUG
> >> >> > org.apache.hadoop.hbase.executor.ExecutorService: Starting executor
> >> >> > service
> >> >> > name=RS_OPEN_ROOT-server4.yun.com,60020,1311856719820,
> >> >> > corePoolSize=1,
> >> >> > maxPoolSize=1
> >> >> >
> >> >> > 2011-07-28 20:38:40,261 INFO
> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every
> >> >> > 10000000ms
> >> >> >
> >> >> >
> >> >> > Log in server5.yun.com:
> >> >> >
> >> >> > 1.It first keeps showing:
> >> >> >
> >> >> > 2011-07-28 20:37:05,616 DEBUG
> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> >> >> > NotServingRegionException; Region is not online: -ROOT-,,0
> >> >> >
> >> >> > 2. Then it writes this per 5 minutes:
> >> >> >
> >> >> > 2011-07-28 20:40:52,814 DEBUG
> >> >> > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats:
> total=1.64
> >> >> > MB,
> >> >> > free=197.54 MB, max=199.18 MB, blocks=0, accesses=0, hits=0,
> >> >> > hitRatio=锟?,
> >> >> > cachingAccesses=0, cachingHits=0, cachingHitsRatio=锟?, evictions=0,
> >> >> > evicted=0, evictedPerRun=NaN
> >> >> >
> >> >> >
> >> >> > I would really appreciate it if  you can give me some advice on
> this
> >> >> > issue.
> >> >> >
> >> >> > Thanks a lot ~
> >> >> >
> >> >> > Best Wishes~
> >> >> >
> >> >> > Woo
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > 2011/7/29 Xian Woo <infinity0222@gmail.com>
> >> >> >
> >> >> >>
> >> >> >> Hi everyone. I have just completed in installing my hbase
and
> there
> >> >> >> are
> >> >> >> some errors when I try to start my cluster.
> >> >> >>
> >> >> >> Setup:
> >> >> >>    -cdh3u1
> >> >> >>    - Hadoop 0.20.2
> >> >> >>    - HBase 0.90.1
> >> >> >>    - 1 Master Node running as NameNode & JobTracker(named
> >> >> >> server3.yun.com)
> >> >> >>    -zookeeper quorum
> >> >> >>    - 2 child nodes(server4.yun.com & server5.yun.com)
running as
> >> >> >> Datanode,
> >> >> >> TaskTracker and RegionServer each.
> >> >> >>
> >> >> >> When I try to start the master and the region servers, I find
that
> >> >> >> some
> >> >> >> issues in my log:
> >> >> >>
> >> >> >> Master log(server3.yun.com):
> >> >> >>
> >> >> >> 1. It first keeps showing this:
> >> >> >> "2011-07-28 20:30:16,056 INFO
> >> >> >> org.apache.hadoop.hbase.master.AssignmentManager: Region has
been
> >> >> >> OPENING
> >> >> >> for too long, reassigning region=-ROOT-,,0.70236052
> >> >> >> 2011-07-28 20:30:16,057 DEBUG
> >> >> >> org.apache.hadoop.hbase.master.AssignmentManager: Region has
> >> >> >> transitioned to
> >> >> >> OPENED, allowing watched event handlers to process"
> >> >> >>
> >> >> >> 2.After several minutes of the same contents written in the
log,
> >> >> >>  then:
> >> >> >> "2011-07-28 20:30:16,412 DEBUG
> >> >> >>
> >> >> >>
> >> >> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> >> >> >> Lookedup root region location,
> >> >> >>
> >> >> >>
> >> >> >>
> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@7e24d517
> ;
> >> >> >> hsa=server5.yun.com:60020
> >> >> >> 2011-07-28 20:30:16,413 DEBUG
> >> >> >>
> >> >> >>
> >> >> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> >> >> >> locateRegionInMeta parentTable=-ROOT-, metaLocation=address:
> >> >> >> server5.yun.com:60020, regioninfo: -ROOT-,,0.70236052, attempt=1
> of
> >> >> >> 10
> >> >> >> failed; retrying after sleep of 1000 because:
> >> >> >> org.apache.hadoop.hbase.NotServingRegionException: Region
is not
> >> >> >> online:
> >> >> >> -ROOT-,,0
> >> >> >>  at
> >> >> >>
> >> >> >>
> >> >> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2318)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1614)
> >> >> >>  at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >> >>  at java.lang.reflect.Method.invoke(Method.java:597)
> >> >> >> at
> >> >> >>
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> >> >> >>  at
> >> >> >>
> >> >> >>
> >> >> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> >> >> >> "
> >> >> >>
> >> >> >> 3.Then it moves back to 1 and continues checking...
> >> >> >>
> >> >> >> Log
> >> >> >>
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Best wishes
> >> >> Gan, Xiyun
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Best wishes
> >> Gan, Xiyun
> >
> >
>
>
>
> --
> Best wishes
> Gan, Xiyun
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message