hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Hbase stops working
Date Fri, 27 Feb 2009 15:58:32 GMT
Hi Andrew,

There are several threads in the hbase-users@ archive that include
useful information on this topic. The executive summary is anything
smaller than 4 reasonable physical nodes (e.g. can support 1 GB
heap for each Hadoop or HBase daemon) is probably insufficient. You
need to provide adequate CPU and heap resources for all of:
   - Hadoop DFS namenode
   - Hadoop DFS datanodes
   - HBase master
   - HBase regionservers
   - Hadoop mapreduce job tracker
   - Hadoop mapreduce task trackers
   - all of the individual map and reduce tasks you are executing concurrently

This stuff is meant for large cluster deployment: 20, 50, 100, 1000
nodes.

We really should collect all of this together into a "For Beginners"
page that points out that the full load of Hadoop and HBase daemons
is meant for cluster deployment and usage, not single machine 
testing, not two or three node small EC2 instances, and so on. 

For example I was not able to get even Hadoop DFS stable under the
load of my application before allocating 20 server class machines
to the problem, never mind HBase on top. And this was during 
initial development and testing.

One problem I do see every Hadoop and HBase user suffering is a 
lack of clear capacity estimation tools. In part this is because
the tools are new and in flux. In part this is because capacity
estimation for load spreading is such a case-by-case application
by application consideration. I hope this can be rectified over
time as the community contributes their experiences. For now it is
a trial and error proposition: Test under load and if Hadoop or
HBase fail for clear overloading cause (thread starvation, OOME,
and the like), add more resources, and repeat.

   - Andy

> From: Andrew McCall <andrew.mccall@goroam.net>
> Subject: Re: Hbase stops working
> To: hbase-user@hadoop.apache.org
> Date: Thursday, February 26, 2009, 10:22 AM
> What is a rough sort of minimum of cores or machines I
> should be looking at for a deployment for development
> purposes? Also where would I be best putting everything?
> 
> Would a two or three single coremachine cluster with be
> enough to run everything? Should I split the jobtracker,
> namenode and hbase master on the three machines in the
> cluster? Or would I be better off getting one bigger
> multicore machine as a jobtracker, namenode and master to
> use for development then add others later?
> 
> Given what I've learned, it looks like I'll be
> spending the next few days moving to EC2, and continuing my
> development there because I think trying to continue on this
> machine is probably a lost cause and waste of time. :)
> 
> Thanks for all the help and feedback everyone.
> 
> Andrew
> 
> On 26 Feb 2009, at 17:20, Jim Kellerman (POWERSET) wrote:
> 
> > With a single core machine, I'm amazed it works at
> all. Between the
> > namenode, data node(s), job tracker, task tracker,
> hbase master and
> > regionserver(s), that's a whole lot of threads (on
> the order of
> > several hundred). Thread starvation is a real
> possibility as the
> > jvm makes no guarantees about which runnable thread
> gets the cpu next.
> > 
> > ---
> > Jim Kellerman, Powerset (Live Search, Microsoft
> Corporation)
> > 
> >> -----Original Message-----
> >> From: jdcryans@gmail.com
> [mailto:jdcryans@gmail.com] On Behalf Of Jean-
> >> Daniel Cryans
> >> Sent: Thursday, February 26, 2009 8:01 AM
> >> To: hbase-user@hadoop.apache.org
> >> Subject: Re: Hbase stops working
> >> 
> >> Andrew,
> >> 
> >> At 512MB I'm sure it swapped, tho it would be
> hard to tell now unless you
> >> installed some monitoring software. Next time, you
> should at least monitor
> >> your job using top to make sure it's the root
> cause of the problem.
> >> 
> >> Something that may help you is in
> >>
> http://wiki.apache.org/hadoop/Hbase/Troubleshooting#7 mainly
> the third
> >> bullet point in Resolution (adding
> -XX:+UseConcMarkSweepGC in hbase-env.sh
> >> in the HBASE_OPTS and uncommenting it).
> >> 
> >> J-D
> >> 
> >> On Thu, Feb 26, 2009 at 10:14 AM, Andrew McCall
> >> <andrew.mccall@goroam.net>wrote:
> >> 
> >>> Hi J-D,
> >>> 
> >>> I'm running the HBase 0.19 release
> downloaded to the website on a pretty
> >>> lacklustre linux machine; 1.6Ghz single core
> processor, 512MB ram with
> >> 1.5GB
> >>> swap space enabled and 80GB hard disk.
> It's a testing machine, I've been
> >>> using for development. It's running Hadoop
> core 0.19 in pseudo
> >> distributed
> >>> mode and running a map/reduce reduce crawl
> using nutch and HBase. I've
> >> been
> >>> planning to move to an EC3 cluster in the near
> future but I've been
> >> putting
> >>> it off since everything seemed to come up and
> work on the testing box.
> >>> 
> >>> It's entirely possible that it is a result
> of a swap, how do I tell?
> >> I've
> >>> looked in all the logs for hadoop, hbase and
> my job and can't see that
> >>> anything stopped logging around then, the logs
> all have entries for
> >> 9:34.
> >>> I've also increased my lease timeout from
> the default to 600,000 ms in
> >> my
> >>> hbase-site, which is below as a result of
> another issue. I did search of
> >> the
> >>> mailing list for the other issue and answers
> suggested that the lease
> >>> timeout should be the same as my map/reduce
> job timeout.
> >>> 
> >>> <configuration>
> >>>       <property>
> >>>          
> <name>hbase.rootdir</name>
> >>>          
> <value>hdfs://localhost:9000/hbase</value>
> >>>           <description>The directory
> shared by region servers.
> >>>           </description>
> >>>         </property>
> >>>         <property>
> >>>          
> <name>hbase.regionserver.lease.period</name>
> >>>           <value>600000</value>
> >>>         </property>
> >>> </configuration>
> >>> 
> >>> Thanks for the help,
> >>> Andrew
> >>> 
> >>> 
> >>> 
> >>> On 26 Feb 2009, at 13:41, Jean-Daniel Cryans
> wrote:
> >>> 
> >>> Andrew,
> >>>> 
> >>>> Oh yes sorry, your setup is in local mode
> so no region server in a
> >>>> different
> >>>> thread. My bad!
> >>>> 
> >>>> That lease timeout probably means that you
> were swapping and a full jvm
> >>>> garbage collecting occurred which blocked
> the main thread. Can you make
> >>>> sure
> >>>> that it did swap at that moment?
> >>>> 
> >>>> It would also be helpful to get the usual
> info like hardware, OS, hbase
> >>>> version, etc.
> >>>> 
> >>>> Thx!
> >>>> 
> >>>> J-D
> >>>> 
> >>>> On Thu, Feb 26, 2009 at 8:35 AM, Andrew
> McCall
> >> <andrew.mccall@goroam.net
> >>>>> wrote:
> >>>> 
> >>>> Hi J-D,
> >>>>> 
> >>>>> I did check the region server logs and
> they didn't seem to be very
> >> useful
> >>>>> at all:
> >>>>> 
> >>>>> Mon Feb 23 22:11:57 GMT 2009 Starting
> regionserver on betty.goroam.net
> >>>>> ulimit -n 1024
> >>>>> 2009-02-23 22:11:58,817 WARN
> >>>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer: Not
> starting a
> >>>>> distinct
> >>>>> region server because hbase.master is
> set to 'local' mode
> >>>>> 
> >>>>> Is all that is in them, once for each
> time I started HBase.
> >>>>> 
> >>>>> I saw the shutdown message, which was
> preceded by this:
> >>>>> 
> >>>>> 2009-02-24 09:34:41,579 INFO
> >>>>>
> org.apache.hadoop.hbase.master.ServerManager:
> >>>>> 127.0.0.1:40076 lease expired
> >>>>> 
> >>>>> I can't find any evidence it's
> related to my map/reduce task, it's the
> >>>>> only
> >>>>> one in the log files and it's
> suspiciously just before the process
> >>>>> shutdown
> >>>>> message - but, I'm new to this and
> honestly don't know what to make of
> >>>>> it.
> >>>>> 
> >>>>> Andrew
> >>>>> 
> >>>>> 
> >>>>> On 26 Feb 2009, at 12:57, Jean-Daniel
> Cryans wrote:
> >>>>> 
> >>>>> Andrew,
> >>>>> 
> >>>>>> 
> >>>>>> For some reason, your region
> server shut down around 9:36:
> >>>>>> 
> >>>>>> 2009-02-24 09:36:05,884 INFO
> org.apache.hadoop.hbase.
> >>>>>> 
> >>>>>> 
> >>>>>>> master.RegionServerOperation:
> process shutdown of server
> >>>>>>> 127.0.0.1:40076
> >>>>>>> :
> >>>>>>> logSplit: false,
> rootRescanned: false, numberOfMetaRegions: 1,
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> But just before that the master
> received a startup message from it.
> >> Did
> >>>>>> you
> >>>>>> take a look at your region server
> log? What does it say around that
> >>>>>> time?
> >>>>>> 
> >>>>>> Thx
> >>>>>> 
> >>>>>> J-D
> >>>>>> 
> >>>>>> On Thu, Feb 26, 2009 at 5:34 AM,
> Andrew McCall <
> >>>>>> andrew.mccall@goroam.net
> >>>>>> 
> >>>>>>> wrote:
> >>>>>>> 
> >>>>>> 
> >>>>>> I've been testing a nuch on
> HBase and I'm running into an issue with
> >> it
> >>>>>> 
> >>>>>>> when it's been running for
> a while.
> >>>>>>> 
> >>>>>>> I'm running hbase 0.19 on
> a single local host pseudo distributed
> >> hadoop
> >>>>>>> 0.19 cluster.
> >>>>>>> 
> >>>>>>> After some time I get the
> following in my logs:
> >>>>>>> 
> >>>>>>> 2009-02-24 09:31:18,664 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> All 1 .META. region(s) scanned
> >>>>>>> 2009-02-24 09:31:18,664 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.metaScanner
> scanning meta region {regionname:
> >> .META.,,1,
> >>>>>>> startKey: <>, server:
> 127.0.0.1:40076}
> >>>>>>> 2009-02-24 09:31:24,293 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.rootScanner scan
> of 1 row(s) of meta region
> >> {regionname:
> >>>>>>> -ROOT-,,0, startKey: <>,
> server: 127.0.0.1:40
> >>>>>>> 076} complete
> >>>>>>> 2009-02-24 09:31:24,293 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.rootScanner
> scanning meta region {regionname: -ROOT-
> >> ,,0,
> >>>>>>> startKey: <>, server:
> 127.0.0.1:40076}
> >>>>>>> 2009-02-24 09:31:27,217 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.metaScanner scan
> of 2 row(s) of meta region
> >> {regionname:
> >>>>>>> .META.,,1, startKey: <>,
> server: 127.0.0.1:40
> >>>>>>> 076} complete
> >>>>>>> 2009-02-24 09:31:27,217 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> All 1 .META. region(s) scanned
> >>>>>>> 2009-02-24 09:31:55,533 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.rootScanner scan
> of 1 row(s) of meta region
> >> {regionname:
> >>>>>>> -ROOT-,,0, startKey: <>,
> server: 127.0.0.1:40
> >>>>>>> 076} complete
> >>>>>>> 2009-02-24 09:32:18,665 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.metaScanner
> scanning meta region {regionname:
> >> .META.,,1,
> >>>>>>> startKey: <>, server:
> 127.0.0.1:40076}
> >>>>>>> 2009-02-24 09:32:35,749 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.rootScanner
> scanning meta region {regionname: -ROOT-
> >> ,,0,
> >>>>>>> startKey: <>, server:
> 127.0.0.1:40076}
> >>>>>>> 2009-02-24 09:32:38,957 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.metaScanner scan
> of 2 row(s) of meta region
> >> {regionname:
> >>>>>>> .META.,,1, startKey: <>,
> server: 127.0.0.1:40
> >>>>>>> 076} complete
> >>>>>>> 2009-02-24 09:32:38,957 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> All 1 .META. region(s) scanned
> >>>>>>> 2009-02-24 09:33:18,771 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.metaScanner
> scanning meta region {regionname:
> >> .META.,,1,
> >>>>>>> startKey: <>, server:
> 127.0.0.1:40076}
> >>>>>>> 2009-02-24 09:33:54,889 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.rootScanner scan
> of 1 row(s) of meta region
> >> {regionname:
> >>>>>>> -ROOT-,,0, startKey: <>,
> server: 127.0.0.1:40
> >>>>>>> 076} complete
> >>>>>>> 2009-02-24 09:33:57,387 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.rootScanner
> scanning meta region {regionname: -ROOT-
> >> ,,0,
> >>>>>>> startKey: <>, server:
> 127.0.0.1:40076}
> >>>>>>> 2009-02-24 09:34:41,579 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.ServerManager:
> >>>>>>> 127.0.0.1:40076 lease expired
> >>>>>>> 2009-02-24 09:34:53,440 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.metaScanner scan
> of 2 row(s) of meta region
> >> {regionname:
> >>>>>>> .META.,,1, startKey: <>,
> server: 127.0.0.1:40
> >>>>>>> 076} complete
> >>>>>>> 2009-02-24 09:34:53,441 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> All 1 .META. region(s) scanned
> >>>>>>> 2009-02-24 09:34:53,441 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.metaScanner
> scanning meta region {regionname:
> >> .META.,,1,
> >>>>>>> startKey: <>, server:
> 127.0.0.1:40076}
> >>>>>>> 2009-02-24 09:35:38,720 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.metaScanner scan
> of 2 row(s) of meta region
> >> {regionname:
> >>>>>>> .META.,,1, startKey: <>,
> server: 127.0.0.1:40
> >>>>>>> 076} complete
> >>>>>>> 2009-02-24 09:35:38,720 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> All 1 .META. region(s) scanned
> >>>>>>> 2009-02-24 09:35:56,951 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.metaScanner
> scanning meta region {regionname:
> >> .META.,,1,
> >>>>>>> startKey: <>, server:
> 127.0.0.1:40076}
> >>>>>>> 2009-02-24 09:36:05,667 INFO
> >>>>>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> >>>>>>> MSG_CALL_SERVER_STARTUP
> >>>>>>> 2009-02-24 09:36:05,668 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.rootScanner scan
> of 1 row(s) of meta region
> >> {regionname:
> >>>>>>> -ROOT-,,0, startKey: <>,
> server: 127.0.0.1:40
> >>>>>>> 076} complete
> >>>>>>> 2009-02-24 09:36:05,884 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.RegionServerOperation:
> process
> >> shutdown
> >>>>>>> of
> >>>>>>> server 127.0.0.1:40076:
> logSplit: false, rootRescanned: false,
> >>>>>>> numberOfMetaRegions: 1,
> >>>>>>> onlineMetaRegions.size(): 1
> >>>>>>> 2009-02-24 09:36:17,689 INFO
> >> org.apache.hadoop.hbase.regionserver.HLog:
> >>>>>>> Splitting 1 log(s) in
> >>>>>>>
> hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076
> >>>>>>> 2009-02-24 09:36:25,034 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.metaScanner scan
> of 2 row(s) of meta region
> >> {regionname:
> >>>>>>> .META.,,1, startKey: <>,
> server: 127.0.0.1:40
> >>>>>>> 076} complete
> >>>>>>> 2009-02-24 09:36:25,034 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> All 1 .META. region(s) scanned
> >>>>>>> 2009-02-24 09:36:36,598 INFO
> >> org.apache.hadoop.hbase.regionserver.HLog:
> >>>>>>> log
> >>>>>>> file splitting completed for
> >>>>>>>
> hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076
> >>>>>>> 2009-02-24 09:36:56,953 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.metaScanner
> scanning meta region {regionname:
> >> .META.,,1,
> >>>>>>> startKey: <>, server:
> 127.0.0.1:40076}
> >>>>>>> 2009-02-24 09:37:05,279 ERROR
> >>>>>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed
> >> openScanner
> >>>>>>>
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer
> >> .java:2065)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServ
> >> er.java:1699)
> >>>>>>>   at
> sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >> pl.java:25)
> >>>>>>>   at
> java.lang.reflect.Method.invoke(Method.java:597)
> >>>>>>>   at
> >>>>>>>
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
> >>>>>>> 2009-02-24 09:37:06,102 INFO
> org.apache.hadoop.ipc.HBaseServer: IPC
> >>>>>>> Server
> >>>>>>> handler 4 on 40076, call
> openScanner([B@1ac5a78, [[B@6799dc,
> >>>>>>> [B@1a66cfc,
> >>>>>>> 9223372036854775807, null)
> from 127.0.0.1
> >>>>>>> :50139: error:
> org.apache.hadoop.hbase.NotServingRegionException:
> >>>>>>> .META.,,1
> >>>>>>>
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer
> >> .java:2065)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServ
> >> er.java:1699)
> >>>>>>>   at
> sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >> pl.java:25)
> >>>>>>>   at
> java.lang.reflect.Method.invoke(Method.java:597)
> >>>>>>>   at
> >>>>>>>
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
> >>>>>>> 2009-02-24 09:37:07,051 WARN
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> Scan one META region:
> {regionname: .META.,,1, startKey: <>, server:
> >>>>>>> 127.0.0.1:40076}
> >>>>>>>
> org.apache.hadoop.hbase.NotServingRegionException:
> >>>>>>>
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer
> >> .java:2065)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServ
> >> er.java:1699)
> >>>>>>>   at
> sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >> pl.java:25)
> >>>>>>>   at
> java.lang.reflect.Method.invoke(Method.java:597)
> >>>>>>>   at
> >>>>>>>
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
> >>>>>>> 
> >>>>>>>   at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >>>>>>> Method)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcc
> >> essorImpl.java:39)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstr
> >> uctorAccessorImpl.java:27)
> >>>>>>>   at
> >>
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(Remot
> >> eExceptionHandler.java:95)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185
> >> )
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.j
> >> ava:73)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.jav
> >> a:129)
> >>>>>>>   at
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137)
> >>>>>>>   at
> org.apache.hadoop.hbase.Chore.run(Chore.java:65)
> >>>>>>> 2009-02-24 09:37:07,230 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> All 1 .META. region(s) scanned
> >>>>>>> 2009-02-24 09:37:50,642 INFO
> >>>>>>>
> org.apache.hadoop.hbase.regionserver.HRegion:
> >>>>>>> Closed .META.,,1
> >>>>>>> 2009-02-24 09:37:56,955 INFO
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> RegionManager.metaScanner
> scanning meta region {regionname:
> >> .META.,,1,
> >>>>>>> startKey: <>, server:
> 127.0.0.1:40076}
> >>>>>>> 2009-02-24 09:37:57,126 ERROR
> >>>>>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed
> >> openScanner
> >>>>>>>
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer
> >> .java:2065)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServ
> >> er.java:1699)
> >>>>>>>   at
> sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >> pl.java:25)
> >>>>>>>   at
> java.lang.reflect.Method.invoke(Method.java:597)
> >>>>>>>   at
> >>>>>>>
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
> >>>>>>> 2009-02-24 09:37:59,129 INFO
> org.apache.hadoop.ipc.HBaseServer: IPC
> >>>>>>> Server
> >>>>>>> handler 2 on 40076, call
> openScanner([B@155bf4e, [[B@82b436,
> >>>>>>> [B@1056757,
> >>>>>>> 9223372036854775807, null)
> from 127.0.0.1
> >>>>>>> :35804: error:
> org.apache.hadoop.hbase.NotServingRegionException:
> >>>>>>> .META.,,1
> >>>>>>>
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer
> >> .java:2065)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServ
> >> er.java:1699)
> >>>>>>>   at
> sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >> pl.java:25)
> >>>>>>>   at
> java.lang.reflect.Method.invoke(Method.java:597)
> >>>>>>>   at
> >>>>>>>
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
> >>>>>>> 2009-02-24 09:38:00,412 WARN
> >>>>>>>
> org.apache.hadoop.hbase.master.BaseScanner:
> >>>>>>> Scan one META region:
> {regionname: .META.,,1, startKey: <>, server:
> >>>>>>> 127.0.0.1:40076}
> >>>>>>>
> org.apache.hadoop.hbase.NotServingRegionException:
> >>>>>>>
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer
> >> .java:2065)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServ
> >> er.java:1699)
> >>>>>>>   at
> sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >> pl.java:25)
> >>>>>>>   at
> java.lang.reflect.Method.invoke(Method.java:597)
> >>>>>>>   at
> >>>>>>>
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
> >>>>>>> 
> >>>>>>>   at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >>>>>>> Method)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcc
> >> essorImpl.java:39)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstr
> >> uctorAccessorImpl.java:27)
> >>>>>>>   at
> >>
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(Remot
> >> eExceptionHandler.java:95)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185
> >> )
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.j
> >> ava:73)
> >>>>>>>   at
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.jav
> >> a:129)
> >>>>>>>   at
> >>>>>>> 
> >>
> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137)
> >>>>>>>   at
> org.apache.hadoop.hbase.Chore.run(Chore.java:65)
> >>>>>>> 
> >>>>>>> This just repeats forever,
> until I restart HBase.
> >>>>>>> 
> >>>>>>> Any help would be greatly
> appreciated.
> >>>>>>> Andrew
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>> 
> >>>


      

Mime
View raw message